Thoughts on Google Fiber for Appleton / ISPs

Tonight I attended a public hearing at Appleton City Hall (PDF) regarding the city's consideration to submit a response to Google's request for information on the Google Fiber for Communities experiment. (Don't miss Google's project overview and other linked pages.) Also, please join the Google Fiber for Appleton Facebook group.

I was pleasantly surprised to see this public hearing bring the attention of Green Bay TV stations WFRV / CBS channel 5 and WGBA / NBC channel 26. The public hearing started late due to another meeting in the same room. There were probably a few more than 20 people in attendance, and the response was overwhelmingly positive. This echoed the current status of the City of Appleton's Survey, which was mentioned to also be overwhelmingly in favor of submitting a proposal. In general, the responses given at the hearing were mostly focused on the points that Appleton should proceed with submitting a favorable proposal in order to remain competitive as a community, to bring additional competition and choices for Internet service, and many other convincing reasons.

There were only 3 responses against: 2 from AT&T representatives in attendance for somewhat obvious reasons in concern for their business, and 1 gentleman concerned with the physical cost necessary to connect to connect gigabit networking to his Apple computer. (He apparently assumed that the fiber would need to be connected directly to his computer. Most computers sold in the past few years already have gigabit Ethernet cards, or they are readily available for less than $50. Additionally, there are many potential uses beyond a computer, such as video and other multimedia.) As also mentioned by another resident at the meeting, even being able to connect at only 100 Mbps (vs. the 1 Gbps / 1,000 Mbps being advertised) would still be about 10x faster than most consumer broadband connections available today.

My Own Thoughts

In general, additional competition for residential Internet service can only be a good thing - whether that competition is from Google or another provider. At my current residence in an apartment just outside the official city limits (in Grand Chute, near the Fox River Mall), my choices for broadband Internet are currently limited to Time Warner Cable and DSL. Wireless / "3G" was tried, and is not current viable for primary / serious use as previously detailed. Interestingly, the reps from AT&T at the hearing used their wireless network as one of their primary arguments against submitting a proposal to Google. After looking into Time Warner Cable, they presented themselves as one of the shadiest operations in town, at least based upon my experience at their local office. I'm currently using AT&T's DSL. While I would like to sign-up for AT&T U-Verse, I'm told that it is not available to my particular building, despite several of my neighbors in very close proximity having the service.

Unlike many of the comments and hype, I would hope that Google's offering is not all about the speed - even though, unfortunately, this is all many residential consumers are aware of or take into consideration. A few questions to consider of any ISP:

• In addition to the speed, what is the latency (or lag)?
• What is the support for IPv6?
• Are industry standards properly followed, such as RFC 2308 - Negative Caching of DNS Queries? Or is the ISP involved in DNS hijacking?
• Is service advertised as "unlimited" truly unlimited, or are there limits involved that are only shown in fine print, if at all?
• What is the ISP's stance on network neutrality?
• What support / allowance is there for operating as a server - either a web server, or something more "residential" such as allowing remote desktop connections, allowing for peer-to-peer file transfers, or playing games that require being able to accept incoming network connections?
• What are the results from the ICSI Netalyzer hosted by UC Berkeley, which tests for most of the above as well other issues?

Google, in particular, has a positive record for properly supporting the above requirements and avoiding the listed issues:

• In order to help solve the issues that many ISPs cause with their crippled DNS servers, Google started offering their own public DNS, promising to improve performance, security, and to keep strict adherence with standards.
• Google is embracing and supporting IPv6:
  • Looking towards IPv6 (Lorenzo Colitti and Erik Kline, 2008-05-13, googleblog.blogspot.com)
  • Google over IPv6 (google.com)
  • Google: IPv6 is easy, not expensive (Carolyn Duffy Marsan, 2009-05-25, networkworld.com)
• Google is an outspoken proponent (googlepublicpolicy.blogspot.com) of network neutrality.
• Google has demonstrated the proper way to provide transparency around issues, such as the reports around their recent app engine outage: Google App Engine downtime postmortem, nearly a perfect model for others (Lenny Rachitsky, 2010-03-05, transparentuptime.com) and When the Power Goes Out At Google (2010-03-08, hardware.slashdot.org).

I can't imagine that Google wouldn't uphold the same principals in providing their own Internet service.

(On a humorous note, I can't help but recall Google's previous ISP / fiber offering: Google TiSP.)

Even if Appleton submits a proposal for and is accepted as a location for Google Fiber, there is no guarantee that I would be in the service area - especially being in a neighboring town. However, I have no doubt that I would still benefit from the increased competition. Additionally, once we're back in the market to buy a a house in the area, the availability of Google Fiber would be a serious consideration. (Someone please buy our house for sale in Wausau! - which happens to be wired with Cat-6 for Gigabit networking.)

Google Sites: Limitations

As excited as I was to start posting content on Google Sites (formerly JotSpot), there are a number of issues / limitations that I think are show-stoppers for now.

Google Sites offers a number of advantages, including a wiki format and interface, and as part of its inclusion into Google Apps, can be hosted on a custom domain.

However, after working with a number of different wiki products, both free and commercial, Google Sites seems excessively simple, to the point of restrictive. One example is when creating a new page, a type of page must be chosen. There then seems to be no option for changing that page's type, short of deleting it and creating a new one.

Deleting pages, however, is what I see to be one of the biggest issue with the current version of Sites. While a page or attachment can be "deleted", either seems to be infinitely recoverable, with no option to "permanently delete". As discussed in this thread, Steven Hind ("Visionary") wrote:

The ability to recover deleted pages is absolutely necessary as Google Sites allows collaboration, and a collaborator may delete a page (unintentionally or deliberately) and the owner may need to get it back.

While that is definitely a valid use-case, site owners should at least be given the option to permanently purge / remove "deleted" items. There are usability reasons, such as difficulty in recreating pages with the same name. More seriously are the security concerns, e.g. if something accidentally gets posted that shouldn't have been. Granted, even if something disappears from Sites, it may still be obtainable through Google Cache or the Internet Archive: Wayback Machine. However, even these services properly respect robot exclusions.

Working with included page images is a similar but even more mysterious issue. Images inserted into a page without being included "as an attachment" seem to function as "hidden attachments". Even when removed from the page, they remain available for later repeated use, with no visible method of removal.

Searching for "delete page" on the Site's Help returns dozens of results, with many users sharing the same concern. Hopefully this is something Google will address in the near future.

In the meantime, I'm still looking for a decent solution where I can share some of my code and projects, as open source, but under "my control". While I'll continue to use dev.java.net, it's really only purposed for Java projects. I have at least one Microsoft .NET project that I'd like to share. Google Code and SourceForge.net are two other options I explored, but again, leave very little control left with the owner. Especially when built on Subversion, the Subversion Obliterate issue makes it practically impossible to perform a removal when needed.

Update (2008-12-27): I just found this: http://groups.google.com/group/sites-help/web/wishlist. "Ability to permanently delete an uploaded file" is included, but it doesn't mention anything about permanent deletion / removal of pages.

Gmail migration with IMAP and the Java Mail API

As I previously posted, Gmail has IMAP support. I've been using it for a while now. It's a great way to use Gmail with a a desktop email client such as Mozilla Thunderbird, as well as having offline access to your email.

IMAP also offers another significant feature to Gmail users: The ability to easily migrate messages between accounts - either between different Gmail accounts or from another source into Gmail. Google offers their own tool for this, but it's only available for commercial and educational accounts - not for their personal accounts like the ones I have.

I attempted to run my transfer using the configured accounts in Microsoft Outlook, very much like Ashish Mohta described on his blog. Things were looking good until I started getting the following error:

The current command did not succeed.
The mail server responded: Unable to append message to folder (Failure).

The first thing I suspected was an Outlook issue, so I switched to Thunderbird but got the same result. I suspected I may be hitting some sort of transfer limit, so I tried downloading everything from my first account to a local folder first. That worked, but then I still wasn't able to upload from the local folder into the second account. A Google search shows a number of other people having the same issue. Google help has two (1, 2) answers that primarily blame formatting incompatibilities, but as indicated by by other results in the Google search, this definitely appears to be the result of exceeding some unposted transfer limits.

There didn't seem to be any way to configure either Outlook or Firebird to "slow things down", without manually moving only a few messages at a time - which would be tedious and prone to error. Additionally, I guessed that both clients were performing additional tasks behind the scenes that were making the issue worse, such as needlessly refreshing folders.

JavaMail API

I figured that my best chance of getting everything how I wanted it was to script the transfer. For this, I turned to the JavaMail API. I've used this before, but mostly just for sending messages over SMTP. Surprisingly, use with Gmail is even listed in the JavaMail API FAQ.

Connecting was quite simple. The only property that I needed to pass into Session.getInstance(…) was a value of "imaps" for "mail.store.protocol". For simplicity, I then used the Store's connect(String host, String user, String password) method to connect.

To open the "All Mail" folder in Gmail, either .getFolder("[Gmail]").getFolder("All Mail") or .getFolder("[Gmail]/"All Mail") works.

I did my transfer in two separate steps - the download than the upload. This was mostly because I also wanted a local, backup copy to keep. By iterating through each Message from the Folder's getMessages(), the Message's writeTo(OutputStream) method allows for easily saving the entire message - including headers and attachments - in MIME format. Similarly, the messages can then be recreated using the MimeMessage(Session, InputStream) constructor, then imported into an IMAP folder using the appendMessages(Message[]) method.

Alternatively - though I haven't tried it - opening a session to each account and using the Folder's copyMessages(Message[], Folder) method looks like it would also be convenient. Note that the "this" Folder is the source, and the Folder parameter is the destination. However, while the method is convenient for copying multiple messages at once, it may need to be coded for only one message at a time, so a pause can be inserted between transfers to account for the Gmail restrictions described above.

To slow down the transfers to keep the Gmail servers happy, after each download or upload I simply called Thread.sleep(long millis). I started out at 2,500 ms, but had this down to 500 ms by the time I finished without any further issues.

Sent Mail

Another issue I had was that I wanted/needed to re-populate my "Sent Mail" folder. This may not always be an issue, but due to any number of reasons in my case, Gmail no longer recognized any of my sent messages as "sent". While Gmail seems to treat "Sent Mail" just like any other "label", the Gmail interface doesn't provide any options to add or remove messages from it, short of deletion. However, it can be modified through IMAP. While copying in each new message, I checked to see if the getSender() method matched one of email addresses. If so, it should be copied into the "Sent Mail" folder rather than "All Items". Note the difference in using the sender rather than the "From" attribute from getFrom(). The later would typically include items that were actually received, probably from an automated sender, where it was only made to appear "from" the same email address.

Originally, I simply imported all mail into the "All Mail" folder, then copied it into "Sent Mail" as needed. However, passing the same MimeMessage into Folder.appendMessages(Message[]) then again to copyMessages(Message[], Folder) for "Sent Mail" results in a ClassCastException as copyMessages expects a IMAPMessage variant of a message that already exists in the IMAP store. The solution would be to obtain a proper reference to the inserted message to copy. The Message[] com.sun.mail.imap.IMAPFolder#addMessages(Message[]) method would allow for exactly this, but it appears that Gmail doesn't currently support the required UIDPLUS extension from RFC 2359. The only apparent remaining option would be to recall getMessages(), then iterate through and find the message matching the one inserted, probably by getMessageID().

Even after all this work, I still ran into an issue with the Gmail web interface showing all my messages in my "Sent Items" as "To: me", rather than the actual recipients. The issue wasn't related to anything with the transfer, but in my Gmail account settings. I had assumed that Gmail calculated "me" as mail sent to any address listed in the "Send mail as" list. Instead - and probably more reasonable - it is almost the opposite. If the email is sent to any address not listed in the "Send mail as" list, it is assumed by Gmail to be "To: me".

Other tools

After I successfully finished my migration using the above method, I did find some other tools that appear to solve many of the same issues. 2 that I found off of related Google searches are imapsync and IMAPSize. (I've not downloaded or tried either of these.)

Gmail has IMAP!

After hearing the promising news on Slashdot, I'm excited to learn that my Google Mail / Gmail account now has IMAP support!

More information is available on a Google Help Page, Getting Started with IMAP for Gmail.

I've been hosting my own email and server for a while now, mainly because I refused to live with one of the disadvantages typical of webmail - not being able to access email when disconnected from the Internet. With support to synchronize messages between Gmail and any IMAP client, including Microsoft Outlook and Mozilla Thunderbird, this is looking to be a very promising replacement to my current setup!

Google redirects (privacy?)

I was just running a Google search after using Firefox's LiveHTTPHeaders (see my Firefox Extensions posting) for some other web development, and found something interesting. It appears that Google started tracking the search results I click on.

For example, after running a search for "Java", the first result listed is for "Java Technology" at java.sun.com. Doing a mouse-over on the link shows the same site in the status bar. However, right-clicking on the link and choosing "Copy Link Location" results in something different: http://www.google.com/url?sa=t&ct=res&cd=1&url=http%3A%2F%2Fjava.sun.com%2F&ei=… (They are probably using some JavaScript, though I haven't dug into the matter in any detail.)

I know that practically all the search engines analyze, aggregate, or otherwise are interested in the search terms entered. However, I think the click-throughs are a much greater privacy concern, and thought the practice was abandoned long ago. (Not true for advertising, which depends on these redirects for determining the number of click-throughs for billing, etc. This I understand and somewhat respect...)

I quickly found that this behavior only exists when I'm logged-in to Google, which I use for my Gmail and other Google services. I'm suspecting this is related to their search history feature, though I currently keep all my search histories as 'paused', so I don't see the need to run all my clicks through their redirector. Additionally, besides the privacy concern, this results in an extra HTTP request/response for any link I click on...

I searched around, and couldn't find any information on why these redirects exist and what they are used for, but couldn't find anything. (If you know, please share!) So I asked Google for some details using the "contact us" link on their Search History Privacy Notice page. I got an automated response back the next day:

Hi,

For assistance, please visit the Google Help Center at http://www.google.com/support?ctx=web, where you'll find extensive, up-to-date information and solutions.

If you don't find the answer to your question in the Google Web Search Help Center and would like additional assistance, you can search the Google Web Search Help group at http://groups.google.com/group/Google_Web_Search_Help or post your question for other members of the group.

Finally, if you require assistance with Google Web Search that can't be found in the online help resources listed above, you can submit your question to the appropriate support team by clicking on one of the "Contact us" links in the Google Web Search Help Center.

Regards,
The Google Team

Well, I had already tried the "Google (Web Search) Help Center". And for the life of me, I can't find any "Contact us" links there, either.

Without seeing any better options, I replied again to the email, and it hasn't bounced back yet. I'll give it a few more days, and might consider sending something to the listed mailing address on their site ("snail mail") if I still don't hear anything back.

I'll post any updates here as I learn more...