Gmail migration with IMAP and the Java Mail API

As I previously posted, Gmail has IMAP support. I've been using it for a while now. It's a great way to use Gmail with a a desktop email client such as Mozilla Thunderbird, as well as having offline access to your email.

IMAP also offers another significant feature to Gmail users: The ability to easily migrate messages between accounts - either between different Gmail accounts or from another source into Gmail. Google offers their own tool for this, but it's only available for commercial and educational accounts - not for their personal accounts like the ones I have.

I attempted to run my transfer using the configured accounts in Microsoft Outlook, very much like Ashish Mohta described on his blog. Things were looking good until I started getting the following error:

The current command did not succeed.
The mail server responded: Unable to append message to folder (Failure).

The first thing I suspected was an Outlook issue, so I switched to Thunderbird but got the same result. I suspected I may be hitting some sort of transfer limit, so I tried downloading everything from my first account to a local folder first. That worked, but then I still wasn't able to upload from the local folder into the second account. A Google search shows a number of other people having the same issue. Google help has two (1, 2) answers that primarily blame formatting incompatibilities, but as indicated by by other results in the Google search, this definitely appears to be the result of exceeding some unposted transfer limits.

There didn't seem to be any way to configure either Outlook or Firebird to "slow things down", without manually moving only a few messages at a time - which would be tedious and prone to error. Additionally, I guessed that both clients were performing additional tasks behind the scenes that were making the issue worse, such as needlessly refreshing folders.

JavaMail API

I figured that my best chance of getting everything how I wanted it was to script the transfer. For this, I turned to the JavaMail API. I've used this before, but mostly just for sending messages over SMTP. Surprisingly, use with Gmail is even listed in the JavaMail API FAQ.

Connecting was quite simple. The only property that I needed to pass into Session.getInstance(…) was a value of "imaps" for "mail.store.protocol". For simplicity, I then used the Store's connect(String host, String user, String password) method to connect.

To open the "All Mail" folder in Gmail, either .getFolder("[Gmail]").getFolder("All Mail") or .getFolder("[Gmail]/"All Mail") works.

I did my transfer in two separate steps - the download than the upload. This was mostly because I also wanted a local, backup copy to keep. By iterating through each Message from the Folder's getMessages(), the Message's writeTo(OutputStream) method allows for easily saving the entire message - including headers and attachments - in MIME format. Similarly, the messages can then be recreated using the MimeMessage(Session, InputStream) constructor, then imported into an IMAP folder using the appendMessages(Message[]) method.

Alternatively - though I haven't tried it - opening a session to each account and using the Folder's copyMessages(Message[], Folder) method looks like it would also be convenient. Note that the "this" Folder is the source, and the Folder parameter is the destination. However, while the method is convenient for copying multiple messages at once, it may need to be coded for only one message at a time, so a pause can be inserted between transfers to account for the Gmail restrictions described above.

To slow down the transfers to keep the Gmail servers happy, after each download or upload I simply called Thread.sleep(long millis). I started out at 2,500 ms, but had this down to 500 ms by the time I finished without any further issues.

Sent Mail

Another issue I had was that I wanted/needed to re-populate my "Sent Mail" folder. This may not always be an issue, but due to any number of reasons in my case, Gmail no longer recognized any of my sent messages as "sent". While Gmail seems to treat "Sent Mail" just like any other "label", the Gmail interface doesn't provide any options to add or remove messages from it, short of deletion. However, it can be modified through IMAP. While copying in each new message, I checked to see if the getSender() method matched one of email addresses. If so, it should be copied into the "Sent Mail" folder rather than "All Items". Note the difference in using the sender rather than the "From" attribute from getFrom(). The later would typically include items that were actually received, probably from an automated sender, where it was only made to appear "from" the same email address.

Originally, I simply imported all mail into the "All Mail" folder, then copied it into "Sent Mail" as needed. However, passing the same MimeMessage into Folder.appendMessages(Message[]) then again to copyMessages(Message[], Folder) for "Sent Mail" results in a ClassCastException as copyMessages expects a IMAPMessage variant of a message that already exists in the IMAP store. The solution would be to obtain a proper reference to the inserted message to copy. The Message[] com.sun.mail.imap.IMAPFolder#addMessages(Message[]) method would allow for exactly this, but it appears that Gmail doesn't currently support the required UIDPLUS extension from RFC 2359. The only apparent remaining option would be to recall getMessages(), then iterate through and find the message matching the one inserted, probably by getMessageID().

Even after all this work, I still ran into an issue with the Gmail web interface showing all my messages in my "Sent Items" as "To: me", rather than the actual recipients. The issue wasn't related to anything with the transfer, but in my Gmail account settings. I had assumed that Gmail calculated "me" as mail sent to any address listed in the "Send mail as" list. Instead - and probably more reasonable - it is almost the opposite. If the email is sent to any address not listed in the "Send mail as" list, it is assumed by Gmail to be "To: me".

Other tools

After I successfully finished my migration using the above method, I did find some other tools that appear to solve many of the same issues. 2 that I found off of related Google searches are imapsync and IMAPSize. (I've not downloaded or tried either of these.)