Saturday, July 26, 2008

Gmail migration with IMAP and the Java Mail API

As I previously posted, Gmail has IMAP support. I've been using it for a while now. It's a great way to use Gmail with a a desktop email client such as Mozilla Thunderbird, as well as having offline access to your email.

IMAP also offers another significant feature to Gmail users: The ability to easily migrate messages between accounts - either between different Gmail accounts or from another source into Gmail. Google offers their own tool for this, but it's only available for commercial and educational accounts - not for their personal accounts like the ones I have.

I attempted to run my transfer using the configured accounts in Microsoft Outlook, very much like Ashish Mohta described on his blog. Things were looking good until I started getting the following error:

The current command did not succeed.
The mail server responded: Unable to append message to folder (Failure).

The first thing I suspected was an Outlook issue, so I switched to Thunderbird but got the same result. I suspected I may be hitting some sort of transfer limit, so I tried downloading everything from my first account to a local folder first. That worked, but then I still wasn't able to upload from the local folder into the second account. A Google search shows a number of other people having the same issue. Google help has two (1, 2) answers that primarily blame formatting incompatibilities, but as indicated by by other results in the Google search, this definitely appears to be the result of exceeding some unposted transfer limits.

There didn't seem to be any way to configure either Outlook or Firebird to "slow things down", without manually moving only a few messages at a time - which would be tedious and prone to error. Additionally, I guessed that both clients were performing additional tasks behind the scenes that were making the issue worse, such as needlessly refreshing folders.

JavaMail API

I figured that my best chance of getting everything how I wanted it was to script the transfer. For this, I turned to the JavaMail API. I've used this before, but mostly just for sending messages over SMTP. Surprisingly, use with Gmail is even listed in the JavaMail API FAQ.

Connecting was quite simple. The only property that I needed to pass into Session.getInstance(…) was a value of "imaps" for "mail.store.protocol". For simplicity, I then used the Store's connect(String host, String user, String password) method to connect.

To open the "All Mail" folder in Gmail, either .getFolder("[Gmail]").getFolder("All Mail") or .getFolder("[Gmail]/"All Mail") works.

I did my transfer in two separate steps - the download than the upload. This was mostly because I also wanted a local, backup copy to keep. By iterating through each Message from the Folder's getMessages(), the Message's writeTo(OutputStream) method allows for easily saving the entire message - including headers and attachments - in MIME format. Similarly, the messages can then be recreated using the MimeMessage(Session, InputStream) constructor, then imported into an IMAP folder using the appendMessages(Message[]) method.

Alternatively - though I haven't tried it - opening a session to each account and using the Folder's copyMessages(Message[], Folder) method looks like it would also be convenient. Note that the "this" Folder is the source, and the Folder parameter is the destination. However, while the method is convenient for copying multiple messages at once, it may need to be coded for only one message at a time, so a pause can be inserted between transfers to account for the Gmail restrictions described above.

To slow down the transfers to keep the Gmail servers happy, after each download or upload I simply called Thread.sleep(long millis). I started out at 2,500 ms, but had this down to 500 ms by the time I finished without any further issues.

Sent Mail

Another issue I had was that I wanted/needed to re-populate my "Sent Mail" folder. This may not always be an issue, but due to any number of reasons in my case, Gmail no longer recognized any of my sent messages as "sent". While Gmail seems to treat "Sent Mail" just like any other "label", the Gmail interface doesn't provide any options to add or remove messages from it, short of deletion. However, it can be modified through IMAP. While copying in each new message, I checked to see if the getSender() method matched one of email addresses. If so, it should be copied into the "Sent Mail" folder rather than "All Items". Note the difference in using the sender rather than the "From" attribute from getFrom(). The later would typically include items that were actually received, probably from an automated sender, where it was only made to appear "from" the same email address.

Originally, I simply imported all mail into the "All Mail" folder, then copied it into "Sent Mail" as needed. However, passing the same MimeMessage into Folder.appendMessages(Message[]) then again to copyMessages(Message[], Folder) for "Sent Mail" results in a ClassCastException as copyMessages expects a IMAPMessage variant of a message that already exists in the IMAP store. The solution would be to obtain a proper reference to the inserted message to copy. The Message[] com.sun.mail.imap.IMAPFolder#addMessages(Message[]) method would allow for exactly this, but it appears that Gmail doesn't currently support the required UIDPLUS extension from RFC 2359. The only apparent remaining option would be to recall getMessages(), then iterate through and find the message matching the one inserted, probably by getMessageID().

Even after all this work, I still ran into an issue with the Gmail web interface showing all my messages in my "Sent Items" as "To: me", rather than the actual recipients. The issue wasn't related to anything with the transfer, but in my Gmail account settings. I had assumed that Gmail calculated "me" as mail sent to any address listed in the "Send mail as" list. Instead - and probably more reasonable - it is almost the opposite. If the email is sent to any address not listed in the "Send mail as" list, it is assumed by Gmail to be "To: me".

Other tools

After I successfully finished my migration using the above method, I did find some other tools that appear to solve many of the same issues. 2 that I found off of related Google searches are imapsync and IMAPSize. (I've not downloaded or tried either of these.)

13 comments:

Marimuthu said...

hi Ziesemer,
you are doing good job, i have seen your blog that was fantastic..
I need one help from you that i am using smtp for sending mail to recipient using javamail ,it's working perfectly but i need to do send one hyper-link to recipient then when user click that link afterwords automatically his request has to activated...
please help me how to do this?

if you know how to do this send your answer to my mail id,
teknoturfian@gmail.com

thanks and regards,
P.Marimuthu,

Mark A. Ziesemer said...

Where are you having the issue? Sending the link, or handling the "activation click" (which shouldn't have anything to do with JavaMail)?

I'd rather help with this in a public forum for everyone to benefit. I'd recommend re-posting your request with some additional detail on the JavaMail forum at forums.sun.com. Feel free to add another comment here with the link for me and/or others to follow.

Andres said...

Hi Mark.

Very helpful report. I recently completed the migration of my 10 year worth email archive over IMAP, and run into the same issue you mention here:

"I still ran into an issue with the Gmail web interface showing all my messages in my "Sent Items" as "To: me", rather than the actual recipients."

I haven´t found much information online regarding this issue. Since some of my older email accounts no longer exist, it is impossible to validate these identities in Gmail (and unvalidated identities don´t seem to solve the "To: me" issue). Are we aware of any solution or workaround to this issue? Thanks!

Mark A. Ziesemer said...

Andres - unfortunately the only information I have is the information I've posted here. Maybe try contacting Google to see if this is something they can help with or provide a solution for?

gaurav said...
This comment has been removed by a blog administrator.
gaurav said...
This comment has been removed by a blog administrator.
gaurav said...

Thanks , for providing a worthily information.
I have tried this method for moving mails from one user account to another . But now i am facing a problem, I have used Folder.appendMessages(Message []) function and after this line I have typed a message on console as
System.out.println("Mesage transmitted successfully") so my program is coming on this line as no exception is generated but the mails are not transferred . can u please provide any suggestion for this.

If u know answer send me at -
gauravjain117@gmail.com

Mark A. Ziesemer said...

gaurav - I or others would need to see more of your code. Please refer to the 2nd half of my above comment on 2008-12-24.

hi said...

hi Ziesemer,
nice to see your reply . I am having two folder objects containing names of folders of mailboxes and for second folder I am writing code as folder2.appendMessages(msgs) and its not generating any exception but still mails are not moved.

If u know answer then you can mail me at gauravjain117@gmail.com

gaurav said...
This comment has been removed by a blog administrator.
Nirav Thaker said...

This post has been quite useful, Thanks!

sammefford said...

Thanks for the post, Mark, the most helpful thing for me was the link to imapsync. I too resorted to JavaMail, but I used it in combination with Spring Integration which I hoped would do much of the heavy lifting for me. All was well getting my messages into MySQL so I could find out which messages had not been migrated yet (avoiding migrating duplicates because of past partially successful migrations). Then when I tried pulling the non-duplicate messages out to migrate them one at a time, I found JavaMail fails against exchange to pull the messages out one at a time by the criteria I tried: date, subject. Plus JavaMail didn't easily give me the Message-ID header, which was disappointing.

imapsync appears to be working for the most part. It avoids duplicates (and is the first tool I've found which does this--thank goodness given how many partially successful transfers I've had). I got imapsync compiled and running in cygwin on WinXP. And I migrated several folders with pretty good success, but my 30K messages in my Inbox have been a nightmare, I think because imapsync tries to bite off all the messages at once. The only way I've gotten it to work is with --minage N --maxage N+50 so I can bite off smaller chunks of my mailbox at a time.

Robottinosino said...

A helpful and well-written post.