Jul 10 2010

Migrating From Zimbra to Google: How We’re Doing It (Part 1 – Notifications)

Published by at 1:51 pm under Gmail,Google Apps,Technical details

A little while back I wrote up a short evaluation of the Google Migration Tool – highlighting the pros and cons I found. After a ton of iterations, I think we may have stumbled upon success. (Knowing my luck the process will now fail miserably because of this post).

The migration process is comprised of 3 main components:

  • End-user notifications
  • Folder renaming
  • Migrating mail

The notification and folder renaming processes are pretty straightforward. The actual migrating of mail is rather complex and I’ll cover the details to that in part 3.

Since Brandeis has chosen to take a rolling migration approach, we have the luxury of moving only a select number of users at a time. When we made the move to Bmail, we found that moving a single department at a time worked well. At least with this method, we can catch a set of users who work in close proximity to each other and can look to one another for some first tier support.

The secret to moving departments on a one-by-one basis is to first create a master spreadsheet containing all users who have a Bmail account, the department, Brandeis affiliation, the number of messages in each account, the number of folders as well as number of Google illegal folders, and the size of the mailbox (we have no quota so this # is important). Sure creating this spreadsheet only captures numbers at a moment in time but it at least gets us a rough idea of how large a user’s mailbox is or how many e-mails they have.

Next we choose a department and work with them on selecting a migration date. Not knowing any one department’s work schedule, this coordination has proven invaluable. It also helps to catch the one off cases where a user may be traveling and having to change to a new e-mail system while on the road would cause some confusion.

Once the department or group has agreed upon a date, a CSV list of their Brandeis UNet IDs is uploaded to our Google Migration Application. Doing so makes the following things happen:

  1. Once uploaded, key dates are calculated.
    Dates from the Google MIgration Application

    Dates from the Google MIgration Application

    1. Add Date – the date they are added to the application.
    2. Reminder Date – 2 business days before they are scheduled to move
    3. Migrate Date – the actual day the migration will take place
    4. Close Date – the date their Bmail account will close
  2. A kick-off notification is sent at 4 PM on the Add Date. This e-mail contains general information about the move, links to documentation, as well as a pre-migration checklist of things to do.
  3. On the evening of the Add Date, a separate process is run which checks a user’s folders for length, illegal characters, Google specific system labels, and folders with 2 or more adjacent spaces. If any are found, a notification is sent to the user highlighting each of these folders and why they will not migrate to Google. (More on this process will be covered in Part 2).
  4. On the Reminder Date, a reminder e-mail (surprise!) is sent to the users at 9:30 AM. This is more a less the same e-mail as the kick-off notification, with some slightly different verbiage since the migration is only 2 business days away.
  5. At 11 PM on the Reminder Date, the folder check process is run again, basically repeating the same e-mail as before. However, if the user fixed all folders from the initial notification, a ‘Thanks for renaming your folders!’ e-mail is sent instead.
  6. At 5 PM on the Migrate Date, the folder check process is run last time, only this time it renames any folders which are still illegal. An e-mail with each folder, before and after being renamed, is sent to the user.
  7. At 5:30 PM on the Migrate Date, the migration begins. We do some magic to calculate the number of e-mails a user has and provide them a rough estimate of how long the migration will take. Last I checked we assume a rate of 2,000 e-mails per hour – so if you have 4,000 e-mails the notification will state 2 – 3 hours to complete.
  8. During the migration, the user may be notified of e-mails which couldn’t be migrated to Google. If this is the case, they will be sent an e-mail with a link to download all e-mails in a folder which could not be migrated, along with the date, subject, from address, and message ID of each individual e-mail which did not get migrated.
  9. Once the migration is complete, a final notification is sent with the next steps – most importantly how to point your mail clients or mobile device to Google instead of Bmail.
  10. No notifications are sent on the Close Date, rather, the Bmail account is silently closed via an automated process.

All notifications and their subject

All notifications are stored in our Google Migration App, making it easy to see who got notified and when. The app also allows users to be contacted directly from the interface, again storing the notification in the database.

Notifications are all made possible by:

  • Turbogears, the Python based web framework which the Google Migration Application was written in
  • Cron to automate and schedule when the notifications run
  • MySQL which stores all the users, dates, notifications, etc. from the Google Migration Application
  • Python + MySQLdb and smtplib (and other misc. modules) which all the scripts are written in

In Part 2 I will explore in a little more detail the folder renaming process, which like everything else, has undergone a few iterations before we got it ‘right’. Finally, in Part 3, when I have a few *free* hours, I’ll document the actual mail migration process.

5 responses so far

5 Responses to “Migrating From Zimbra to Google: How We’re Doing It (Part 1 – Notifications)”

  1. Thangarajon 18 Jul 2010 at 3:59 am

    Hello,

    We’re in the process of migrating our own zimbra (4) system to google apps – education edition. Would be great if you could share some of your migration scripts 🙂

  2. Thangarajon 19 Jul 2010 at 9:38 am

    Also, https://sites.google.com/a/brandeis.edu/googleapps/pre-migration#TOC-Email-Bmail-to-Gmail- is not available to the public. Does it have private data or can you make it public?

    Thanks!

  3. Kevin Gibsonon 31 Jan 2011 at 6:19 pm

    We are also in the process of migrating from zimbra to Google Apps. Are you willing to share your scripts that check folder length, etc?

  4. Nick Ragusaon 01 Feb 2011 at 9:51 pm

    Hi Kevin – I posted some of the important functions from the migration app here. Hope this helps!

    Nick

  5. Kevin Gibsonon 02 Feb 2011 at 12:00 pm

    Thank you not sure how I missed that or maybe it wasn’t public until now. I really appreciate it.

    kg.

Protected by Akismet
Blog with WordPress

Welcome Guest | Login (Brandeis Members Only)