Multiple outages planned June 20-24

The Mozilla Foundation has had much of its server infrastructure hosted by Meer.net since we left AOL. Meer.net has set up a new colocation facility a few miles away from the one we were originally using, which is a much nicer facility with better access control and server racks that are much easier to work with. We’ve been in a “transition” phase for a few months, with some servers hosted in each facility, moving a few things here and there to the new facility as time and opportunity allows. We’re now down to the final servers to be moved, which have been getting put off because they’re all the end-user-facing servers running the various websites and developer services, and taking those down to move them means impacting end-user services.

But even those have to be moved sometime, and we’ll be attempting to move them during the week of June 20 to 24. During that week, there will be sporadic outages of various services, anywhere from a few minutes to a few hours each, as various services or servers get moved.

The following public-facing services will be affected:

Bonsai,
Bugzilla,
Despot,
Hendrix,
LXR,
Tinderbox
CVS,
CVS-mirror,
CVS-www,
IRC
mozilla.org email and mailing lists, mozillafoundation.org email
www.mozilla.org (2 of the 3 servers)
www.bugzilla.org
developer.mozilla.org (“Devmo”) (not developer-test)
wiki.mozilla.org,
reporter.mozilla.org,
planet.mozilla.org
ftp-staging
primary DNS service for almost every domain we own
Talkback services

We have lots of new servers. The plan is to rack a bunch of the new servers in the new facility and start moving services over by staging them on the new machines, then switching DNS at the point the new machines are ready to take over. This can be done with minimal downtime on almost all of the above services except for Talkback, which will probably need to be shut down and moved in the original servers, because as far as I know, it’s a pain in the butt to set up, and nobody wants to do it all over again 🙂

Individual outages will be announced on nagios (I’ll put a link in the middle of the main page) somewhere between a couple hours and a day in advance of each outage.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.