Most of this was written by Myk, but since he doesn’t have a blog, I’m paraphrasing it and posting it here.
Our firewall upgrade at the Meer colocation facility this evening was successful. The upgrade caused 23 seconds of downtime while we switched firewalls and several minutes of slowness until we fixed a misconfigured firewall parameter. The new firewall (plus redundant failover firewall) is now up and running, and normal access to all services is restored.
We’ve been in the process of moving from one facility to another, and for the interim, there is a direct ethernet link between the two facilities, but behind our firewall. There are only 6 machines left at the old facility currently, and until tonight our traffic was still being routed there (and then across the cross-colo link to get to the other 50 or so servers at the new facility). The new firewalls we upgraded to are located at the new facility, so our Internet traffic now goes directly there, and only those 6 machines are still depending on the cross-colo link now.
Many thanks go to Alex Polvi, Joe Rhett (of meer), Myk Melez, and Chase Philips for their excellent work making the upgrade happen!
We’re going to upgrade the firewall in our meer.net colocation facility this evening (Friday) at 6pm PDT (Sat 1am UTC). The upgrade affects many services, including Bugzilla, Bonsai, Tinderbox, LXR, CVS, mozilla.org mail, mozillafoundation.org mail, devmo, the wikis, IRC, and the mailing lists. We expect no more than a few seconds of downtime, but we’ll work for up to an hour to resolve unexpected problems before rolling back to the current firewall, so there may be up to an hour of downtime.
If you have any problems trying to reach any mozilla.org services after the changeover has occured, try first to find us on IRC at irc.mozilla.org #bmo, or if the IRC server is unreachable, mail me at firstname.lastname@example.org (which isn’t handled on mozilla.org’s network).