On Friday, November 25th, we will be cutting over to new faster Internet uplinks and new firewalls in the Meer collocation facility. This is to increase security and capacity to support our core infrastructure.
The upgrade affects many services, including Bugzilla, Bonsai, Tinderbox, LXR, CVS, mozilla.org mail, mozillafoundation.org mail, Mozilla Developer Center, the wikis, IRC, the Report a Broken Website tool, and the mailing lists. We will do everything possible to minimize downtime and impact to users, but we anticipate some effect when we do the final cutover and failover testing. During the outage window, we will also be rebooting several of the servers in order to install some pending upgrades.
We will start this work at 5pm PST (1am UTC Saturday) and will be done by 7pm PST (3am UTC). Expect sporadic access to any Internet facing infrastructure during this time.
Once complete we will be performing a full suite of tests to ensure stability and functionality. We will send out an update when we have completed the work. If you see any issues relating to the upgrade after 7pm, please email firstname.lastname@example.org or email@example.com, or comment on the #bmo channel on irc.mozilla.org.
It was discovered after we upgraded bugzilla.mozilla.org to version 2.20 that some of the architectural changes in how Bugzilla talks to the database servers had quite by accident sideswiped the ability to run searches against a replicated database server, causing those searches to always run against the master. The net effect on Bugzilla is that on large installations (such as ours), Bugzilla would periodically lock up for a bit because of having to wait for database locks.
I applied the patch from bug 313695 to correct this problem to bugzilla.mozilla.org about 3 hours ago, and so far things are looking pretty good. I haven’t seen any freezeups yet since then, and things are staying pretty snappy. On the other hand, we’re also right at the bottom end of our off-peak time of day for traffic levels.
I also bumped up the send timeout in apache to match what we had before we moved to the new server a few months back (bug 299577). This allows some longer-running searches (and sending of bugmail on bugs with lots of CCs) to actually complete.
If you notice any new problems with Bugzilla as of this evening, come find me on #bmo on irc.mozilla.org.
For any non-Mozilla Bugzilla folks reading this, yes, this fix will be included in 2.20.1. 🙂
…to both me and to Firefox. 🙂
One year ago yesterday, I joined the Mozilla Foundation as a paid sysadmin, just in time to wrestle with the server overload as we released Firefox 1.0 the following day. It was pretty cool to see Firefox 1.0 get released on my birthday last year. Now both Firefox and I are another year older.
Here’s to another great year!