Category: Work

If you didn’t already read it elsewhere, we’re going to be changing the news.mozilla.org domain name to point at the Giganews servers in the morning (PST) on Monday, January 9th. The message pointers are different, so if you read the newsgroups via NNTP (rather than email), you may need to unsubscribe and resubscribe from each of the groups after the change is made in order to get your “last read” pointers synced up. If all of the new posts start showing up as already-read to you, this is likely what you’ve run into.

For more information, see the News Migration FAQ.

I’ve been sort of out of touch with the world outside of my email and IRC since returning from the Firefox Summit last week. I finally poked my head out and looked around tonight (read: I fired up my RSS client for the first time since returning home), and I must say, it really looks like things are in full swing for Mozilla, and for the Mozilla sysadmins in particular, now. 🙂

It’s probably no secret that I was the only sysadmin for Mozilla’s network for quite a while, and was starting to feel pretty buried. Over the last couple months, we’ve slowly been picking up additional staff, and with Justin officially coming on board last week, we finally have a large enough team (and a dedicated manager) to have some hope of keeping up with things. 🙂 We still have a HUGE backlog on the server ops buglist, but we’ve made tremendous progress in the last couple weeks, and it’s looking quite possible that we might just get caught up with the backlog by spring (which is good considering I previously had no hope of ever catching up 😉 ). Knowing how things run around here, I’m sure there’ll be more thrown at us, but at least we won’t still be sitting on bugs that were opened in 1998.

The Firefox 1.5 release was quite a bit of excitement as well, just in the fact that our servers kept up with it. It was so cool to see all the hard work from the last 9 months or so of beefing up our infrastructure paying off in such big ways. If you haven’t already read it, go read Polvi’s blog from a couple days ago. He sums it up good enough that I don’t feel the need to repeat it (both about the release and our general sysadmin attitude these days).

On Friday, December 9th, we will be migrating the remaining six machines at the Karlstad collocation facility to our Richard collocation facility.

The move affects many services/websites, including CVS, wiki.mozilla.org, mail.mozilla.org, Nagios, Tinderbox, Bonsai (full list at the end of this note). We will do everything possible to minimize downtime and impact to users, but we anticipate some effect when we physically move the servers. During the outage window, we will also be rebooting several of the servers in order to install some pending upgrades.

We will start this work at 2pm PST (10pm UTC) and will be done by 5pm PST (1am UTC Saturday). Expect sporadic access to any of the listed services during this time.

Once complete we will be performing a full suite of tests to ensure stability and functionality. We will send out an update when we have completed the work. If you see any issues related to the move after 5pm, please email sysadmins@mozilla.org or justin@mozilla.com, or comment on the #bmo channel on irc.mozilla.org.

Impacted Services:

bittorrent.mozilla.org
bonsai.mozilla.org
bonsai-l10n.mozilla.org
build-graphs.mozilla.org
cvs.mozilla.org
cvs-mirror.mozilla.org
cvs-www.mozilla.org
despot.mozilla.org
doctor.mozilla.org
hendrix.mozilla.org
litmus.mozilla.org
lxr.mozilla.org
mail.mozilla.org
nagios.mozilla.org
planet.mozilla.org
reporter.mozilla.org
testrunner.mozilla.org
tinderbox.mozilla.org
webtools.mozilla.org
wiki.mozilla.org

On Friday, November 25th, we will be cutting over to new faster Internet uplinks and new firewalls in the Meer collocation facility. This is to increase security and capacity to support our core infrastructure.

The upgrade affects many services, including Bugzilla, Bonsai, Tinderbox, LXR, CVS, mozilla.org mail, mozillafoundation.org mail, Mozilla Developer Center, the wikis, IRC, the Report a Broken Website tool, and the mailing lists. We will do everything possible to minimize downtime and impact to users, but we anticipate some effect when we do the final cutover and failover testing. During the outage window, we will also be rebooting several of the servers in order to install some pending upgrades.

We will start this work at 5pm PST (1am UTC Saturday) and will be done by 7pm PST (3am UTC). Expect sporadic access to any Internet facing infrastructure during this time.

Once complete we will be performing a full suite of tests to ensure stability and functionality. We will send out an update when we have completed the work. If you see any issues relating to the upgrade after 7pm, please email sysadmins@mozilla.org or justin@mozilla.com, or comment on the #bmo channel on irc.mozilla.org.

It was discovered after we upgraded bugzilla.mozilla.org to version 2.20 that some of the architectural changes in how Bugzilla talks to the database servers had quite by accident sideswiped the ability to run searches against a replicated database server, causing those searches to always run against the master. The net effect on Bugzilla is that on large installations (such as ours), Bugzilla would periodically lock up for a bit because of having to wait for database locks.

I applied the patch from bug 313695 to correct this problem to bugzilla.mozilla.org about 3 hours ago, and so far things are looking pretty good. I haven’t seen any freezeups yet since then, and things are staying pretty snappy. On the other hand, we’re also right at the bottom end of our off-peak time of day for traffic levels.

I also bumped up the send timeout in apache to match what we had before we moved to the new server a few months back (bug 299577). This allows some longer-running searches (and sending of bugmail on bugs with lots of CCs) to actually complete.

If you notice any new problems with Bugzilla as of this evening, come find me on #bmo on irc.mozilla.org.

For any non-Mozilla Bugzilla folks reading this, yes, this fix will be included in 2.20.1. 🙂