I’ve been sort of out of touch with the world outside of my email and IRC since returning from the Firefox Summit last week. I finally poked my head out and looked around tonight (read: I fired up my RSS client for the first time since returning home), and I must say, it really looks like things are in full swing for Mozilla, and for the Mozilla sysadmins in particular, now.
It’s probably no secret that I was the only sysadmin for Mozilla’s network for quite a while, and was starting to feel pretty buried. Over the last couple months, we’ve slowly been picking up additional staff, and with Justin officially coming on board last week, we finally have a large enough team (and a dedicated manager) to have some hope of keeping up with things. We still have a HUGE backlog on the server ops buglist, but we’ve made tremendous progress in the last couple weeks, and it’s looking quite possible that we might just get caught up with the backlog by spring (which is good considering I previously had no hope of ever catching up ). Knowing how things run around here, I’m sure there’ll be more thrown at us, but at least we won’t still be sitting on bugs that were opened in 1998.
The Firefox 1.5 release was quite a bit of excitement as well, just in the fact that our servers kept up with it. It was so cool to see all the hard work from the last 9 months or so of beefing up our infrastructure paying off in such big ways. If you haven’t already read it, go read Polvi’s blog from a couple days ago. He sums it up good enough that I don’t feel the need to repeat it (both about the release and our general sysadmin attitude these days).
On Friday, December 9th, we will be migrating the remaining six machines at the Karlstad collocation facility to our Richard collocation facility.
The move affects many services/websites, including CVS, wiki.mozilla.org, mail.mozilla.org, Nagios, Tinderbox, Bonsai (full list at the end of this note). We will do everything possible to minimize downtime and impact to users, but we anticipate some effect when we physically move the servers. During the outage window, we will also be rebooting several of the servers in order to install some pending upgrades.
We will start this work at 2pm PST (10pm UTC) and will be done by 5pm PST (1am UTC Saturday). Expect sporadic access to any of the listed services during this time.
Once complete we will be performing a full suite of tests to ensure stability and functionality. We will send out an update when we have completed the work. If you see any issues related to the move after 5pm, please email firstname.lastname@example.org or email@example.com, or comment on the #bmo channel on irc.mozilla.org.