The hardware behind

I recently did up a diagram of how our Bugzilla site was set up, mostly for the benefit of other sysadmins trying to find the various pieces of it.  Several folks expressed interest in sharing it with the community just to show an example of how we were set up.  So I cleaned it up a little, and here it is:

Bugzilla Physical Layout Diagram
Click the image for a full-size (readable) version

At first glance it looks somewhat excessive just for a Bugzilla, but since the Mozilla Project lives and dies by the content of this site, all work pretty much stops if it doesn’t work, so it’s one of our highest-priority sites to keep operating at all times for developer support.  The actual hardware required to run the site at full capacity for the amount of users we get hitting it is a little less than half of what’s shown in the diagram.

We have the entire site set up in two different datacenters (SJC1 is our San Jose datacenter, PHX1 is our Phoenix datacenter).  Thanks to the load balancers taking care of the cross-datacenter connections for the master databases, it’s actually possible to run it from both sites concurrently to split the load.  But because of the amount of traffic Bugzilla does to the master databases, and the latency in connection setup over that distance, it’s a little bit slow from whichever datacenter isn’t currently hosting the master, so we’ve been trying to keep DNS pointed at just one of them to keep it speedy.

This still works great as a hot failover, though, which got tested in action this last Sunday when we had a system board failure on the master database server in Phoenix.  Failing the entire site over to San Jose took only minutes, and the tech from HP showed up to swap the system board 4 hours later.  The fun part was that I had only finished setting up this hot failover setup about a week prior, so the timing couldn’t have been any better for that system board failure.  If it had happened any sooner we might have been down for a long time waiting for the server to get fixed.

When everything is operational, we’re trying to keep it primarily hosted in Phoenix.  As you can see in the diagram, the database servers in Phoenix are using solid-state disks for the database storage. The speed improvement when running large queries that is gained by using these instead of traditional spinning disks is just amazing.  I haven’t done any actual timing to get hard facts on that, but the difference is large enough that you can easily notice it just from using the site.

Upgrading to version 3.4.3

We’re finally at the point where I can say we’re ready to upgrade Bugzilla @ Mozilla this weekend.  We’re aiming for Sunday evening (probably 6pm PST).  I’ll post again when I know how long it’ll be down for (and that’ll be included in the eventual downtime notice on the IT blog as well).

There’s a staging copy set up at and I would appreciate people playing around with it and finding anything that might be broken before we get it to production.  Before filing bugs, make sure to check the detailed status linked from the red box at the top of every page to make sure it’s not already listed (and you can also see my progress on cosmetic issues and so forth, there).

It will be down for a while at some point tonight when I reload it with an up-to-date snapshot of the production server (and that’ll be my test to find out how long it’ll take to upgrade it, too).  I’m super excited because this has been a long time coming. 🙂

Bugzilla news feed now available

So, thanks to Fowl’s comment in my previous entry, I tried out setting up the Google news/blogsearch keyword feeds for Bugzilla in Google Reader, and then individually sharing each of the entries that were actually about Bugzilla, and then using the shared items feed from Google Reader as the newsfeed.  It turns out to work quite well.

The built-in RSS widget that ships with WordPress completely chokes on Google Reader’s shared items feed though, and I ended up installing the SimplePie plugin for WordPress to handle the feed parsing (thanks to srikat on #wordpress for suggesting that).  SimplePie turns out to have a much more flexible template system, and it looks pretty good.  If you’re viewing this on my website, the results are over there on the right.

The feed icon in the title is linked to the source feed, so you can use that as a subscription URL if you want to subscribe to my sanitized Bugzilla news feed.  Now you can keep up on all the Bugzilla mentions in the news and blogosphere without having to wade through all the exterminator jokes, monster VW Beetles, and mentions of bug reports in every project’s bugzilla site.

Bugzilla in the news

Dear LazyWeb…   🙂

I like to know how Bugzilla’s being talked about out in the world, so I subscribe to Google’s handy news and blog alert services where you can put in keywords and they’ll send you an email every time a new blog or news article shows up containing that/those keywords.  This works really well, except that you also get a bunch of crap with it.  People who post to message boards with “Bugzilla” as a username.  Message posts on insect websites about this huge bug someone found.  Articles about modified VW Beetles that have been turned into monster trucks.  Every mention in anyone’s blog about bug reports for any number of major projects that use Bugzilla for their bug trackers.  Etc Etc Etc.

I’ve long desired to create an RSS feed where I could collect that and republish all those links without all the cruft in it, i.e. just the stuff that was actually about Bugzilla as a product/project.  I really like Jon Gruber’s feed, where he puts the link and the title and a small blurb of his own comments about it.  Note that the linked title goes to the article being talked about, not an entry on his blog, which is the key feature I’m looking for (and what makes it different than just creating a category for it in the blog).  Something just like that would be awesome if there were a plugin for WordPress that would do it.  I’ve searched on and off and never found anything that would do this.  If anyone knows of one, please let me know in the comments here. update

On Friday, I pushed a small update to that fixed bug 452799, where users who didn’t have ‘canconfirm’ privs in Bugzilla were posting bugs that had a status of NEW rather than UNCONFIRMED.

This morning, I pushed an update to containing a plethora of additional fixes to address concerns raised since the Bugzilla upgrade.  This morning, we’ve picked up fixes for:

  • Bug 452793: (The other half of the issue which was fixed Friday) The default status selected when you file a new bug and do have ‘canconfirm’ privs is now NEW instead of UNCONFIRMED.
  • Bug 452810: The wording surrounding the checkbox to add youself to the CC now says “Add me to the CC list” when you aren’t on it, instead of just “myself.”
  • Bug 452734: The keyword chooser has been replaced with keyword autocomplete.  NOTE: If you installed the greasemonkey script to remove the keyword chooser, you’ll probably have to remove that script to get the autocomplete, since it hooks on the same event listener.
  • Bug 452798: The CC list is now visible again by default, and as a bonus, it’s now searchable via Firefox’s find-as-you-type feature.
  • Bug 452733: The [Classification] is no longer shown in front of the bug summary.
  • Bug 452746: The link to the bug in the header no longer contains an extra space.
  • Bug 452891: The “visually jarring” dashed border next to the line numbers in the Diff Viewer has been removed.
  • Bug 452749: The midair page once again specifies who you midaired with.
  • Bug 344559: Add a Commit button near the form fields at the top of the show_bug page so you don’t have to scroll to the bottom of the comments if you’re only changing a field at the top.

Fixes for admins:

  • Bug 452898: Milestones can once again be marked inactive.
  • Bug 452914: Multiple problems were fixed in the flag editor related to the “fixed in version” field not being dealt with correctly on a product change.

Hopefully this fixes up some of the more major concerns people had.  There’s still more to come.  At this point I’m plannng on daily pushes to production as the fixes become available.

UPDATE: Some people are reporting broken CSS and things looking strange…  hold the Shift key and hit Reload if that’s you.  Your browser is probably caching the old CSS.