The hardware behind

I recently did up a diagram of how our Bugzilla site was set up, mostly for the benefit of other sysadmins trying to find the various pieces of it.  Several folks expressed interest in sharing it with the community just to show an example of how we were set up.  So I cleaned it up a little, and here it is:

Bugzilla Physical Layout Diagram
Click the image for a full-size (readable) version

At first glance it looks somewhat excessive just for a Bugzilla, but since the Mozilla Project lives and dies by the content of this site, all work pretty much stops if it doesn’t work, so it’s one of our highest-priority sites to keep operating at all times for developer support.  The actual hardware required to run the site at full capacity for the amount of users we get hitting it is a little less than half of what’s shown in the diagram.

We have the entire site set up in two different datacenters (SJC1 is our San Jose datacenter, PHX1 is our Phoenix datacenter).  Thanks to the load balancers taking care of the cross-datacenter connections for the master databases, it’s actually possible to run it from both sites concurrently to split the load.  But because of the amount of traffic Bugzilla does to the master databases, and the latency in connection setup over that distance, it’s a little bit slow from whichever datacenter isn’t currently hosting the master, so we’ve been trying to keep DNS pointed at just one of them to keep it speedy.

This still works great as a hot failover, though, which got tested in action this last Sunday when we had a system board failure on the master database server in Phoenix.  Failing the entire site over to San Jose took only minutes, and the tech from HP showed up to swap the system board 4 hours later.  The fun part was that I had only finished setting up this hot failover setup about a week prior, so the timing couldn’t have been any better for that system board failure.  If it had happened any sooner we might have been down for a long time waiting for the server to get fixed.

When everything is operational, we’re trying to keep it primarily hosted in Phoenix.  As you can see in the diagram, the database servers in Phoenix are using solid-state disks for the database storage. The speed improvement when running large queries that is gained by using these instead of traditional spinning disks is just amazing.  I haven’t done any actual timing to get hard facts on that, but the difference is large enough that you can easily notice it just from using the site.

Upgrading to version 3.4.3

We’re finally at the point where I can say we’re ready to upgrade Bugzilla @ Mozilla this weekend.  We’re aiming for Sunday evening (probably 6pm PST).  I’ll post again when I know how long it’ll be down for (and that’ll be included in the eventual downtime notice on the IT blog as well).

There’s a staging copy set up at and I would appreciate people playing around with it and finding anything that might be broken before we get it to production.  Before filing bugs, make sure to check the detailed status linked from the red box at the top of every page to make sure it’s not already listed (and you can also see my progress on cosmetic issues and so forth, there).

It will be down for a while at some point tonight when I reload it with an up-to-date snapshot of the production server (and that’ll be my test to find out how long it’ll take to upgrade it, too).  I’m super excited because this has been a long time coming. 🙂

Bugzilla news feed now available

So, thanks to Fowl’s comment in my previous entry, I tried out setting up the Google news/blogsearch keyword feeds for Bugzilla in Google Reader, and then individually sharing each of the entries that were actually about Bugzilla, and then using the shared items feed from Google Reader as the newsfeed.  It turns out to work quite well.

The built-in RSS widget that ships with WordPress completely chokes on Google Reader’s shared items feed though, and I ended up installing the SimplePie plugin for WordPress to handle the feed parsing (thanks to srikat on #wordpress for suggesting that).  SimplePie turns out to have a much more flexible template system, and it looks pretty good.  If you’re viewing this on my website, the results are over there on the right.

The feed icon in the title is linked to the source feed, so you can use that as a subscription URL if you want to subscribe to my sanitized Bugzilla news feed.  Now you can keep up on all the Bugzilla mentions in the news and blogosphere without having to wade through all the exterminator jokes, monster VW Beetles, and mentions of bug reports in every project’s bugzilla site.

Bugzilla in the news

Dear LazyWeb…   🙂

I like to know how Bugzilla’s being talked about out in the world, so I subscribe to Google’s handy news and blog alert services where you can put in keywords and they’ll send you an email every time a new blog or news article shows up containing that/those keywords.  This works really well, except that you also get a bunch of crap with it.  People who post to message boards with “Bugzilla” as a username.  Message posts on insect websites about this huge bug someone found.  Articles about modified VW Beetles that have been turned into monster trucks.  Every mention in anyone’s blog about bug reports for any number of major projects that use Bugzilla for their bug trackers.  Etc Etc Etc.

I’ve long desired to create an RSS feed where I could collect that and republish all those links without all the cruft in it, i.e. just the stuff that was actually about Bugzilla as a product/project.  I really like Jon Gruber’s feed, where he puts the link and the title and a small blurb of his own comments about it.  Note that the linked title goes to the article being talked about, not an entry on his blog, which is the key feature I’m looking for (and what makes it different than just creating a category for it in the blog).  Something just like that would be awesome if there were a plugin for WordPress that would do it.  I’ve searched on and off and never found anything that would do this.  If anyone knows of one, please let me know in the comments here. update

On Friday, I pushed a small update to that fixed bug 452799, where users who didn’t have ‘canconfirm’ privs in Bugzilla were posting bugs that had a status of NEW rather than UNCONFIRMED.

This morning, I pushed an update to containing a plethora of additional fixes to address concerns raised since the Bugzilla upgrade.  This morning, we’ve picked up fixes for:

  • Bug 452793: (The other half of the issue which was fixed Friday) The default status selected when you file a new bug and do have ‘canconfirm’ privs is now NEW instead of UNCONFIRMED.
  • Bug 452810: The wording surrounding the checkbox to add youself to the CC now says “Add me to the CC list” when you aren’t on it, instead of just “myself.”
  • Bug 452734: The keyword chooser has been replaced with keyword autocomplete.  NOTE: If you installed the greasemonkey script to remove the keyword chooser, you’ll probably have to remove that script to get the autocomplete, since it hooks on the same event listener.
  • Bug 452798: The CC list is now visible again by default, and as a bonus, it’s now searchable via Firefox’s find-as-you-type feature.
  • Bug 452733: The [Classification] is no longer shown in front of the bug summary.
  • Bug 452746: The link to the bug in the header no longer contains an extra space.
  • Bug 452891: The “visually jarring” dashed border next to the line numbers in the Diff Viewer has been removed.
  • Bug 452749: The midair page once again specifies who you midaired with.
  • Bug 344559: Add a Commit button near the form fields at the top of the show_bug page so you don’t have to scroll to the bottom of the comments if you’re only changing a field at the top.

Fixes for admins:

  • Bug 452898: Milestones can once again be marked inactive.
  • Bug 452914: Multiple problems were fixed in the flag editor related to the “fixed in version” field not being dealt with correctly on a product change.

Hopefully this fixes up some of the more major concerns people had.  There’s still more to come.  At this point I’m plannng on daily pushes to production as the fixes become available.

UPDATE: Some people are reporting broken CSS and things looking strange…  hold the Shift key and hit Reload if that’s you.  Your browser is probably caching the old CSS.

Please don’t shoot the Bugzilla devs

So got upgraded to Bugzilla 3.2 last night.  Since the upgrade, there’s been a lot of complaints about the new UI.

First off, given the differences in the way Mozilla uses Bugzilla compared to a lot of other places, some of these complaints are valid.  But, please try to be polite and state exactly why you think you have issues and suggest ways for improvement.  Don’t just run around saying it sucks or file bugs stating that you’re ticked off at the world because we broke your workflow.

One of the primary complaints Bugzilla as a product has received over the years is how the UI is ugly and hard to manage.  The last year or so the Bugzilla developers have been spending a lot of effort to fix that problem, with the assistance of professional UI designers.  Some of them are taking personal offense to some of the feedback we’ve gotten so far this morning about the UI changes because it makes them feel like all the work over the last year was for nothing if everyone just wants the old UI back.

Yes, in some cases, maybe you just have to suck it up and learn a new way to do things.  In others, there’s probably a lot of room for us to still clean things up.  In either case, please don’t burn the Bugzilla devs in effigy or anything. 🙂  Be kind on the bugs you file (but do file them).  Be constructive.  Don’t say “This and this are bad they way they are now, please put them back how they were.”  Do tell us “this is my usage case and what I need to do with Bugzilla, and here’s why the old way helped me be efficient doing this. Let’s come up with a way for it to be easy for me to do this again.”  In all honesty, I bet there’s use cases that weren’t thought of in the current design, and maybe it was just overlooked.  Give them the benefit of the doubt, and let us work with you to get something set up that makes your life easy again (maybe we’ll come up with something even better than both the old way and the current way, who knows?)

The major upgrade to 3.2 is done.  All the schema changes that took hours to run are in place.  Deploying changes to the UI at this point is just be the flip of a switch and it’ll just be live with no downtime at all, in most cases, so we can continue to tweak as we go over the next few weeks.  But please try not to get pissed at us and let us help fix it.  We really weren’t intentionally trying to break your world, you know. 🙂

What do you want to see while you wait?

On, when you run a search, if your browser supports “server push,” Bugzilla will show you an interim page while the search runs.  Currently it shows an animated dino head (left) chomping on bugs, and the text “Please wait while your bugs are munched retrieved.”  It’s cute and all, but it’s kind of getting old.  And being that the page is entirely a cosmetic thing designed to entertain you while you wait, we should change it out once in a while anyway.  We’re planning to upgrade Bugzilla tomorrow night, and it’s the perfect opportunity to spice it up a little.

Gerv’s got a few ideas over on bug 438362… a neat javascript backed game where you click on the ants or somesuch.  If you like it or think you’d hate it, comment here (go try the mockups on the bug first).

If you have other ideas, or can implement one of the existing ones, feel free to post them on the bug.  I have a couple ideas, but no artistic skills to implement them…

  • A Mozilla dino standing there waiting for bugs – Buggie walks over to him carrying a basket of critters and hands it to him.
  • Buggie standing there with his hand shielding his eyes from the sun, turning his head back and forth like he’s looking for something…

Maybe if we have several of these things, it could randomly pick one each time.

Buggie poses for you, courtesy of Dave Shea.

Bugzilla: 10 years ago today

On April 7, 1998, Terry Weissman announced the creation of, a new bug tracking system for keeping track of bugs in the Mozilla code base.

Bugzilla is here.  She’s very young, and fragile.  But if you treat her kindly, she’ll remember your bug reports for you.  When she grows up a little, she’ll become invaluable in helping track what is actually being done to the codebase.

On April 15th, a mere 8 days later, the first person requested the source code.

I like bugzilla!  Its cool!  If I wanted to use it for my own (non-Mozilla) project, how can I go about getting a copy?

But alas, it was still a proprietary Netscape product at the time.  In fact, we learn from Terry in that thread:

you need to be aware that it is built on top of the Kiva application server stuff, and on top of a database (I think it’s an Oracle database). Neither of these things are free.

Wow. Not only were they not free, they were expensive.  Kiva cost about $35,000 at the time, and Oracle…  if you had to ask, it was too much. 🙂  Not only that, but:

first we’d have to convince the folks here at Netscape who wrote it (not me) that this is something they want to do.  That might to be doable, but it hasn’t been attempted yet.  It’s always possible that someone at Netscape has plans to make money off of selling that code.

But that request apparently bore fruit.  On August 26, 1998, with this post, Terry Weissman announced that a new completely rewritten version of Bugzilla had been deployed on  What’s more, this new version included the source code being available for download, and it ran under Apache using MySQL for the database.

The first checkin to CVS was at 11:15pm PDT, August 25, 1998.

Happy 10th anniversary to the open source Bugzilla Project!

Awesome Bugzilla Publicity

So I’ve been a bit busy with things lately and I’m just starting to catch up on a couple podcasts I regularly listen to, one of them being CNET’s Buzz Out Loud, which I’m still a few weeks behind on. This is where Molly Wood, Tom Merritt, and Jason Howell discuss tech news for 20 or 30 minutes every weekday. Well, back in the last week of June, they were talking about Bill Gates having his last week at Microsoft as a full-timer, and it somehow turned into a conversation about Bugzilla. 🙂  It was so awesome, especially Molly’s last line at the end before they got back to the actual story. 🙂

Source: CNET Buzz Out Loud Podcast, Episode 752 @ time index 20:50

TOM: It’s Bill Gates’ last week on the job at Microsoft, although he’ll still be working there, after his last day, part time.

MOLLY: Whether they want him to or not.

TOM: He’ll be the..  I think we speculated he’ll be a bug squasher?

MOLLY: Yeah, something like that

TOM: He’ll just be getting..  they’ll be filing bugs to Bill and he’ll just like hunt ’em down

MOLLY: They’ll give him his own special Bugzilla queue.

TOM: Yeah…  Heh, yeah, like they’re using Bugzilla.

MOLLY: Who isn’t?  Everybody uses some form of Bugzilla for bugs.

TOM: That’s a good question: Is Microsoft using Bugzilla …

MOLLY: (interrupting) Sure!

TOM: … or do they have their own proprietary bug-killing system …

MOLLY: (interrupting) Oh, come on…

TOM: … er, bug-tracking system?  Er, you — Microsoft using an open source bug tracking system?

MOLLY: Oh, well, I guess

TOM: I dunno, it’s just a question in my mind.

MOLLY: I was like who? Everyone… what? Come on…  Bugzilla, it’s just what you use!

Bugzilla at OSCON 2008

Bugzilla will again be participating at the Mozilla booth at OSCONMax and I will both be there from 2:30 to 3:30 on Thursday afternoon for sure, and you may find us hanging out at other times as well.  If there’s any other Bugzilla folks going to OSCON, please drop in and help us. 🙂  Last year Bugzilla got a pretty good reception (we generated more non-Mozilla-employee traffic at the Mozilla booth than Firefox did!)

If you’re trying to find me at OSCON, follow this link for where I plan to be (the sessions listed are subject to change based on my whims as we get closer to OSCON of course).