Much of this post is taken from a message I posted to the developers list a few days ago, so my apologies in advance to anyone reading it again. I’ve expanded on a few things and added the information about the upcoming meeting, so it’s probably worth re-reading.
For those unaware of the context, Frédéric Buclin last week announced that he was stepping down from his Assistant Project Manager position after 9 years.
To Frédéric: Thanks again (and again!) for all your hard work over the years! As stretched as I’ve been for time myself it has been a true godsend to have you picking up my slack the last few years. You will be missed!
To everyone else: For the time being, I’ll be handling approval requests, so if you have something up for approval and it’s not getting attention, I’m the one to pester.
This is sort of the end of an era for the Bugzilla project… Both Frédéric and Max (who left to work at Google a couple years ago and stepped down from his position earlier this year for lack of time) have been with the project for much longer than most people ever stick with a single employer in IT-related jobs (of which an open source project of this magnitude has a lot of similarities). For an open source project, that’s outright amazing, as people tend to come and go a lot in most projects. It’s kind of surprising that I’ve been around longer than them, but I’m kind of a “lifer” in some ways, and in reality I’ve had a good break from the project for the last few years because Max and Frédéric have been mostly taking care of everything while I’ve been busy with other things.
So it’s time to begin a new era. Since I’ve had a good break to clear the monotony I’m going to be trying to get more involved myself again (which I’ve been saying ever since Max left, but I have a lot more incentive now). I’d also like to kickstart a new team to lead the project, and kind of re-organize if you will. We have a number of positions within the project for various functions, which we’ve never really paid attention to as people moved on. So some questions we’ll be asking at our upcoming meeting are:
- What positions in our existing structure do we have open?
- Do we still need them all?
- Are there new positions we have a need for that we should create?
- Who should fill them?
- Do the existing holders of positions that we still need and haven’t been vacated want to keep doing them?
We also have some other “reinventing the project” type topics while we’re at it. There’s a number of things we’ve been talking about doing for a long time that we never really moved on, and some of the big elusive dreams (the big UI overhaul!) have actually been making progress as well, lately. When we’re in the middle of big changes like this, I think it’s a good time to review where we are, get everyone on the same page, and tackle some of these things we keep talking about.
We also have a lot of new useful technology at our disposal since the last time we had a project meeting. We’re going to experiment with using Google Hangouts for the meeting this time, and using their feature to stream the Hangout via YouTube for those who want to watch without participating. We’ll also keep our usual meeting IRC channel open so people who don’t have a Google+ account and don’t want to get one can still participate and ask questions via IRC.
The preliminary agenda and participation instructions have been posted at https://wiki.mozilla.org/Bugzilla:Meetings. The meeting will be held on Wednesday, July 17th, at 14:00 UTC. And before anyone complains about the time, this was the best time to avoid inconveniencing the largest number of people. The Bugzilla Project has a global pool of contributors, and we have active contributors in a wide variety of time zones. The time chosen puts the meeting in the middle of the night for the fewest number of people. Those on the west coast in the US will probably have to get up a little early, and those in eastern Australia will be up a little later.
A lot of the emails and comments I’ve gotten since Frédéric’s announcement have been really positive, so I’m encouraged by the number of people who are still committed to keeping Bugzilla vibrant! We’ll see you at the meeting on Wednesday!
I recently did up a diagram of how our Bugzilla site was set up, mostly for the benefit of other sysadmins trying to find the various pieces of it. Several folks expressed interest in sharing it with the community just to show an example of how we were set up. So I cleaned it up a little, and here it is:
At first glance it looks somewhat excessive just for a Bugzilla, but since the Mozilla Project lives and dies by the content of this site, all work pretty much stops if it doesn’t work, so it’s one of our highest-priority sites to keep operating at all times for developer support. The actual hardware required to run the site at full capacity for the amount of users we get hitting it is a little less than half of what’s shown in the diagram.
We have the entire site set up in two different datacenters (SJC1 is our San Jose datacenter, PHX1 is our Phoenix datacenter). Thanks to the load balancers taking care of the cross-datacenter connections for the master databases, it’s actually possible to run it from both sites concurrently to split the load. But because of the amount of traffic Bugzilla does to the master databases, and the latency in connection setup over that distance, it’s a little bit slow from whichever datacenter isn’t currently hosting the master, so we’ve been trying to keep DNS pointed at just one of them to keep it speedy.
This still works great as a hot failover, though, which got tested in action this last Sunday when we had a system board failure on the master database server in Phoenix. Failing the entire site over to San Jose took only minutes, and the tech from HP showed up to swap the system board 4 hours later. The fun part was that I had only finished setting up this hot failover setup about a week prior, so the timing couldn’t have been any better for that system board failure. If it had happened any sooner we might have been down for a long time waiting for the server to get fixed.
When everything is operational, we’re trying to keep it primarily hosted in Phoenix. As you can see in the diagram, the database servers in Phoenix are using solid-state disks for the database storage. The speed improvement when running large queries that is gained by using these instead of traditional spinning disks is just amazing. I haven’t done any actual timing to get hard facts on that, but the difference is large enough that you can easily notice it just from using the site.
We’re finally at the point where I can say we’re ready to upgrade Bugzilla @ Mozilla this weekend. We’re aiming for Sunday evening (probably 6pm PST). I’ll post again when I know how long it’ll be down for (and that’ll be included in the eventual downtime notice on the IT blog as well).
There’s a staging copy set up at https://bugzilla-stage-tip.mozilla.org/ and I would appreciate people playing around with it and finding anything that might be broken before we get it to production. Before filing bugs, make sure to check the detailed status linked from the red box at the top of every page to make sure it’s not already listed (and you can also see my progress on cosmetic issues and so forth, there).
It will be down for a while at some point tonight when I reload it with an up-to-date snapshot of the production server (and that’ll be my test to find out how long it’ll take to upgrade it, too). I’m super excited because this has been a long time coming.
So, thanks to Fowl’s comment in my previous entry, I tried out setting up the Google news/blogsearch keyword feeds for Bugzilla in Google Reader, and then individually sharing each of the entries that were actually about Bugzilla, and then using the shared items feed from Google Reader as the newsfeed. It turns out to work quite well.
The built-in RSS widget that ships with WordPress completely chokes on Google Reader’s shared items feed though, and I ended up installing the SimplePie plugin for WordPress to handle the feed parsing (thanks to srikat on #wordpress for suggesting that). SimplePie turns out to have a much more flexible template system, and it looks pretty good. If you’re viewing this on my website, the results are over there on the right.
The feed icon in the title is linked to the source feed, so you can use that as a subscription URL if you want to subscribe to my sanitized Bugzilla news feed. Now you can keep up on all the Bugzilla mentions in the news and blogosphere without having to wade through all the exterminator jokes, monster VW Beetles, and mentions of bug reports in every project’s bugzilla site.
I like to know how Bugzilla’s being talked about out in the world, so I subscribe to Google’s handy news and blog alert services where you can put in keywords and they’ll send you an email every time a new blog or news article shows up containing that/those keywords. This works really well, except that you also get a bunch of crap with it. People who post to message boards with “Bugzilla” as a username. Message posts on insect websites about this huge bug someone found. Articles about modified VW Beetles that have been turned into monster trucks. Every mention in anyone’s blog about bug reports for any number of major projects that use Bugzilla for their bug trackers. Etc Etc Etc.
I’ve long desired to create an RSS feed where I could collect that and republish all those links without all the cruft in it, i.e. just the stuff that was actually about Bugzilla as a product/project. I really like Jon Gruber’s feed, where he puts the link and the title and a small blurb of his own comments about it. Note that the linked title goes to the article being talked about, not an entry on his blog, which is the key feature I’m looking for (and what makes it different than just creating a category for it in the blog). Something just like that would be awesome if there were a plugin for WordPress that would do it. I’ve searched on and off and never found anything that would do this. If anyone knows of one, please let me know in the comments here.
On Friday, I pushed a small update to bugzilla.mozilla.org that fixed bug 452799, where users who didn’t have ‘canconfirm’ privs in Bugzilla were posting bugs that had a status of NEW rather than UNCONFIRMED.
This morning, I pushed an update to bugzilla.mozilla.org containing a plethora of additional fixes to address concerns raised since the Bugzilla upgrade. This morning, we’ve picked up fixes for:
- Bug 452793: (The other half of the issue which was fixed Friday) The default status selected when you file a new bug and do have ‘canconfirm’ privs is now NEW instead of UNCONFIRMED.
- Bug 452810: The wording surrounding the checkbox to add youself to the CC now says “Add me to the CC list” when you aren’t on it, instead of just “myself.”
- Bug 452734: The keyword chooser has been replaced with keyword autocomplete. NOTE: If you installed the greasemonkey script to remove the keyword chooser, you’ll probably have to remove that script to get the autocomplete, since it hooks on the same event listener.
- Bug 452798: The CC list is now visible again by default, and as a bonus, it’s now searchable via Firefox’s find-as-you-type feature.
- Bug 452733: The [Classification] is no longer shown in front of the bug summary.
- Bug 452746: The link to the bug in the header no longer contains an extra space.
- Bug 452891: The “visually jarring” dashed border next to the line numbers in the Diff Viewer has been removed.
- Bug 452749: The midair page once again specifies who you midaired with.
- Bug 344559: Add a Commit button near the form fields at the top of the show_bug page so you don’t have to scroll to the bottom of the comments if you’re only changing a field at the top.
Fixes for admins:
- Bug 452898: Milestones can once again be marked inactive.
- Bug 452914: Multiple problems were fixed in the flag editor related to the “fixed in version” field not being dealt with correctly on a product change.
Hopefully this fixes up some of the more major concerns people had. There’s still more to come. At this point I’m plannng on daily pushes to production as the fixes become available.
UPDATE: Some people are reporting broken CSS and things looking strange… hold the Shift key and hit Reload if that’s you. Your browser is probably caching the old CSS.
So bugzilla.mozilla.org got upgraded to Bugzilla 3.2 last night. Since the upgrade, there’s been a lot of complaints about the new UI.
First off, given the differences in the way Mozilla uses Bugzilla compared to a lot of other places, some of these complaints are valid. But, please try to be polite and state exactly why you think you have issues and suggest ways for improvement. Don’t just run around saying it sucks or file bugs stating that you’re ticked off at the world because we broke your workflow.
One of the primary complaints Bugzilla as a product has received over the years is how the UI is ugly and hard to manage. The last year or so the Bugzilla developers have been spending a lot of effort to fix that problem, with the assistance of professional UI designers. Some of them are taking personal offense to some of the feedback we’ve gotten so far this morning about the UI changes because it makes them feel like all the work over the last year was for nothing if everyone just wants the old UI back.
Yes, in some cases, maybe you just have to suck it up and learn a new way to do things. In others, there’s probably a lot of room for us to still clean things up. In either case, please don’t burn the Bugzilla devs in effigy or anything. :) Be kind on the bugs you file (but do file them). Be constructive. Don’t say “This and this are bad they way they are now, please put them back how they were.” Do tell us “this is my usage case and what I need to do with Bugzilla, and here’s why the old way helped me be efficient doing this. Let’s come up with a way for it to be easy for me to do this again.” In all honesty, I bet there’s use cases that weren’t thought of in the current design, and maybe it was just overlooked. Give them the benefit of the doubt, and let us work with you to get something set up that makes your life easy again (maybe we’ll come up with something even better than both the old way and the current way, who knows?)
The major upgrade to 3.2 is done. All the schema changes that took hours to run are in place. Deploying changes to the UI at this point is just be the flip of a switch and it’ll just be live with no downtime at all, in most cases, so we can continue to tweak as we go over the next few weeks. But please try not to get pissed at us and let us help fix it. We really weren’t intentionally trying to break your world, you know.
On bugzilla.mozilla.org, when you run a search, if your browser supports “server push,” Bugzilla will show you an interim page while the search runs. Currently it shows an animated dino head (left) chomping on bugs, and the text “Please wait while your bugs are munched retrieved.” It’s cute and all, but it’s kind of getting old. And being that the page is entirely a cosmetic thing designed to entertain you while you wait, we should change it out once in a while anyway. We’re planning to upgrade Bugzilla tomorrow night, and it’s the perfect opportunity to spice it up a little.
If you have other ideas, or can implement one of the existing ones, feel free to post them on the bug. I have a couple ideas, but no artistic skills to implement them…
- A Mozilla dino standing there waiting for bugs – Buggie walks over to him carrying a basket of critters and hands it to him.
- Buggie standing there with his hand shielding his eyes from the sun, turning his head back and forth like he’s looking for something…
Maybe if we have several of these things, it could randomly pick one each time.
On April 7, 1998, Terry Weissman announced the creation of bugzilla.mozilla.org, a new bug tracking system for keeping track of bugs in the Mozilla code base.
Bugzilla is here. She’s very young, and fragile. But if you treat her kindly, she’ll remember your bug reports for you. When she grows up a little, she’ll become invaluable in helping track what is actually being done to the codebase.
On April 15th, a mere 8 days later, the first person requested the source code.
I like bugzilla! Its cool! If I wanted to use it for my own (non-Mozilla) project, how can I go about getting a copy?
But alas, it was still a proprietary Netscape product at the time. In fact, we learn from Terry in that thread:
you need to be aware that it is built on top of the Kiva application server stuff, and on top of a database (I think it’s an Oracle database). Neither of these things are free.
Wow. Not only were they not free, they were expensive. Kiva cost about $35,000 at the time, and Oracle… if you had to ask, it was too much. :) Not only that, but:
first we’d have to convince the folks here at Netscape who wrote it (not me) that this is something they want to do. That might to be doable, but it hasn’t been attempted yet. It’s always possible that someone at Netscape has plans to make money off of selling that code.
But that request apparently bore fruit. On August 26, 1998, with this post, Terry Weissman announced that a new completely rewritten version of Bugzilla had been deployed on bugzilla.mozilla.org. What’s more, this new version included the source code being available for download, and it ran under Apache using MySQL for the database.
The first checkin to CVS was at 11:15pm PDT, August 25, 1998.
Happy 10th anniversary to the open source Bugzilla Project!
So I’ve been a bit busy with things lately and I’m just starting to catch up on a couple podcasts I regularly listen to, one of them being CNET’s Buzz Out Loud, which I’m still a few weeks behind on. This is where Molly Wood, Tom Merritt, and Jason Howell discuss tech news for 20 or 30 minutes every weekday. Well, back in the last week of June, they were talking about Bill Gates having his last week at Microsoft as a full-timer, and it somehow turned into a conversation about Bugzilla. :) It was so awesome, especially Molly’s last line at the end before they got back to the actual story.
Source: CNET Buzz Out Loud Podcast, Episode 752 @ time index 20:50
TOM: It’s Bill Gates’ last week on the job at Microsoft, although he’ll still be working there, after his last day, part time.
MOLLY: Whether they want him to or not.
TOM: He’ll be the.. I think we speculated he’ll be a bug squasher?
MOLLY: Yeah, something like that
TOM: He’ll just be getting.. they’ll be filing bugs to Bill and he’ll just like hunt ‘em down
MOLLY: They’ll give him his own special Bugzilla queue.
TOM: Yeah… Heh, yeah, like they’re using Bugzilla.
MOLLY: Who isn’t? Everybody uses some form of Bugzilla for bugs.
TOM: That’s a good question: Is Microsoft using Bugzilla …
MOLLY: (interrupting) Sure!
TOM: … or do they have their own proprietary bug-killing system …
MOLLY: (interrupting) Oh, come on…
TOM: … er, bug-tracking system? Er, you — Microsoft using an open source bug tracking system?
MOLLY: Oh, well, I guess
TOM: I dunno, it’s just a question in my mind.
MOLLY: I was like who? Everyone… what? Come on… Bugzilla, it’s just what you use!