One Hour of Terror

“One hour of terror” — This is how we’ve jokingly started referring to the first hour of every month (measured on GMT) because of the bug in the 1.0 version of Firefox which causes it to only check for updates between the first of the month and the first Sunday of the month. Firefox checks with itself once per hour to see whether it’s been long enough since the last time it checked the server for updates to check again. And any version 1.0 of Firefox that happens to be running at midnight GMT is going to have that check fire within the first hour of the clock ticking over past midnight. This absolutely SLAMs our servers with every known copy of Firefox 1.0 (shame on people for not upgrading) checking in during that first hour instead of how they’re usually spread across the entire day.

Today is May 1st. Last night, at midnight GMT, was that hour. See the bandwidth graphs. We’d been hoping to have our new hardware set up already by now, but it just arrived this last week, and we haven’t had time to configure it yet. Sky (which has been handling the application update service all by itself until now) took a beating right at midnight (17:00 on the charts I linked above). Within a few minutes, I managed to clone the webserver configuration onto Star (a machine about to be deployed for use by the Talkback services, but which the Talkback folks haven’t actually set anything up on yet) and added Star to the rotation so the requests were split between Sky and Star.

The scary part? When you look at those graphs, you’ll notice Star’s bandwidth skyrocket when it was added to the rotation, but Sky’s bandwidth didn’t go down at all. This means Star didn’t ease any load off of Sky at all, it just picked up load that hadn’t been making it through to begin with. Ooof.

We served over a million requests during that first hour (850,000 on sky and 200,000 on star), and about half a million during the second hour (split close to evenly). And the graphs make it plain to see that we weren’t serving all the requests that were coming in.

Next month we’ll be ready for it.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.