Digging people

June 3, 2007 by
Filed under: Work 

So yesterday afternoon, Alex Faaborg blogged about some new features in Firefox 3. No big deal until it got posted on digg.com. The blog server could take it. It’s in the load balancing cluster behind a caching proxy server, which doesn’t even notice this kind of traffic. But Alex had posted his images in his personal space on people.mozilla.com, which is a single server which isn’t really considered production critical on IT’s priority list. Now, even though this is a single server, it’s not exactly sucky hardware. The machine should have been more than capable of handling a slashdotting and getting dugg at the same time. So we were all pretty surprised when it fell over.

Apache kept dying, and spitting out errors about failing to setuid to the apache user. After much banging of heads, Justin Dolske found a relevant forum post in one of the Gentoo forums of all places, which pointed the finger at per-user process limits, and using ulimit in the initscript to override them. Using ulimit turned out not to be necessary, but it did get me looking in the right places.

Mozilla employees get shell accounts on people.mozilla.com (makes it easier for them to manage the webspace there, and several folks use it to run irssi in screen to keep a session to irc.mozilla.org open). In order to keep users from bogging down the machine, we had used pam_limit to limit user logins to 100 processes per user in /etc/security/limit.conf. Well, it turns out that this limit applies to both root and apache as well. So when apache spawned that 100th process to handle that many concurrent connections, it hit that limit and died. Now, root is immune to process limits, however, limits set for root still apply to any setuid processes spawned by root, if that limit is lower than the user being setuid to. So setting a specific (higher) limit for apache in limit.conf wasn’t enough. Had to bump it up for root as well.

But that did the job. The site was back up in no time, happily serving all the images any Digg user could want to go with Alex’s blog, and still keeping an 0.03 load average. Next time someone’s images get posted to Digg or Slashdot, people.mozilla.com will be ready.


2 Responses to “Digging people”

  1. Jo Hermans on June 4th, 2007 6:57 am

    Couldn’t you have used the MaxClients settings in the configuration ?

  2. justdave on June 4th, 2007 2:48 pm

    MaxClients was set to 4000. What Apache thinks its MaxClients is is irrelevant when the OS won’t let you create that many processes.

Feel free to leave a comment...
and oh, if you want a pic to show with your comment, go get a gravatar!

  • Find me on Twitter

    Me: @justdavemiller
    The Bugzilla Project: @bugzilla
  • Recent Tweets by @justdavemiller

    justdavemiller RT @rjbs: If you're a CPAN author, consider changing your password on PAUSE, even to itself. It will get rehashed much more securely. #perl
    justdavemiller @david_cornelson @Support my situation as well. What sucks is it used to work and they took it away.
    justdavemiller 8th place in Wheeled Vehicle goes to #HudsonvilleRMS #msor12
    justdavemiller 4th place in Elastic Launch Glider goes to #HudsonvilleRMS #msor12
    justdavemiller 6th place in Can't Judge A Powder goes to #HudsonvilleRMS #msor12
  • Archives