stage.m.o, shift-reloaded
Just like ftp.m.o, shift-reloaded, stage.mozilla.org is getting an overhaul as well. If you read through preed’s post that I just linked, you’ll see the plans for stage included in there as well. For various reasons (most of them including LDAP and testing issues) they ended up not happening at the same time, and we ended up doing only the ftp/archive part during that last outage window. We’re finally ready to proceed with the stage.m.o half of this, and the best part is for this chunk of the puzzle, we don’t even need an outage window. We’ll be running the new stage along side of the old one for a little while to let people try it out, make sure their upload scripts still work, and bring issues to our attention before the old one goes away.
The items listed on preed’s earlier post that still have to happen are:
- All files will be virus scanned before becoming available. We currently virus scan all builds, but depending on a number of factors, it was possible for unscanned-builds to appear on the FTP site for a window of time; we’ve removed this window.
- Interactive shell accounts on the FTP farm will be replaced with sftp-only accounts.
Non-interactive Accounts For Uploaders
One of the changes we’re making is that shell accounts are going away. The new server will have you chrooted into the staging area, and you will be limited to scp, sftp, rsync, and a small subset of file management commands (such as mv, cp, chmod, chown, chgrp, etc) invoked via ssh. This is the reason that if you have any scripted uploads, you need to test them on the new server to make sure they still work, and whether you can modify them so they will if they don’t. If you are doing something on stage currently that really needs full shell access, we’ll probably be happy to accomodate you, just on some other machine. Come talk to us or file a bug.
We are also moving from local accounts on the staging server to LDAP-based accounts, to make management of permissions easier. In a few cases, this might mean your username will change (in most cases, the affected people have been contacted already). Accounts that haven’t been used in the last year will not been ported over. If you haven’t connected to stage in the last year you’ll need to file a bug in the FTP: Staging component to get your access back. Accounts will be getting enabled over the course of the next 24 hours. If you want to get in sooner than that, come find me on irc and I’ll manually toggle your account. The new machine is located at stage-new.mozilla.org.
Virus Scanning
The ftp file tree was almost a terabyte in size before. Now, with it combined with archive.m.o, it’s 2.2 terabytes. Keeping just one copy of that sitting around is decently expensive. Keeping multiple copies of it live is just cost prohibitive. On the current staging system, for lack of disk space, there’s no way to prevent the files from going to the mirrors before getting virus scanned. The virus scanner would come along and scan the newly-uploaded files and then yank any with viruses found back out. So there is a small window of time when something with a virus in it could make it to the mirrors and then would disappear off the mirrors again the next time they synced.
With the new staging system, we’re making use of a new (to Linux — BSD has apparently had it for years) filesystem technology called unionfs which allows us to layer the filesystem. If it helps to visualize it, think of it like a multi-layer photo in Photoshop, where each layer is transparent by default, and each pixel in the photo is a file on the filesystem. When we make changes to a file, we’re only really changing the pixel in that position in the topmost layer (that pixel is now no longer transparent). From here on out, this post gets a little technical, to explain how it’s set up for those that are curious. If you’re not the technical type, you can feel free to stop reading here. There’s nothing beyond this point that will affect your ability to upload if you have an account.
The main ftp tree (which is NFS-mounted from a huge disk array) gets mounted read-only as the “base layer” of a unionfs mount. We then put another layer on top of that which contains the chroot jail environment (the executables and libraries needed for jailed users to have minimal functionality while connected, but don’t need to be visible to the mirrors). There is another read-only layer on top of that (Layer B on the left) which is used for the virus scanning (more on that in a moment). The top layer (Layer A in the diagram) is the one that the end-users actually write to when they make changes to files. This last layer records all of the additions, deletions, and modifications to files that the users make so that those files appear changed to the unified filesystem view that the users see, but the underlying filesystem that the mirrors rsync off of isn’t touched.
The layers can be reordered and changed from read-write to read-only at any time, so when it comes time to push the files to the mirrors, we move Layer B to the top and make it read/write, change Layer A to read-only. This way, nobody can make changes to it while we’re scanning it. We then virus-scan layer A directly. Once the scan completes, and any infected files have been removed, we then move the changes from this layer down to the real live ftp/archive layer at the bottom which the mirrors can see when they rsync. Layer A is then cleared off so it’s ready to be swapped with Layer B again on the next pass.
Localizing Asterisk for China
This last week, we set up a bunch of the IT infrastructure at Mozilla’s new Chinese office. My primary part of the process was setting up their phone system. We used Asterisk, of course, seeing as how we already use that for our phone systems in Mountain View and Toronto. Asterisk has a really cool feature that lets you put localized sound files in for the voice prompts, and each device and incoming phone line can be set up to default to a particular language. It will use the localized files for that language, if available, and fall back on the English ones if they aren’t. You can also allow users to change which language they get with a little careful scripting (”Press 1 for English”, etc). We set it up so that dialing from any phones in the Beijing office will get Chinese prompts, dialing into the Beijing office from outside will get Chinese prompts, and dialing into the Beijing office via the links to our other offices will get English prompts.
One of the difficulties we’ve run into is that there’s no official Chinese language pack for Asterisk, and the only unofficial Chinese language pack we could find is fairly incomplete. You’ll be listening to something in Chinese (like the instructions for voicemail) and suddenly get a word or two of English in the middle of it. 
I ended up spending a fair portion of this week trying to set up a nice friendly web app the folks in the office can use to easily see which files have been localized and which haven’t, and allow them to record their own localized files and have them automatically go where they need to go. Other folks might find it useful, so I’ll try to get it posted somewhere once I get it fixed up a little (it’s a bit of a quick and dirty hack still right now, but it’s getting there).
Digging people
So yesterday afternoon, Alex Faaborg blogged about some new features in Firefox 3. No big deal until it got posted on digg.com. The blog server could take it. It’s in the load balancing cluster behind a caching proxy server, which doesn’t even notice this kind of traffic. But Alex had posted his images in his personal space on people.mozilla.com, which is a single server which isn’t really considered production critical on IT’s priority list. Now, even though this is a single server, it’s not exactly sucky hardware. The machine should have been more than capable of handling a slashdotting and getting dugg at the same time. So we were all pretty surprised when it fell over.
Apache kept dying, and spitting out errors about failing to setuid to the apache user. After much banging of heads, Justin Dolske found a relevant forum post in one of the Gentoo forums of all places, which pointed the finger at per-user process limits, and using ulimit in the initscript to override them. Using ulimit turned out not to be necessary, but it did get me looking in the right places.
Mozilla employees get shell accounts on people.mozilla.com (makes it easier for them to manage the webspace there, and several folks use it to run irssi in screen to keep a session to irc.mozilla.org open). In order to keep users from bogging down the machine, we had used pam_limit to limit user logins to 100 processes per user in /etc/security/limit.conf. Well, it turns out that this limit applies to both root and apache as well. So when apache spawned that 100th process to handle that many concurrent connections, it hit that limit and died. Now, root is immune to process limits, however, limits set for root still apply to any setuid processes spawned by root, if that limit is lower than the user being setuid to. So setting a specific (higher) limit for apache in limit.conf wasn’t enough. Had to bump it up for root as well.
But that did the job. The site was back up in no time, happily serving all the images any Digg user could want to go with Alex’s blog, and still keeping an 0.03 load average. Next time someone’s images get posted to Digg or Slashdot, people.mozilla.com will be ready.
Cron job output overload
We have a mailing list at Mozilla which receives mail sent to root at any of our servers. The majority of this mail is cron job output. I have filters set up in my Zimbra account to filter the cron job mail specifically into a folder separate from the rest of the mail to that mailing list. I was on vacation last week, and the last day before I left, I completely deleted the contents of that folder. On my return, that folder contained 26,373 messages in it, all dated within the last week. Trying to separate the nuisance mail from the real problems is kind of impossible by hand with that volume.
Obviously one task is to eliminate the nuisance mail. This has to be done carefully, because typically you still want to get errors from cron jobs, but you don’t want the general output. And not all jobs are good about their use of standard error and standard output, so often you can’t just devnull the standard out and expect to only get mail when there’s a problem. So fixing the nuisance mail sometimes means writing a wrapper script for a cron job that does some grep or awk work to filter the output. But even with the nuisance mail gone, it’s a lot of mail to sift through to find any possible real problems.
So, I filed bug 377043 with an idea for a tool to do some automated analysis of all this cron job output. Keep track of patterns and point out things that need looking at, etc. Unfortunately both cron jobs and data analysis are pretty popular topics (and usually not related to each other) so Google isn’t helping me much trying to search for existing tools. Does anyone know of any existing tools that do something similar to this that we might either be able to use, or build upon?
Vacation and work travel
So this last week I’ve been on vacation, but just hanging out at home hoping to catch up on some things. One of the projects I’ve been working on this week is trying to write a driver for lirc to use a USB-attached IR receiver on Mac OS X. One of my MythTV boxes is running on Mac OS X, and it’s annoying to have the little white Apple remote be the only one that works on it (it’s a nice simple remote, but there’s just not enough buttons on it to be useful for a full entertainment center). I’ve been hoping to get that driver working before I left so my wife could use a real remote while I’m gone. Not quite there yet, not much time left. I made major progress on it this afternoon though while the kids were watching the new movies they got in their Easter baskets.
Tomorrow afternoon (Monday) I leave to head out to Mountain View for our quarterly all-hands meeting at Mozilla. The following week I’ll be attending an Asterisk training program put on by Digium in San Jose (teach me everything I need to know to run the PBX system at Mozilla), so I’ll be away from home for two weeks. It’s always fun visiting Mozilla, but it’s not going to be fun being away from the family for that long.
CVS Checkin mail
I just checked in a change to the script the Mozilla cvs server uses to send email. It’s one that’s been a long time coming, to get it to use a more reliable way to send the email so that high load conditions won’t prevent the message from delivering. I don’t expect any trouble with it, but if you’re on one of the mailing lists or newsgroups that get new check-in messages, do let me know if you notice anything strange.
Thanks!
MacFUSE
So, yesterday, I overheard someone mentioning MacFUSE in a conversation and my reaction was pretty much “hey, someone ported that to the Mac?!??” Indeed, and I’m apparently a little out of touch because it’s been out for a while. This is one really handy tool for a sysadmin.
Anyone who’s admined an RPM-based system knows that when you install a newer version of something, it will often drop in new configuration files, either along side the existing one with an .rpmnew extension on the end, or move your existing one to filename.rpmsave before writing it out to the original location. Of course, if you’re behaving like a proper sysadmin, you have to go look and see what they changed and try to merge your existing config with the new one. This is best accomplished using a visual diff/merge utility. The best one of these I’ve seen for Linux is called meld. Unfortunately it has a ton of dependencies (mostly related to GNOME and X11) which are a bit much to install on every server in your farm, especially if there’s no other reason you need X11 on them.
Now back to MacFUSE. Apple distributes an awesome visual diff/merge utility with the Mac OS X developer tools called FileMerge, which works a lot like meld (but of course, it’s a bit more polished). MacFUSE is a kernel extension that allows plugins to define interfaces to arbitrary filesystems. One of the plugins available for it is called “sshfs.” sshfs uses the sftp server available via most ssh servers to access the filesystem of almost any machine you can ssh into. This means if I can open an ssh connection to one of my servers, I can now mount its filesystem in the Finder in Mac OS X (and also via the shell). This makes it easy to navigate to the server’s /etc directory and type “opendiff config1.conf config1.conf.rpmnew” which invokes Apple’s FileMerge to compare/merge those two files. It’s a huge effort saver for a sysadmin using OS X on their workstation.
Mad props to Amit Singh and all the people helping him make MacFUSE a reality!
The results are in
Wow, that was fast. They said they’d have the results of the RHCE exam posted sometime in the next 3 business days (which would mean sometime next week), but I had an email waiting for me when I got home tonight with the results. I passed. So I’m now officially a Red Hat Certified Engineer.
The exam is done
So I think the exam went pretty well. I know I passed the first half, I’ll get the results of the second half and the overall sometime in the next three business days. Now I have to get out of here in a hurry to try to get home ahead of the incoming storm at home. It’s likely I’ll be driving at least part of the way through the middle of a blizzard. Originally I was planning on staying here another day (I’m in Chicago for the class), and heading home tomorrow, but the weather forecast looks like if I don’t go now I’ll be stuck here until Tuesday or so.
The class and the exam were a lot of fun. Lots of challenging problems and it was a lot of fun to fix them up. I did learn enough new tricks from the class part to have made it well worth my while taking the class. I’d highly recommend it to anyone with good Linux experience that’s interested. Just make sure you have an employer willing to pay for it because it’s expensive.
This should be fun
I’m taking the RHCE exam today. Based on the last four days of RH300 classes (which have been awesomely fun) this should be a blast. I love fixing things so even if I don’t pass I think it will be great fun taking the exam.
-
stuff
