Migrating stage.mozilla.org to a new host has been in the works for a long time. It’s been stalled for a couple months while we worked with the developers of the unionfs filesystem (mostly by providing them with crash reports) to stabilize it to the point where we felt comfortable putting it to use on a production service. That time has finally come. The alternative to unionfs (keeping 9 terrabytes of disk space online at once, completely dedicated to the FTP staging process) was really not cost effective, and it was really worth waiting for.
This Thursday, March 13th, we’ll be switching the DNS for the stage.mozilla.org domain name to point at the new box. The SSH host keys have been copied over, so most people will never notice. The existing box, effective immediately, is accessible using the name stage-old.mozilla.org. If for any reason you’re nervous about switching your upload process to the new box this coming Thursday, you can use the stage-old name to keep using the old one, for now. Stage-old will go away on Tuesday, March 25th, so you’ll have until then to resolve any issues you have (or get IT to resolve them if you think it’s our problem). Feel free to file a bug against Server Ops if you need help.
Be sure to read my previous post for details of how the new system will work (it is a little different, though most well-behaved upload scripts shouldn’t be affected). The most noticeable change will be that there will now be a slight delay between when you upload something and when it shows up on the ftp server, since we’re now virus-scanning what you upload before it gets made available on the ftp server. On the current system that we’re moving away from, the virus scan runs after the files are placed on the ftp server, and then the files are yanked afterwards if they get flagged. This of course, could let something get out there briefly, which we obviously don’t want.
On behalf of those of us who use stage.mozilla we thank you 🙂
Also, can I suggest you file this in “Work” and not “Personal”?
Thanks for the heads-up. I could swear I checked “Work” when I started working on it, but I did have a crash while writing it, and I must have gotten bitten by the session restore or something.
My god, 9TB? What all is that comprised of?
The FTP server has about 2.6 TB of stuff on it last time I looked. The 9 TB reference is because without the fancy unionfs scheme, we’d need 3 copies of it online at the same time to be able to virus scan things without pushing them to the visible ftp server.
It’s mostly a selection of nightly builds since the begining of time, and so forth. 🙂