stage.mozilla.org take 3?
So as you may or may not have noticed, the stage.mozilla.org update previously advertised wound up getting partially reverted about a day after it was deployed. After getting the full production load on it, we wound up crashing it several times again. There’s just not much we can do to emulate real users using WinSCP to upload files from our load testing scripts. 
We’ve gotten some new patches to the unionfs filesystem driver that attempt to fix some of the crashes we’ve been getting. Unfortunately our only real way to test them is to throw it back into production and see what happens. As such, over the next week or two, the stage.mozilla.org domain name will be swapping back and forth between the old machine and the new one periodically as we test things. If you were following the directions given in the previous announcement this shouldn’t affect you at all, but I thought it would be good to give people a heads-up. Obviously this means today’s deadline for the old machine to remain available has been averted, and it’ll probably still be around for another week or two at least.
If you absolutely need to reach the old machine, it’s at stage-old.mozilla.org. The new machine is at stage-new.mozilla.org. The stage.mozilla.org domain name could point at either one of them at any given time for the next week or two while we continue testing. If possible it’d be great if you can continue to use stage.mozilla.org and follow where the domain points so you can help with the testing. But if you run into any problems, feel free to use stage-old.mozilla.org just to guarantee the old way of access.
#build on irc.mozilla.org is the place to ask if you have questions or have any issues.
stage.mozilla.org moving *finally*
Migrating stage.mozilla.org to a new host has been in the works for a long time. It’s been stalled for a couple months while we worked with the developers of the unionfs filesystem (mostly by providing them with crash reports) to stabilize it to the point where we felt comfortable putting it to use on a production service. That time has finally come. The alternative to unionfs (keeping 9 terrabytes of disk space online at once, completely dedicated to the FTP staging process) was really not cost effective, and it was really worth waiting for.
This Thursday, March 13th, we’ll be switching the DNS for the stage.mozilla.org domain name to point at the new box. The SSH host keys have been copied over, so most people will never notice. The existing box, effective immediately, is accessible using the name stage-old.mozilla.org. If for any reason you’re nervous about switching your upload process to the new box this coming Thursday, you can use the stage-old name to keep using the old one, for now. Stage-old will go away on Tuesday, March 25th, so you’ll have until then to resolve any issues you have (or get IT to resolve them if you think it’s our problem). Feel free to file a bug against Server Ops if you need help.
Be sure to read my previous post for details of how the new system will work (it is a little different, though most well-behaved upload scripts shouldn’t be affected). The most noticeable change will be that there will now be a slight delay between when you upload something and when it shows up on the ftp server, since we’re now virus-scanning what you upload before it gets made available on the ftp server. On the current system that we’re moving away from, the virus scan runs after the files are placed on the ftp server, and then the files are yanked afterwards if they get flagged. This of course, could let something get out there briefly, which we obviously don’t want.
Bugzilla Summer of Code 2008
The Bugzilla project is once again going to be participating in the Google Summer of Code under the Mozilla banner. Gerv has the details. If you know of any projects that would be good, be sure to add them to the brainstorming page. For reference, here’s what we signed up for last year (although we never got any takers to work on them - probably for lack of advertising, which we’ll do this year).
How to fix the anime industry
Justin Sevakis over at Anime News Network a couple days ago posted an open letter to the anime industry. It’s really good and worth a read. For those who haven’t been paying attention to the anime scene, his letter does a really good job of explaining the current state of the industry and why anime distribution of questionable legality happens the way it does.
Personally, I’m one of those people who will buy the DVDs of a show that I’ve watched when it finally becomes available in the US, because I believe in supporting the artists who create this stuff, and that’s the easiest way to do so. But as Justin points out in his letter, 90% of the good stuff never makes it to the US, ever. You can look back a few posts in my blog to see a list of some of the series I’ve watched. There’s a lot. Hardly any of them have DVDs or other merchandise available in the US yet. I haven’t shelled out very much money, just because there’s no one to give it to. Some of them actually did get licensed (Pretty Cure) and then the licensee never did anything with them. Some actually made it to market (Ojamajo Doremi/Magical DoReMi) but didn’t survive halfway through the first season in the US (even though there were 4 seasons of it in Japan) because of poor marketing, or (by some accounts) poor execution on dubbing/editing of the English version. My kids love Magical DoReMi, and we’ve managed to accumulate a fair bit of merchandise (doll sets, accessories, DVDs) recently for them to play with, found on clearance at local stores and on eBay. And it wasn’t because of the US marketing (because they took it off the air years ago), but because we saw the fansubbed Japanese versions.
There are shows that do make it and do well. Digimon, Pokémon, and Mega Man are good examples, from which we own a good few dozen DVDs. And one mustn’t forget Naruto (although nobody in my household is a fan of that one). But from the total size of the market, that’s a very small number of shows.
I really hope the industry takes Justin’s letter to heart. It’d make my day to be able to get this stuff from official sources, even if it cost money, although I’d personally prefer ad-supported websites 
. It’s a global economy now, and thanks to the Internet, it’s a world market. It’s time to give up on the region-locked licensing and just distribute globally from the get-go.
Trying to find that long lost tab
I’m very much a power user. I use my web browser constantly, for both work and play, and get extensive use out of tabbed browsing. I keep web pages that are related to the same task in tabs in the same window, and open a new window when I’m shifting gears to work on another task. And I often go back and forth between tasks as things come up or I need a break from the routine, or whatever, so eventually I wind up with a situation like right now where I have 12 windows open, and about half of them have 5 or 10 tabs in them.
Now I’m looking for a specific tab, and it was sort of a one-off thing, and I don’t remember which window it’s in. And it’s not the frontmost tab in the window it’s in, so I can’t just look in the Window menu to find it.
Now I’m thinking it would be really cool if the Window menu had submenus for each window that had multiple tabs in it, which listed the tabs in that window. Then I could just mouse over the windows in the Window menu and glance through the submenus looking for it. Bug 405933 filed.
Outdated extensions are depressing sometimes
There’s a Thunderbird extension that I use that still hasn’t been updated since Thunderbird 1.5, that I have a hard time living without. It’d be awesome if it got updated to work with Thunderbird 2.
It’s Sync On Arrival. This one makes Thunderbird download all of the new messages in an IMAP folder as soon as it sees notification from the server that new messages have arrived. There’s two main benefits this gives me. 1) Thunderbird feels a lot faster, because 90% of the time when I go to look at a message, it will have already downloaded it and can show me the cached version, instead of my having to wait for it to download the message from the server when I open it. 2) It’s essentially “always offline-ready”. Since the messages are always being downloaded when they come in, I can put the laptop to sleep, wake it up somewhere else where I don’t have a network connection, and already have the messages there to look at, without having to have told Thunderbird that I wanted to switch to offline mode before putting the laptop to sleep. Switching to offline mode then becomes just toggling the state and being done with it instead of having to wait for Thunderbird to download everything before traveling, or even better: having to look at an email at a random time when you weren’t expecting to be offline, and having it already be there instead of being out of luck.
Now the situation isn’t quite as dire as it sounds… the extension still works with Thunderbird 2, it just doesn’t know that it does. But having to use the Nightly Tester Tools to override the maxVersion on it every time there’s a Thunderbird update gets annoying. There’s been plenty of comments in the discussion board for it on the addons site saying it works in 2.0 (and a few that say it doesn’t work, but I haven’t experienced any of the problems they’re reporting). I’ve also emailed the author about it, and never got a response. If I knew more about writing extensions and had time to play with it I’d love to fix it, but unfortunately I don’t. It’d be really awesome if someone did though.
Alternatively, if anyone knows of another extension that does a similar job that also works on Thunderbird 2, let me know in the comments here.
I remember the original ExtendFirefox contest had a category for updates to existing extensions that hadn’t been updated for Firefox 2 yet. Maybe Thunderbird needs to do something like that to encourage people to update their extensions.
You can donate money to Bugzilla now
Frank Hecker over at the Mozilla Foundation has just posted about the Foundation’s new directed giving program, which includes the ability to earmark money for the Bugzilla project when you donate now. This has been a long time coming. Every so often we’ve had people coming into IRC or asking on our mailing lists where they could donate money and we’ve never really had anything to point them at other than our T-shirts in the Mozilla Store, or the Foundation’s donation page, which didn’t provide any way to say you wanted the money to go to Bugzilla. Well, now there’s somewhere to go where the money will actually get back to the Bugzilla project, on the Bugzilla project’s donation page.
Free SIP on Leopard exists after all!
So it’s been a few weeks since Mac OS X 10.5 (Leopard) came out. One of the major things that immediately hit a lot of people was that every known third-party SIP client stopped working. I work from home. I have an extension number at work that rings on a Polycom phone in my home office, thanks to the magic of VoIP. This same VoIP technology (namely SIP) has allowed me (up until I upgraded to Leopard) to run a “soft phone” program on my laptop to allow me to connect to the same phone system when I was out of my office. With the help of a set of headphones, a laptop actually makes a halfway-decent phone.
The two free (as in beer, not freedom) products previously available that I knew of were SJPhone and X-Lite. Both of these broke on Leopard. SJPhone hasn’t been updated in years, and the material on their website makes it look like the Mac version was an afterthought anyway, so I don’t hold high hopes for them ever updating it. X-Lite is a pared down version of a commercial product called eyeBeam. eyeBeam just got updated for Leopard this last week. An X-Lite update is expected “by the end of the year”. The obvious reason for the delay is to encourage people who are frustrated enough to throw money at it to upgrade to eyeBeam instead of waiting. 
There *are* two free products that have been updated for Leopard which do SIP to a generic PBX of your choosing. Those are SightSpeed and Gizmo Project. Unfortunately, both of these require you to register with their service, and sign in on their service, and your generic-PBX-of-your-choosing account is a secondary login (if you don’t log into their service, your generic one won’t connect either).
The world is in really dire need right now of a good open source solution for SIP on the Mac. If any of the above programs were open source, I would bet we would have had patches posted somewhere within days (if not hours) to make them work on Leopard.
UPDATE: I was going to post this hoping to get some discussion going and/or someone to point out something that works that I missed. But before I could post it, I found one! XMeeting not only works on Leopard (it apparently didn’t break — there hasn’t been a release since July), but it also supports DTMF (touch tones) during the call, which is the one thing that’s been missing from all the other free stuff I’ve tried so far. Touch tones during a call are pretty important for things like entering the password for a conference call. XMeeting supports video, too (and so does Mozilla’s phone system).
stage.m.o, shift-reloaded
Just like ftp.m.o, shift-reloaded, stage.mozilla.org is getting an overhaul as well. If you read through preed’s post that I just linked, you’ll see the plans for stage included in there as well. For various reasons (most of them including LDAP and testing issues) they ended up not happening at the same time, and we ended up doing only the ftp/archive part during that last outage window. We’re finally ready to proceed with the stage.m.o half of this, and the best part is for this chunk of the puzzle, we don’t even need an outage window. We’ll be running the new stage along side of the old one for a little while to let people try it out, make sure their upload scripts still work, and bring issues to our attention before the old one goes away.
The items listed on preed’s earlier post that still have to happen are:
- All files will be virus scanned before becoming available. We currently virus scan all builds, but depending on a number of factors, it was possible for unscanned-builds to appear on the FTP site for a window of time; we’ve removed this window.
- Interactive shell accounts on the FTP farm will be replaced with sftp-only accounts.
Non-interactive Accounts For Uploaders
One of the changes we’re making is that shell accounts are going away. The new server will have you chrooted into the staging area, and you will be limited to scp, sftp, rsync, and a small subset of file management commands (such as mv, cp, chmod, chown, chgrp, etc) invoked via ssh. This is the reason that if you have any scripted uploads, you need to test them on the new server to make sure they still work, and whether you can modify them so they will if they don’t. If you are doing something on stage currently that really needs full shell access, we’ll probably be happy to accomodate you, just on some other machine. Come talk to us or file a bug.
We are also moving from local accounts on the staging server to LDAP-based accounts, to make management of permissions easier. In a few cases, this might mean your username will change (in most cases, the affected people have been contacted already). Accounts that haven’t been used in the last year will not been ported over. If you haven’t connected to stage in the last year you’ll need to file a bug in the FTP: Staging component to get your access back. Accounts will be getting enabled over the course of the next 24 hours. If you want to get in sooner than that, come find me on irc and I’ll manually toggle your account. The new machine is located at stage-new.mozilla.org.
Virus Scanning
The ftp file tree was almost a terabyte in size before. Now, with it combined with archive.m.o, it’s 2.2 terabytes. Keeping just one copy of that sitting around is decently expensive. Keeping multiple copies of it live is just cost prohibitive. On the current staging system, for lack of disk space, there’s no way to prevent the files from going to the mirrors before getting virus scanned. The virus scanner would come along and scan the newly-uploaded files and then yank any with viruses found back out. So there is a small window of time when something with a virus in it could make it to the mirrors and then would disappear off the mirrors again the next time they synced.
With the new staging system, we’re making use of a new (to Linux — BSD has apparently had it for years) filesystem technology called unionfs which allows us to layer the filesystem. If it helps to visualize it, think of it like a multi-layer photo in Photoshop, where each layer is transparent by default, and each pixel in the photo is a file on the filesystem. When we make changes to a file, we’re only really changing the pixel in that position in the topmost layer (that pixel is now no longer transparent). From here on out, this post gets a little technical, to explain how it’s set up for those that are curious. If you’re not the technical type, you can feel free to stop reading here. There’s nothing beyond this point that will affect your ability to upload if you have an account.
The main ftp tree (which is NFS-mounted from a huge disk array) gets mounted read-only as the “base layer” of a unionfs mount. We then put another layer on top of that which contains the chroot jail environment (the executables and libraries needed for jailed users to have minimal functionality while connected, but don’t need to be visible to the mirrors). There is another read-only layer on top of that (Layer B on the left) which is used for the virus scanning (more on that in a moment). The top layer (Layer A in the diagram) is the one that the end-users actually write to when they make changes to files. This last layer records all of the additions, deletions, and modifications to files that the users make so that those files appear changed to the unified filesystem view that the users see, but the underlying filesystem that the mirrors rsync off of isn’t touched.
The layers can be reordered and changed from read-write to read-only at any time, so when it comes time to push the files to the mirrors, we move Layer B to the top and make it read/write, change Layer A to read-only. This way, nobody can make changes to it while we’re scanning it. We then virus-scan layer A directly. Once the scan completes, and any infected files have been removed, we then move the changes from this layer down to the real live ftp/archive layer at the bottom which the mirrors can see when they rsync. Layer A is then cleared off so it’s ready to be swapped with Layer B again on the next pass.
New anime for the fall
A couple promising series I’ve started watching from this fall’s lineup of anime are Shugo Chara! and Clannad.
Shugo Chara! is insanely cute. My first impression of it was kind of a Lizzie McGuire with a bit of magic thrown in, and set in late elementary school instead of high school. The main character, Hinamori Amu, doesn’t know how to present her feelings to those around her, and often acts like a tough girl around everyone at school, when inside, she’s really a quiet, delicate girl who likes boys and pink frilly stuff and is just scared to express herself. The first episode starts off with her rescuing a nerdy kid from a couple bullies, just by showing up and stating they were blocking her path. Her reputation as a tough girl precedes her. The bullies, upon realizing who she is, rattle off some stories they’ve heard about how tough she is and run off scared silly. Amu says “Who actually starts these rumors?” The nerdy kid is in awe and asks for an autograph, to which she scoffs at him for also blocking the way, and walks off. But this doesn’t deter him, as he still thinks she’s the coolest girl he’s ever met. The scene at the above right is as she’s walking away from the cutest boy in the school after telling him off for daring to talk to her when she first met him. “Don’t act like we’re buddies or anything” she told him. The little “Chibi Amu” coming out and attacking herself for not being nice to him is where my Lizzie McGuire reference came from (this happens frequently throughout the show, where the “Chibi Amu” will come out and tell the audience what she’s really thinking when she’s too scared to say it). In private, she continually tells us and herself about how she’s really a nice person and just wishes she could drop the tough girl act once in a while, but she worries that no one would believe it was her. The magic part comes in when she finds three eggs in her bed that turn out to be “Guardian Characters” (hence the name of the show - Shugo is Japanese for guard or protect) who, through various magic, help her to act out her real feelings. This is obviously aimed at a pre-teen audience, but it’s sooo cute, and my 6 year old daughter is already begging me to let her watch the next episode.
Clannad is aimed at a much older audience (late teens to adult probably). It’s a high school drama based on a manga that’s been out for a while and is really popular, apparently (I’ve heard about it all over the place even before the anime was announced). I’ve never read the manga, and have no idea what the storyline is about other than what’s on the short descriptions on the various anime sites, so I have no idea what I’m in for. The story description on Wikipedia makes me wonder if I’ll really like it (it doesn’t sound like something I would normally watch from that description), but having seen the first two episodes, it’s certainly intriguing enough to draw me in and make me want to continue watching. We’ll see how it goes. So far it’s really good. Hopefully it’ll turn out better than School Days did (which had a very good story until the last two episodes when everything sort of went crazy and it had a very freaky ending).
-
stuff
