Cron job output overload

We have a mailing list at Mozilla which receives mail sent to root at any of our servers. The majority of this mail is cron job output. I have filters set up in my Zimbra account to filter the cron job mail specifically into a folder separate from the rest of the mail to that mailing list. I was on vacation last week, and the last day before I left, I completely deleted the contents of that folder. On my return, that folder contained 26,373 messages in it, all dated within the last week. Trying to separate the nuisance mail from the real problems is kind of impossible by hand with that volume.

Obviously one task is to eliminate the nuisance mail. This has to be done carefully, because typically you still want to get errors from cron jobs, but you don’t want the general output. And not all jobs are good about their use of standard error and standard output, so often you can’t just devnull the standard out and expect to only get mail when there’s a problem. So fixing the nuisance mail sometimes means writing a wrapper script for a cron job that does some grep or awk work to filter the output. But even with the nuisance mail gone, it’s a lot of mail to sift through to find any possible real problems.

So, I filed bug 377043 with an idea for a tool to do some automated analysis of all this cron job output. Keep track of patterns and point out things that need looking at, etc. Unfortunately both cron jobs and data analysis are pretty popular topics (and usually not related to each other) so Google isn’t helping me much trying to search for existing tools. Does anyone know of any existing tools that do something similar to this that we might either be able to use, or build upon?

Vacation and work travel

So this last week I’ve been on vacation, but just hanging out at home hoping to catch up on some things.  One of the projects I’ve been working on this week is trying to write a driver for lirc to use a USB-attached IR receiver on Mac OS X.  One of my MythTV boxes is running on Mac OS X, and it’s annoying to have the little white Apple remote be the only one that works on it (it’s a nice simple remote, but there’s just not enough buttons on it to be useful for a full entertainment center).  I’ve been hoping to get that driver working before I left so my wife could use a real remote while I’m gone.  Not quite there yet, not much time left.  I made major progress on it this afternoon though while the kids were watching the new movies they got in their Easter baskets.

Tomorrow afternoon (Monday) I leave to head out to Mountain View for our quarterly all-hands meeting at Mozilla.  The following week I’ll be attending an Asterisk training program put on by Digium in San Jose (teach me everything I need to know to run the PBX system at Mozilla), so I’ll be away from home for two weeks.  It’s always fun visiting Mozilla, but it’s not going to be fun being away from the family for that long.