Matthew Palmer: Retitling Blog Posts
I can't believe that Andrew Pollock didn't go with What a load
of Pollocks for his recent blog post. I'm most disappointed.
I can't believe that Andrew Pollock didn't go with What a load
of Pollocks for his recent blog post. I'm most disappointed.
For part of the 32 hours I spent inside a plane crossing the Atlantic, I
watched Recount.
I wanted Sarah to see it as well, so we got it with NetFlix and watched it
tonight.
I have no idea how factually accurate the movie is, but I'll assume all of
the bits that were on the public record were vaguely correct. I'll assume
the chad factor with
the ballots was correct. The movie portrayed the US electoral system as a
complete and utter shambles. The thing I found most striking was that the
recounts keep having deadlines that weren't met, and the whole recount got
discarded as a result. The process seemed incredibly flawed.
It got me wondering how the Australian vote counting goes, given it's
completely manual. I might have to volunteer as a scrutineer when we wind up
back in Canberra.
I highly recommend watching this movie if you can get your hands on it.
I got a voicemail message at home from UPS for a Christopher Pollock. I've
previously received some mail for him as well, so I went searching, and it
turns out that he lives elsewhere in my apartment complex.
Not long after we moved to the US, we were picking up a pizza from Pizza
Hut, and someone else was also picking up a pizza for "Pollock". It seems my
last name is more common than I thought.
Neil:
$ host -t txt manpages.debian.net manpages.debian.net descriptive text "Javier Pen~a <jfs@debian.org>" manpages.debian.net descriptive text "PGP D6 91 BB 89 88 7A F7 A0 BA E4 27 FD 7A 6A FE DA" manpages.debian.net descriptive text "PGP 86AA DB04 D2F5 CC55 3A4D 940F B1A9 DD82 DC81 4B09"
I'm wading through my email and saw a recent announcement about the
miniconfs that have been accepted for linux.conf.au 2009, and Debian was
conspicuous in its absence. That's a bit of a shame.
We're heading off on a vacation, the first real "put our feet up" kind of
vacation since our belated honeymoon a couple of years ago.
Our ultimate destination is the Seychelles, a tiny set of islands off the
east coast of Africa, near Madagascar. Sarah's aunty and uncle live here.
Her uncle's the deputy commissioner for the United Nations Indian Ocean Tuna
Commission.
Getting there is pretty heinous. We're flying to Houston, then to Dubai,
then to the Seychelles. Originally we were leaving tomorrow, but Sarah
noticed that we only had an hour to transfer in Houston, and when she
checked with the airline, they agreed (despite them booking the entire trip
in the first place). So now we've changed to a flight today, with a night in
a hotel in Dubai. Emirates only fly to the Seychelles a couple of times a
week, so we really don't want to miss the Dubai-Seychelles leg. The flight
from Houston to Dubai is 15 hours. That's worse than San Francisco to
Sydney. Not looking forward to sitting on a plane for that long.
Flying on September 11 is an interesting experience so far. It seems that
the airlines are "randomly selecting" all passengers who aren't US citizens
for secondary inspections at the security checkpoints. I was vaguely aware,
but had largely forgotten, that if they print SSSS in the bottom corner of
your bording pass, then you've been randomly selected. The TSA guy was very
apologetic, as he proceeded to completely dismantle our carry-on luggage.
The other fun aspect is that hurricane Ike is busy making its way towards
Houston. It's not supposed to make landfall until Saturday, but since
they've already evacuated some cities south of Houston, I'm really hoping it
doesn't bugger up our flights.
We're in the Seychelles for 10 days, and then we're going to spend 3 days in
Dubai on the way back. It'll be during Ramadan, which is apparently an
unpopular time for Westerners, so all of the hotels are nice and cheap. It
looks like it's going to be pretty hot as well.
I saw that the inimitable Steve Kemp has cranked
out an RBL
for hosts that do brute force SSH attacks within seconds of thinking
about doing it.
As I have a passing interest in such things, and discovered that his sample
submission client is really more geared towards people using iptables with
the LOG target (and I use the ULOG target), I cranked out this moderately
flexible Python client for logs written by ulogd this evening.
If you use Netfilter rules like mine, you can use my script after
/var/log/ulog/syslogemu.log gets rotated with:
$ ./report_iptables_ulog.py --log /var/log/ulog/syslogemu.log.1 --prefix SSH_brute_force
It's hard to believe, but it's a year ago today since the worst day of my
life.
In so many ways, it feels like so much longer, yet the memories still haunt
me.
I was playing racquet ball with Kendall yesterday, and I was running
backwards, towards the front wall to try and get behind the ball, and my
right ankle rolled underneath me as I was putting my right foot down.
There was a really nasty internal crunching sound, and it hurt like hell, so
I thought that maybe I'd broken it. I was able to walk home from the Club,
so I figured it wasn't broken, but the swelling was pretty bad.
Several hours later at Urgent Care, they confirmed with an x-ray that
it wasn't broken, so it's now strapped up and in an ankle brace. I have to
keep off it for 10 days.
I seem to have pretty bad ankles in terms of rolling. I quite often stop
myself from stepping down on my foot when the ankle's rolled inwards
slightly. Back in May I rolled my left ankle a bit, just crossing the
street, and it's still sore.
John Goerzen
writes an entertaining piece on his ongoing battle with Dell to stop
receiving advertising from them.
(The comments on the post are rather interesting in themselves)
I say "good on you, John" for taking a stand. Now that I know of these
prohibitory orders, I might try them out on Pennysaver so I stop getting
their stupid catalogs.
A while ago our junk mail situation got completely out of hand, and at the
time I found 41pounds.org, which for a
very reasonable $41, did a very good job of stopping pretty much all of the
addressed spam.
About the only thing we get now is a small wad of catalogs that USPS
delivers to every resident, which isn't explicitly address to us by name,
just the apartment number. That and these stupid "checks" from Washington
Mutual that we can use against our credit card account with them. They
give me the screaming heebie jeebies, and I shred them on sight, because
I've seen how insecure cheques are.
Eddy Petri or wrote
about doing some sort of user supported thing for snapshot.debian.net, which I was
saddened to learn from his post had reached capacity in May.
I think rather than relying on the generosity of random users, the Debian
Project should just throw some money at the problem. It's got more than
enough. Disk is cheap.
I'd think that with a few fully loaded SR2421's and an
Ethernet switch, you'd be laughing.
Somehow (thankfully) I've managed to avoid much to do with backups in my
career as a sysadmin. It's always been someone else's responsibility, that's
been fine by me.
For some reason, I've been more paranoid lately, and with all of the crazy
shit I've been doing to daedalus, I wanted to have a backup, just
in case. Maybe it's old age.
So a while ago, I went and had a stab at setting up Amanda, since pretty much whenever I've
been near a bunch of sysadmins and someone asks for a recommendation on what
to use, "Amanda" is the answer.
I jumped through all of the hoops to have caesar be the backup
server, and daedalus be a client. I set up a virtual tape library
on caesar. I think I then lost interest for a while, until I was
going to upgrade daedalus to Etch, and I wanted to have a
successful backup first.
I had the holding disk and the virtual tape library on the same filesystem,
and it was nice and big, yet I kept running into this error about
insufficient space. "dump larger than available tape space" was the
complaint.
I tried making the filesystem larger. I tried adding more virtual tapes.
Nothing would placate it. After another round of Googling, I discovered that
the problem was because Amanda would only only span a maximum of four tapes,
and the filesystem I was trying to backup was way bigger than that. So I
told it to use up to 10 tapes in a run (runtapes 10) and then lo
and behold, I got a successful backup out of the sucker.
So that was mostly all good, but occasionally a backup would fail, or fail
once an succeed on a retry, because tar exited non-zero, with "file changed
as we read it". I did some more Googling tonight, and discovered this
Amanda/GNU tar compatibility matrix.
It turns out that the version of tar and the version of Amanda shipped in
Debian Etch aren't compatible with each other. Suck! Oh wait, they're
maintained by the same person! Argh!
So aside from the occasional backup failing due to a file being written to
at the time of the backup, it's working fairly well. My DSL link isn't the
fastest, so a full backup of daedalus can take over 48 hours to
complete, but I'm not paying for inbound bandwidth at home, so it's not a
big deal.
I initially thought that to do a restore, I'd have to run
amrecover on daedalus, and then I'd have to stream the
contents of multiple tapes back to it at the appalling uplink speed of my DSL
connection, but fortunately amrecover has a sethost
command, so I can restore just the files I need on another machine local to
caesar at LAN speeds, and just transfer the files I restore to
daedalus. So that's all pretty neat.
I quite like the FTP-client-like interface of amrecover. It's cool
how you can use setdate and just go back in time.
I had the opportunity to do a test restore recently when one of my users
nuked his Maildir directory by accident. Unfortunately I was unable to
recover the needed files in that case, and I'm not entirely sure what the
problem was, so there's some more testing to be done before I'll be fully
confident that I can rely on it.
There's a lot of configuration options available as well, and I think I got
the config I'm currently using from a template or something, so there's
probably a fair bit of tweaking I can do.
Oh, and I'm constantly reminded of this quote from a previous life, where
technical details bubbled too far up non-technical management:
"Who is Amanda, and what is her role in the backups?"
One of the things I've wanted to do since I discovered that Caltrain (and the VTA light rail) are well-equipped to take
bikes, was go for an outing further afield on our bikes using one of these
methods of public transport.
So we finally got around to doing that yesterday, when we threw our bikes on
the train, and BART, and went for a ride
around Golden
Gate Park.
The grand plan was to bike to the Caltrain station, take Caltrain to
Milbrae, BART to downtown, then bike from there to the park, and then catch
an afternoon showing of The
Dark Knight in IMAX at the Metreon, then bike back to the
BART station, and reverse the whole thing.
We had a minor hiccup in that Sarah had just gotten her bike back from a
friend she'd lent it to, and as we were about to head to the Caltrain
station, discovered that it a broken spoke. She managed to extract the spoke
and we got to the station with a couple of minutes to spare.
It turned out that Mike's Bikes were
fairly close to the BART station, so we stopped in there on way to the park,
and they were able to fix the spoke on the spot in 20 minutes for the
princely sum of $21.
Unfortunately the weather was pretty typical summer San Francisco weather,
and it was quite cold and foggy. It didn't get any better as we got closer
to the park (and the coast), so the ride around the park itself wasn't
terribly exciting. The hill we had to get over (Hayes Street) was a bit of a
slog. On the way back we saw the filming of a Japanese Nissan commercial
(complete with right-hand drive car).
I'd done some research into bike parking beforehand, and found that all of
the parking garages are required to provide bicycle
parking, and there's a pretty comprehensive
list.
So we parked our bikes in the Moscone Center garage, which was about a block
from the Metreon, and went and caught the movie.
The Dark Knight was really good. Very... dark. I really liked Ledger's
portrayal of the Joker, much better than the original. I wonder if they'll
just keep re-imagining the Batman movies over and over? There was pretty good
continuity from Batman
Begins. I thought the voice of Batman was a bit ridiculous though, at
least initially. You kind of got used to it after a while.
Anyway, we got back to the parking garage, and some lowlife had cut through
the cable lock that we'd used to lock our bikes to each other and the rack,
and stolen Sarah's bike! Annoyed does not begin to describe it.
We trundled off the seven or eight blocks to the Hall of Justice to file a
report. The SFPD weren't the least bit interested, really. The somewhat
astounding thing was the bike racks were right behind the cashier's office,
and there was video surveillance, and someone still managed to just walk
right in, cut the cable, and walk out with the bike. We checked with the
duty manager, and he reviewed the video footage and said they have footage
of him walking in and then walking out with the bike. So hopefully the
police will get hold that.
We have to see if we can dredge up the purchase documentation for the bike
and see if the serial number is recorded. I have no idea if we'll ever see
the bike again. Our renter's insurance may cover it, but I suspect the
deductible won't make it worth our while to make a claim.
I have to say (and it's not like it's a recent discovery or anything) that
the Bay Area's public transportation is a joke. It cost us $32 to go from
Mountain View to Downtown San Francisco, via Caltrain and then BART. Our
Prius takes about 11 gallons at worst, and with gas prices being at say
$4.30 a gallon, that's about $47 a tank. We get about 400 miles out of a
tank usually, and it's vaguely 40 miles from home to the guts of Golden Gate
Park. So it's cheaper, and faster to drive to San Francisco than it is to
use a bicycle and public transport. Particularly when you factor in the risk
of theft of your bicycle.
Our next bike lock will be one of those fancy Kevlar ones.
Well that happened,
and uneventfully as well.
Now if only IntaServe actually
made it possible to update glue records, everything would be done...
What is it with domain registrars and glue records? Are glue records that
much of an edge case these days that no one bothers having an interface for
managing them?
I've found a new colo for my server, and Brent has kindly offered to do the
remote hands work for me tomorrow. Fingers crossed it'll be fairly smooth...
I just became aware of this electric
lawn mower via Woot!
I never minded mowing the lawn, when it was a decent kind of yard (i.e.
level). This mower looks like it'd take the fun to a whole new level: hardly
any noise, and no smoke.
I just need a yard now...
So hot off the back of my successful
upgrade from Sarge to Etch, I thought I'd try the new "Etch and a Half"
2.6.24 kernel on daedalus
Let's just say that it's times like these that I'm glad I got the
serial-over-LAN thing working, and have remote power.
I spent many hours today rebooting, trying to get to the bottom of why it
wouldn't work.
At first, I thought it was hanging, but then I realised it was dropping to a
shell in the initramfs, and just not making that very obvious because the
serial over LAN console is a bit crappy.
Once I realised what was going on, I did some poking around.
It seemed like udev wasn't getting started properly, so when it
went to assemble the mirrors, that failed, because the component devices
couldn't be found, then it freaked out because it couldn't mount the root
filesystem.
If I ran the relevant bits of /scripts/init-premount/udev by hand,
the SCSI devices appeared, and I could manually assemble the mirror, and
mount the root filesystem, which was handy, because I also discovered that
you can boot with debug on the kernel command line, and it logs the
initramfs run to /tmp/initramfs.debug. So that was a convenient way
of preserving the log, because there seemed to be some characters in it that
made inspecting it over the serial console difficult, and there was no way
to get it off the machine from the initramfs environment.
As far as I can determine, it's telling udev to start, but it certainly
isn't still running when it bombs out to a shell after failing to mount the
root filesystem. It's not immediately clear if there's something later on
that is stopping it again. I've put the log here in
case anyone's interested in looking at it. This was with the addition of "x"
to the options of the shebang line of
/usr/share/initramfs-tools/scripts/init-premount/udev so that I
could see what was happening when /scripts/init-premount/udev ran.
So I don't think the kernel itself is at fault, it's some sort of weird
udev/initramfs interaction.
Rather than further rebooting the tripe out of daedalus, I'll have
to see if I can reproduce the problem on a less important machine locally to
do further debugging. In the meantime, I'll have to stick with 2.6.18.
Well that
certainly wasn't worth the amount of angst I'd pre-allocated it.
The upgrade went reasonably smoothly. Lowlights included:
Next.