Some of my recent time has been devoted to making our boot media more Mac friendly, which has entailed rather a lot of rebooting. This would have been fine, if tedious, except that some number of boots would fall over with either a clearly impossible kernel panic or userspace segfaulting in places that made no sense. Something was clearly wrong. Crashes that shouldn't happen are generally an indication of memory corruption. The question is how that corruption is being triggered. Hunting that down wasn't terribly easy.
My first thought was that we were possibly managing to load the kernel over a region used by UEFI code. UEFI defines two types of code - boot services and runtime services. While runtime services code and data must be preserved by the OS, in theory boot services code and data is available to the OS once the firmware has exited. In practice,
that's not true. It seemed entirely possible that the kernel might be ending up on top of some of that boot services code or data and getting trodden on. Grub now has
code to avoid putting the kernel on boot services, so testing the latest code seemed like a good plan. But no, crashes still happened.
That pretty much ruled out the bootloader. My next thought was that executing some of the firmware code was triggering a write to some other memory that contained the kernel.
Josh Boyer suggested the next trick, which was to try marking the kernel read-only to see whether anything was hitting it. x86 lets you mark pages as read-only - any attempt to write to them should take a fault. UEFI functions are executed in the context of the kernel, so share the same page tables. That let me rule this out, since everything still went just as wrong and I wasn't taking an extra fault first.
However, at this point I was reasonably happy that it wasn't the kernel itself being overwritten - faults were occurring in userspace code as well. That was a pretty strong indication that what was happening was continuing to happen once userspace had started, so it wasn't a direct response to a firmware call. I made sure of that by stubbing out all the calls that could be triggered after kernel initialisation, and saw the same failures. Once all attempts to be clever have failed, it's time to just start using brute force. The kernel lets you reserve areas of RAM by passing arguments like
memmap=0xlength$0xstart to block
length bytes starting at
start from being used. It took a while, but I finally found a 256MB range that made a difference - reserving it resulted in the machine booting reliably, letting the OS use it resulted in occasional crashes.
Definite progress. Comparing that memory range to the EFI memory map was helpful. There were several blocks of UEFI boot services data present there, which really seemed like too much of a coincidence. By reserving each of them in turn, I'd traced it down to a single 31MB region of boot service data - that is, memory reserved by the firmware for use by the UEFI boot services. Per spec, this is available to the OS once the boot environment has been exited. Nothing other than the OS should be touching this after boot, but something clearly was. Tracking down what was far easier than I expected, although the first attempt was a failure. Setting it read-only should have triggered a fault, but didn't. That was rather confusing. But, rather than give up, I patched the kernel to fill the region with 0xff at kernel init. Then I booted the system, read it back and looked for values that weren't 0xff. I got this:
00000000 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
*
001568a0 ff ff ff ff ff ff ff ff ff ff ff ff 84 00 00 00 ................
001568b0 00 20 a7 ac 46 00 00 00 00 00 00 00 00 00 06 01 . ..F...........
001568c0 c2 0b 0c 00 ff ff ff ff ff ff ff ff ff ff ff ff ................
001568d0 ff ff 0a 04 f0 03 82 0d 40 00 00 00 ff ff ff ff ........@.......
001568e0 ff ff 00 21 00 36 9a 80 ff ff ff ff ff ff 00 7e ...!.6.........~
001568f0 00 09 43 48 41 2d 47 75 65 73 74 01 04 02 04 0b ..CHA-Guest.....
00156900 16 32 08 0c 12 18 24 30 48 60 6c 2d 1a 0e 18 1a .2....$0H l-....
00156910 ff ff 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00156920 00 00 00 00 00 00 00 dd 09 00 10 18 02 00 10 01 ................
00156930 00 00 dd 1e 00 90 4c 33 0e 18 1a ff ff 00 00 00 ......L3........
00156940 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00156950 00 00 bd ea f8 b3 ff ff ff ff ff ff ff ff ff ff ................
00156960 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
*
022fb000
That's a lot of 0xffs (around 31MB of them) with one small section that contains an 802.11 probe packet with the SSID of the hospital across the road from my house. Apple support network booting off wireless networks. It seems that the firmware brought up the wireless card, associated with this network (it's the only public one nearby) and then left the card DMAing packets into RAM. The read-only page attribute only applies to CPU-initiated accesses, so it could do this without triggering a page fault. It also explained why it was so random - whether memory corruption occurred would depend on whether a packet appeared between that memory being used by the OS and the kernel reinitialising the wireless card. It certainly explains why I couldn't reproduce it when I left the machine repeatedly rebooting on the bus home.
How do we fix this? Unsure. With luck disconnecting the UEFI driver in the bootloader should quiesce the hardware, but without testing I'm not sure of that yet. For now it's just another example of firmware managing to break expectations in deeply strange ways.
comments
I first learned that David
Hicks had written an autobiography
when I saw some press coverage of there being an attempt by the Australian
Government to ban the sale of it, so I asked my sister if she could get me a
copy for my birthday before any ban happened. The book doesn't appear to be
available on Amazon.
I sat down and tried to read it almost immediately, but at the time found his
writing style to be rather inarticulate, so I put the book down for a month or
so. When I next had a platelet donation to make, I took it along with me, and
forced myself to read it for duration of the donation.
On the second reading, I managed to engage with the book, and from that point
on, once I made it through his early years in that initial forced reading
session, it ended up being quite enthralling. I plugged my way through the 456
pages in a bit over a month I guess.
We'll never truly know the full story of David Hicks, because I'm sure that
unless there's some sort of inquiry (and even then, there's multiple country's
governments involved), the full story won't come out.
I was always sympathetic to Hicks' plight, since I don't believe that anyone
should be held without due process let alone for anywhere near as long as he
was.
Certainly, the way the book reads, Hicks comes across as a very young, naive,
but principled individual. Like many Australians, he got the travel bug, and by
way of it, got to interact with some different cultures, which awakened a
hunger for social justice in him (at least that's my interpretation of his
portrayal).
I guess I have a different mental image of torture than Hicks did, because I
didn't really find any incidences of what I'd consider torture in his
story. There was plenty of despicable, inhumane treatment though, so I can see
how the line might get blurry, particularly if you're the one experiencing it.
Specifically on the (mis)treatment of the detainees: it was pretty bad, based
on the book. I'm surprised there haven't been more whistle blowers that have
come out. I can't believe that the many soldiers who were stationed there could
treat detainees the way Hicks says they were treated and then can look
themselves in the mirror. The ICRC seemed well apprised of the
detainees conditions (since they seemed to have the most access to them of any
non-military people). I don't know what confidentiality is imposed on them in
return for that access, though.
The fact that he's able to string a sentence together at all after the
treatment he describes for the length of time he received it, I think is pretty
amazing. I couldn't imagine having to endure what he says he went through.
I did feel quite appalled by my own country's complete disregard for his
plight, and similarly in the way the US Government treated these detainees for
as long as they did. Given there are still 171 detainees in Guantanamo today,
it makes me wonder what the conditions and treatment are like there now. It
certainly seems to have fallen out of the mainstream press.
Overall, not the most well written book, but a fascinating and shocking account
of what happened to one person. I think Hicks should be congratulated for the
courage to relive his experiences to get them down on paper, and Random House
should also be congratulated for publishing his work, which I'm sure put them
under some political pressure.
I have no idea how well the book has sold, or how widely it has been read, but
I certainly recommend it.
since I was travelling in the pre-ultimate week for a few days, there was no
report about my RC bugs fixing activities. so here we go with an overview
covering two weeks:
Now we have:
Do you think there's a way to provide a unified version?
The Debian Project s annual conference is slowly coming closer. Visiting places around the world since 2000, DebConf will take place this year in Managua, Nicaragua, in July and will again provide an opportunity for people who contribute to Debian to meet in person for face-to-face collaboration and discussions.
At DebConf we try to bring together as many Debian contributors as possible, including those who could not afford to attend from their own resources. You can help make DebConf12 a success by your organisation becoming a sponsor, or by donating money as an individual.
Individual 
I ve 
Continuing at same pace (one release every two days), I can announce 
A new version of the 
I was playing with my new Laptop (a 
