Search Results: "shaka"

Review: The Unbroken, by C.L. Clark

Series:	Magic of the Lost #1
Publisher:	Orbit
Copyright:	March 2021
ISBN:	0-316-54267-9
Format:	Kindle
Pages:	490

The Unbroken is the first book of a projected fantasy trilogy. It is C.L. Clark's first novel. Lieutenant Touraine is one of the Sands, the derogatory name for the Balladairan Colonial Brigade. She, like the others of her squad, are conscript soldiers, kidnapped by the Balladairan Empire from their colonies as children and beaten into "civilized" behavior by Balladairan training. They fought in the Balladairan war against the Taargens. Now, they've been reassigned to El-Wast, capital city of Qaz l, the foremost of the southern colonies. The place where Touraine was born, from which she was taken at the age of five. Balladaire is not France and Qaz l is not Algeria, but the parallels are obvious and strongly implied by the map and the climates. Touraine and her squad are part of the forces accompanying Princess Luca, the crown princess of the Balladairan Empire, who has been sent to take charge of Qaz l and quell a rebellion. Luca's parents died in the Withering, the latest round of a recurrent plague that haunts Balladaire. She is the rightful heir, but her uncle rules as regent and is reluctant to give her the throne. Qaz l is where she is to prove herself. If she can bring the colony in line, she can prove that she's ready to rule: her birthright and her destiny. The Qaz li are uninterested in being part of Luca's grand plan of personal accomplishment. She steps off her ship into an assassination attempt, foiled by Touraine's sharp eyes and quick reactions, which brings the Sand to the princess's attention. Touraine's reward is to be assigned the execution of the captured rebels, one of whom recognizes her and names her mother before he dies. This sets up the core of the plot: Qaz li rebellion against an oppressive colonial empire, Luca's attempt to use the colony as a political stepping stone, and Touraine caught in between. One of the reasons why I am happy to see increased diversity in SFF authors is that the way we tell stories is shaped by our cultural upbringing. I was taught to tell stories about colonialism and rebellion in a specific ideological shape. It's hard to describe briefly, but the core idea is that being under the rule of someone else is unnatural as well as being an injustice. It's a deviation from the way the world should work, something unexpected that is inherently unstable. Once people unite to overthrow their oppressors, eventual success is inevitable; it's not only right or moral, it's the natural path of history. This is what you get when you try to peel the supremacy part away from white supremacy but leave the unshakable self-confidence and bedrock assumption that the universe cares what we think. We were also taught that rebellion is primarily ideological. One may be motivated by personal injustice, but the correct use of that injustice is to subsume it into concepts such as freedom and democracy. Those concepts are more "real" in some foundational sense, more central to the right functioning of the world, than individual circumstance. When the now-dominant group tells stories of long-ago revolution, there is no personal experience of oppression and survival in which to ground the story; instead, it's linked to anticipatory fear in the reader, to the idea that one's privileges could be taken away by a foreign oppressor and that the counter to this threat is ideological unity. Obviously, not every white fantasy author uses this story shape, but the tendency runs deep because we're taught it young. You can see it everywhere in fantasy, from Lord of the Rings to Tigana. The Unbroken uses a much different story shape, and I don't think it's a coincidence that the author is Black. Touraine is not sympathetic to the Qaz li. These are not her people and this is not her life. She went through hell in Balladairan schools, but she won a place, however tenuous. Her personal role model is General Cantic, the Balladairan Blood General who was also one of her instructors. Cantic is hard as nails, unforgiving, unbending, and probably a war criminal, but also the embodiment of a military ethic. She is tough but fair with the conscript soldiers. She doesn't put a stop to their harassment by the regular Balladairan troops, but neither does she let it go too far. Cantic has power, she knows how to keep it, and there is a place for Touraine in Cantic's world. And, critically, that place is not just hers: it's one she shares with her squad. Touraine's primary loyalty is not to Balladaire or to Qaz l. It's to the Sands. Her soldiers are neither one thing nor the other, and they disagree vehemently among themselves about what Qaz l and their other colonial homes should be to them, but they learned together, fought together, and died together. That theme is woven throughout The Unbroken: personal bonds, third and fourth loyalties, and practical ethics of survival that complicate and contradict simple dichotomies of oppressor and oppressed. Touraine is repeatedly offered ideological motives that the protagonist in the typical story shape would adopt. And she repeatedly rejects them for personal bonds: trying to keep her people safe, in a world that is not looking out for them. The consequence is that this book tears Touraine apart. She tries to walk a precarious path between Luca, the Qaz li, Cantic, and the Sands, and she falls off that path a lot. Each time I thought I knew where this book was going, there's another reversal, often brutal. I tend to be a happily-ever-after reader who wants the protagonist to get everything they need, so this isn't my normal fare. The amount of hell that Touraine goes through made for difficult reading, worse because much of it is due to her own mistakes or betrayals. But Clark makes those decisions believable given the impossible position Touraine is in and the lack of role models she has for making other choices. She's set up to fail, and the price of small victories is to have no one understand the decisions that she makes, or to believe her motives. Luca is the other viewpoint character of the book (and yes, this is also a love affair, which complicates both of their loyalties). She is the heroine of a more typical genre fantasy novel: the outsider princess with a physical disability and a razor-sharp mind, ambitious but fair (at least in her own mind), with a trusted bodyguard advisor who also knew her father and a sincere desire to be kinder and more even-handed in her governance of the colony. All of this is real; Luca is a protagonist, and the reader is not being set up to dislike her. But compared to Touraine's grappling with identity, loyalty, and ethics, Luca is never in any real danger, and her concerns start to feel too calculated and superficial. It's hard to be seriously invested in whether Luca proves herself or gets her throne when people are being slaughtered and abused. This, I think, is the best part of this book. Clark tells a traditional ideological fantasy of learning to be a good ruler, but she puts it alongside a much deeper and more complex story of multi-faceted oppression. She has the two protagonists fall in love with each other and challenges them to understand each other, and Luca does not come off well in this comparison. Touraine is frustrated, impulsive, physical, and sometimes has catastrophically poor judgment. Luca is analytical and calculating, and in most ways understands the political dynamics far better than Touraine. We know how this story usually goes: Luca sees Touraine's brilliance and lifts her out of the ranks into a role of importance and influence, which Touraine should reward with loyalty. But Touraine's world is more real, more grounded, and more authentic, and both Touraine and the reader know what Luca could offer is contingent and comes with a higher price than Luca understands. (Incidentally, the cover of The Unbroken, designed by Lauren Panepinto with art by Tommy Arnold, is astonishingly good at capturing both Touraine's character and the overall feeling of the book. Here's a larger version.) The writing is good but uneven. Clark loves reversals, and they did keep me reading, but I think there were too many of them. By the end of the book, the escalation of betrayals and setbacks was more exhausting than exciting, and I'd stopped trusting anything good would last. (Admittedly, this is an accurate reflection of how Touraine felt.) Touraine's inner monologue also gets a bit repetitive when she's thrashing in the jaws of an emotional trap. I think some of this is first-novel problems of over-explaining emotional states and character reasoning, but these problems combine to make the book feel a bit over-long. I'm also not in love with the ending. It's perhaps the one place in the book where I am more cynical about the politics than Clark is, although she does lay the groundwork for it. But this book is also full of places small and large where it goes a different direction than most fantasy and is better for it. I think my favorite small moment is Touraine's quiet refusal to defend herself against certain insinuations. This is such a beautiful bit of characterization; she knows she won't be believed anyway, and refuses to demean herself by trying. I'm not sure I can recommend this book unconditionally, since I think you have to be in the mood for it, but it's one of the most thoughtful and nuanced looks at colonialism and rebellion I can remember seeing in fantasy. I found it frustrating in places, but I'm also still thinking about it. If you're looking for a political fantasy with teeth, you could do a lot worse, although expect to come out the other side a bit battered and bruised. Followed by The Faithless, and I have no idea where Clark is going to go with the second book. I suppose I'll have to read and find out. Content note: In addition to a lot of violence, gore, and death, including significant character death, there's also a major plague. If you're not feeling up to reading about panic caused by contageous illness, proceed with caution. Rating: 7 out of 10

Review: Maskerade, by Terry Pratchett

Series:	Discworld #18
Publisher:	Harper
Copyright:	1995
Printing:	February 2014
ISBN:	0-06-227552-6
Format:	Mass market
Pages:	360

Maskerade is the 18th book of the Discworld series, but you probably could start here. You'd miss the introduction of Granny Weatherwax and Nanny Ogg, which might be a bit confusing, but I suspect you could pick it up as you went if you wanted. This is a sequel of sorts to Lords and Ladies, but not in a very immediate sense. Granny is getting distracted and less interested in day-to-day witching in Lancre. This is not good; Granny is incredibly powerful, and bored and distracted witches can go to dark places. Nanny is concerned. Granny needs something to do, and their coven needs a third. It's not been the same since they lost their maiden member. Nanny's solution to this problem is two-pronged. First, they'd had their eye on a local girl named Agnes, who had magic but who wasn't interested in being a witch. Perhaps it was time to recruit her anyway, even though she'd left Lancre for Ankh-Morpork. And second, Granny needs something to light a fire under her, something that will get her outraged and ready to engage with the world. Something like a cookbook of aphrodisiac recipes attributed to the Witch of Lancre. Agnes, meanwhile, is auditioning for the opera. She's a sensible person, cursed her whole life by having a wonderful personality, but a part of her deep inside wants to be called Perdita X. Dream and have a dramatic life. Having a wonderful personality can be very frustrating, but no one in Lancre took either that desire or her name seriously. Perhaps the opera is somewhere where she can find the life she's looking for, along with another opportunity to try on the Perdita name. One thing she can do is sing; that's where all of her magic went. The Ankh-Morpork opera is indeed dramatic. It's also losing an astounding amount of money for its new owner, who foolishly thought owning an opera would be a good retirement project after running a cheese business. And it's haunted by a ghost, a very tangible ghost who has started killing people. I think this is my favorite Discworld novel to date (although with a caveat about the ending that I'll get to in a moment). It's certainly the one that had me laughing out loud the most. Agnes (including her Perdita personality aspect) shot to the top of my list of favorite Discworld characters, in part because I found her sensible personality so utterly relatable. She is fascinated by drama, she wants to be in the middle of it and let her inner Perdita goth character revel in it, and yet she cannot help being practical and unflappable even when surrounded by people who use far too many exclamation points. It's one thing to want drama in the abstract; it's quite another to be heedlessly dramatic in the moment, when there's an obviously reasonable thing to do instead. Pratchett writes this wonderfully. The other half of the story follows Granny and Nanny, who are unstoppable forces of nature and a wonderful team. They have the sort of long-standing, unshakable adult friendship between very unlike people that's full of banter and minor irritations layered on top of a deep mutual understanding and respect. Once they decide to start investigating this supposed opera ghost, they divvy up the investigative work with hardly a word exchanged. Planning isn't necessary; they both know each other's strengths. We've gotten a lot of Granny's skills in previous books. Maskerade gives Nanny a chance to show off her skills, and it's a delight. She effortlessly becomes the sort of friendly grandmother who blends in so well that no one questions why she's there, and thus manages to be in the middle of every important event. Granny watches and thinks and theorizes; Nanny simply gets into the middle of everything and talks to everyone until people tell her what she wants to know. There's no real doubt that the two of them are going to get to the bottom of anything they want to get to the bottom of, but watching how they get there is a delight. I love how Pratchett handles that sort of magical power from a world-building perspective. Ankh-Morpork is the Big City, the center of political power in most of the Discworld books, and Granny and Nanny are from the boondocks. By convention, that means they should either be awed or confused by the city, or gain power in the city by transforming it in some way to match their area of power. This isn't how Pratchett writes witches at all. Their magic is in understanding people, and the people in Ankh-Morpork are just as much people as the people in Lancre. The differences of the city may warrant an occasional grumpy aside, but the witches are fully as capable of navigating the city as they are their home town. Maskerade is, of course, a parody of opera and musicals, with Phantom of the Opera playing the central role in much the same way that Macbeth did in Wyrd Sisters. Agnes ends up doing the singing for a beautiful, thin actress named Christine, who can't sing at all despite being an opera star, uses a truly astonishing excess of exclamation points, and strategically faints at the first sign of danger. (And, despite all of this, is still likable in that way that it's impossible to be really upset at a puppy.) She is the special chosen focus of the ghost, whose murderous taunting is a direct parody of the Phantom. That was a sufficiently obvious reference that even I picked up on it, despite being familiar with Phantom of the Opera only via the soundtrack. Apart from that, though, the references were lost on me, since I'm neither a musical nor an opera fan. That didn't hurt my enjoyment of the book in the slightest; in fact, I suspect it's part of why it's in my top tier of Discworld books. One of my complaints about Discworld to date is that Pratchett often overdoes the parody to the extent that it gets in the way of his own (excellent) characters and story. Maybe it's better to read Discworld novels where one doesn't recognize the material being parodied and thus doesn't keep getting distracted by references. It's probably worth mentioning that Agnes is a large woman and there are several jokes about her weight in Maskerade. I think they're the good sort of jokes, about how absurd human bodies can be, not the mean sort? Pratchett never implies her weight is any sort of moral failing or something she should change; quite the contrary, Nanny considers it a sign of solid Lancre genes. But there is some fat discrimination in the opera itself, since one of the things Pratchett is commenting on is the switch from full-bodied female opera singers to thin actresses matching an idealized beauty standard. Christine is the latter, but she can't sing, and the solution is for Agnes to sing for her from behind, something that was also done in real opera. I'm not a good judge of how well this plot line was handled; be aware, going in, if this may bother you. What did bother me was the ending, and more generally the degree to which Granny and Nanny felt comfortable making decisions about Agnes's life without consulting her or appearing to care what she thought of their conclusions. Pratchett seemed to be on their side, emphasizing how well they know people. But Agnes left Lancre and avoided the witches for a reason, and that reason is not honored in much the same way that Lancre refused to honor her desire to go by Perdita. This doesn't seem to be malicious, and Agnes herself is a little uncertain about her choice of identity, but it still rubbed me the wrong way. I felt like Agnes got steamrolled by both the other characters and by Pratchett, and it's the one thing about this book that I didn't like. Hopefully future Discworld books about these characters revisit Agnes's agency. Overall, though, this was great, and a huge improvement over Interesting Times. I'm excited for the next witches book. Followed in publication order by Feet of Clay, and later by Carpe Jugulum in the thematic sense. Rating: 8 out of 10

It all started with the big bang! We nearly lost 33 of 36 disks on a Proxmox/Ceph Cluster; this is the story of how we recovered them. At the end of 2020, we eventually had a long outstanding maintenance window for taking care of system upgrades at a customer. During this maintenance window, which involved reboots of server systems, the involved Ceph cluster unexpectedly went into a critical state. What was planned to be a few hours of checklist work in the early evening turned out to be an emergency case; let s call it a nightmare (not only because it included a big part of the night). Since we have learned a few things from our post mortem and RCA, it s worth sharing those with others. But first things first, let s step back and clarify what we had to deal with. The system and its upgrade One part of the upgrade included 3 Debian servers (we re calling them server1, server2 and server3 here), running on Proxmox v5 + Debian/stretch with 12 Ceph OSDs each (65.45TB in total), a so-called Proxmox Hyper-Converged Ceph Cluster.

First, we went for upgrading the Proxmox v5/stretch system to Proxmox v6/buster, before updating Ceph Luminous v12.2.13 to the latest v14.2 release, supported by Proxmox v6/buster. The Proxmox upgrade included updating corosync from v2 to v3. As part of this upgrade, we had to apply some configuration changes, like adjust ring0 + ring1 address settings and add a mon_host configuration to the Ceph configuration. During the first two servers reboots, we noticed configuration glitches. After fixing those, we went for a reboot of the third server as well. Then we noticed that several Ceph OSDs were unexpectedly down. The NTP service wasn t working as expected after the upgrade. The underlying issue is a race condition of ntp with systemd-timesyncd (see #889290). As a result, we had clock skew problems with Ceph, indicating that the Ceph monitors clocks aren t running in sync (which is essential for proper Ceph operation). We initially assumed that our Ceph OSD failure derived from this clock skew problem, so we took care of it. After yet another round of reboots, to ensure the systems are running all with identical and sane configurations and services, we noticed lots of failing OSDs. This time all but three OSDs (19, 21 and 22) were down:

% sudo ceph osd tree
ID CLASS WEIGHT   TYPE NAME      STATUS REWEIGHT PRI-AFF
-1       65.44138 root default
-2       21.81310     host server1
 0   hdd  1.08989         osd.0    down  1.00000 1.00000
 1   hdd  1.08989         osd.1    down  1.00000 1.00000
 2   hdd  1.63539         osd.2    down  1.00000 1.00000
 3   hdd  1.63539         osd.3    down  1.00000 1.00000
 4   hdd  1.63539         osd.4    down  1.00000 1.00000
 5   hdd  1.63539         osd.5    down  1.00000 1.00000
18   hdd  2.18279         osd.18   down  1.00000 1.00000
20   hdd  2.18179         osd.20   down  1.00000 1.00000
28   hdd  2.18179         osd.28   down  1.00000 1.00000
29   hdd  2.18179         osd.29   down  1.00000 1.00000
30   hdd  2.18179         osd.30   down  1.00000 1.00000
31   hdd  2.18179         osd.31   down  1.00000 1.00000
-4       21.81409     host server2
 6   hdd  1.08989         osd.6    down  1.00000 1.00000
 7   hdd  1.08989         osd.7    down  1.00000 1.00000
 8   hdd  1.63539         osd.8    down  1.00000 1.00000
 9   hdd  1.63539         osd.9    down  1.00000 1.00000
10   hdd  1.63539         osd.10   down  1.00000 1.00000
11   hdd  1.63539         osd.11   down  1.00000 1.00000
19   hdd  2.18179         osd.19     up  1.00000 1.00000
21   hdd  2.18279         osd.21     up  1.00000 1.00000
22   hdd  2.18279         osd.22     up  1.00000 1.00000
32   hdd  2.18179         osd.32   down  1.00000 1.00000
33   hdd  2.18179         osd.33   down  1.00000 1.00000
34   hdd  2.18179         osd.34   down  1.00000 1.00000
-3       21.81419     host server3
12   hdd  1.08989         osd.12   down  1.00000 1.00000
13   hdd  1.08989         osd.13   down  1.00000 1.00000
14   hdd  1.63539         osd.14   down  1.00000 1.00000
15   hdd  1.63539         osd.15   down  1.00000 1.00000
16   hdd  1.63539         osd.16   down  1.00000 1.00000
17   hdd  1.63539         osd.17   down  1.00000 1.00000
23   hdd  2.18190         osd.23   down  1.00000 1.00000
24   hdd  2.18279         osd.24   down  1.00000 1.00000
25   hdd  2.18279         osd.25   down  1.00000 1.00000
35   hdd  2.18179         osd.35   down  1.00000 1.00000
36   hdd  2.18179         osd.36   down  1.00000 1.00000
37   hdd  2.18179         osd.37   down  1.00000 1.00000

Our blood pressure increased slightly! Did we just lose all of our cluster? What happened, and how can we get all the other OSDs back? We stumbled upon this beauty in our logs:

kernel: [   73.697957] XFS (sdl1): SB stripe unit sanity check failed
kernel: [   73.698002] XFS (sdl1): Metadata corruption detected at xfs_sb_read_verify+0x10e/0x180 [xfs], xfs_sb block 0xffffffffffffffff
kernel: [   73.698799] XFS (sdl1): Unmount and run xfs_repair
kernel: [   73.699199] XFS (sdl1): First 128 bytes of corrupted metadata buffer:
kernel: [   73.699677] 00000000: 58 46 53 42 00 00 10 00 00 00 00 00 00 00 62 00  XFSB..........b.
kernel: [   73.700205] 00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
kernel: [   73.700836] 00000020: 62 44 2b c0 e6 22 40 d7 84 3d e1 cc 65 88 e9 d8  bD+.."@..=..e...
kernel: [   73.701347] 00000030: 00 00 00 00 00 00 40 08 00 00 00 00 00 00 01 00  ......@.........
kernel: [   73.701770] 00000040: 00 00 00 00 00 00 01 01 00 00 00 00 00 00 01 02  ................
ceph-disk[4240]: mount: /var/lib/ceph/tmp/mnt.jw367Y: mount(2) system call failed: Structure needs cleaning.
ceph-disk[4240]: ceph-disk: Mounting filesystem failed: Command '['/bin/mount', '-t', u'xfs', '-o', 'noatime,inode64', '--', '/dev/disk/by-parttypeuuid/4fbd7e29-9d25-41b8-afd0-062c0ceff05d.cdda39ed-5
ceph/tmp/mnt.jw367Y']' returned non-zero exit status 32
kernel: [   73.702162] 00000050: 00 00 00 01 00 00 18 80 00 00 00 04 00 00 00 00  ................
kernel: [   73.702550] 00000060: 00 00 06 48 bd a5 10 00 08 00 00 02 00 00 00 00  ...H............
kernel: [   73.702975] 00000070: 00 00 00 00 00 00 00 00 0c 0c 0b 01 0d 00 00 19  ................
kernel: [   73.703373] XFS (sdl1): SB validate failed with error -117.

The same issue was present for the other failing OSDs. We hoped, that the data itself was still there, and only the mounting of the XFS partitions failed. The Ceph cluster was initially installed in 2017 with Ceph jewel/10.2 with the OSDs on filestore (nowadays being a legacy approach to storing objects in Ceph). However, we migrated the disks to bluestore since then (with ceph-disk and not yet via ceph-volume what s being used nowadays). Using ceph-disk introduces these 100MB XFS partitions containing basic metadata for the OSD. Given that we had three working OSDs left, we decided to investigate how to rebuild the failing ones. Some folks on #ceph (thanks T1, ormandj + peetaur!) were kind enough to share how working XFS partitions looked like for them. After creating a backup (via dd), we tried to re-create such an XFS partition on server1. We noticed that even mounting a freshly created XFS partition failed:

synpromika@server1 ~ % sudo mkfs.xfs -f -i size=2048 -m uuid="4568c300-ad83-4288-963e-badcd99bf54f" /dev/sdc1
meta-data=/dev/sdc1              isize=2048   agcount=4, agsize=6272 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=0
data     =                       bsize=4096   blocks=25088, imaxpct=25
         =                       sunit=128    swidth=64 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=1608, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
synpromika@server1 ~ % sudo mount /dev/sdc1 /mnt/ceph-recovery
SB stripe unit sanity check failed
Metadata corruption detected at 0x433840, xfs_sb block 0x0/0x1000
libxfs_writebufr: write verifer failed on xfs_sb bno 0x0/0x1000
cache_node_purge: refcount was 1, not zero (node=0x1d3c400)
SB stripe unit sanity check failed
Metadata corruption detected at 0x433840, xfs_sb block 0x18800/0x1000
libxfs_writebufr: write verifer failed on xfs_sb bno 0x18800/0x1000
SB stripe unit sanity check failed
Metadata corruption detected at 0x433840, xfs_sb block 0x0/0x1000
libxfs_writebufr: write verifer failed on xfs_sb bno 0x0/0x1000
SB stripe unit sanity check failed
Metadata corruption detected at 0x433840, xfs_sb block 0x24c00/0x1000
libxfs_writebufr: write verifer failed on xfs_sb bno 0x24c00/0x1000
SB stripe unit sanity check failed
Metadata corruption detected at 0x433840, xfs_sb block 0xc400/0x1000
libxfs_writebufr: write verifer failed on xfs_sb bno 0xc400/0x1000
releasing dirty buffer (bulk) to free list!releasing dirty buffer (bulk) to free list!releasing dirty buffer (bulk) to free list!releasing dirty buffer (bulk) to free list!found dirty buffer (bulk) on free list!bad magic number
bad magic number
Metadata corruption detected at 0x433840, xfs_sb block 0x0/0x1000
libxfs_writebufr: write verifer failed on xfs_sb bno 0x0/0x1000
releasing dirty buffer (bulk) to free list!mount: /mnt/ceph-recovery: wrong fs type, bad option, bad superblock on /dev/sdc1, missing codepage or helper program, or other error.

Ouch. This very much looked related to the actual issue we re seeing. So we tried to execute mkfs.xfs with a bunch of different sunit/swidth settings. Using -d sunit=512 -d swidth=512 at least worked then, so we decided to force its usage in the creation of our OSD XFS partition. This brought us a working XFS partition. Please note, sunit must not be larger than swidth (more on that later!). Then we reconstructed how to restore all the metadata for the OSD (

activate.monmap, active, block_uuid, bluefs, ceph_fsid, fsid, keyring, kv_backend, magic, mkfs_done, ready, require_osd_release, systemd, type, whoami

). To identify the UUID, we can read the data from ceph --format json osd dump , like this for all our OSDs (Zsh syntax ftw!):

synpromika@server1 ~ % for f in  0..37  ; printf "osd-$f: %s\n" "$(sudo ceph --format json osd dump   jq -r ".osds[]   select(.osd==$f)   .uuid")"
osd-0: 4568c300-ad83-4288-963e-badcd99bf54f
osd-1: e573a17a-ccde-4719-bdf8-eef66903ca4f
osd-2: 0e1b2626-f248-4e7d-9950-f1a46644754e
osd-3: 1ac6a0a2-20ee-4ed8-9f76-d24e900c800c
[...]

Identifying the corresponding raw device for each OSD UUID is possible via:

synpromika@server1 ~ % UUID="4568c300-ad83-4288-963e-badcd99bf54f"
synpromika@server1 ~ % readlink -f /dev/disk/by-partuuid/"$ UUID "
/dev/sdc1

The OSD s key ID can be retrieved via:

synpromika@server1 ~ % OSD_ID=0
synpromika@server1 ~ % sudo ceph auth get osd."$ OSD_ID " -f json 2>/dev/null   jq -r '.[]   .key'
AQCKFpZdm0We[...]

Now we also need to identify the underlying block device:

synpromika@server1 ~ % OSD_ID=0
synpromika@server1 ~ % sudo ceph osd metadata osd."$ OSD_ID " -f json   jq -r '.bluestore_bdev_partition_path'    
/dev/sdc2

With all of this, we reconstructed the keyring, fsid, whoami, block + block_uuid files. All the other files inside the XFS metadata partition are identical on each OSD. So after placing and adjusting the corresponding metadata on the XFS partition for Ceph usage, we got a working OSD hurray! Since we had to fix yet another 32 OSDs, we decided to automate this XFS partitioning and metadata recovery procedure. We had a network share available on /srv/backup for storing backups of existing partition data. On each server, we tested the procedure with one single OSD before iterating over the list of remaining failing OSDs. We started with a shell script on server1, then adjusted the script for server2 and server3. This is the script, as we executed it on the 3rd server. Thanks to this, we managed to get the Ceph cluster up and running again. We didn t want to continue with the Ceph upgrade itself during the night though, as we wanted to know exactly what was going on and why the system behaved like that. Time for RCA! Root Cause Analysis So all but three OSDs on server2 failed, and the problem seems to be related to XFS. Therefore, our starting point for the RCA was, to identify what was different on server2, as compared to server1 + server3. My initial assumption was that this was related to some firmware issues with the involved controller (and as it turned out later, I was right!). The disks were attached as JBOD devices to a ServeRAID M5210 controller (with a stripe size of 512). Firmware state:

synpromika@server1 ~ % sudo storcli64 /c0 show all   grep '^Firmware'
Firmware Package Build = 24.16.0-0092
Firmware Version = 4.660.00-8156
synpromika@server2 ~ % sudo storcli64 /c0 show all   grep '^Firmware'
Firmware Package Build = 24.21.0-0112
Firmware Version = 4.680.00-8489
synpromika@server3 ~ % sudo storcli64 /c0 show all   grep '^Firmware'
Firmware Package Build = 24.16.0-0092
Firmware Version = 4.660.00-8156

This looked very promising, as server2 indeed runs with a different firmware version on the controller. But how so? Well, the motherboard of server2 got replaced by a Lenovo/IBM technician in January 2020, as we had a failing memory slot during a memory upgrade. As part of this procedure, the Lenovo/IBM technician installed the latest firmware versions. According to our documentation, some OSDs were rebuilt (due to the filestore->bluestore migration) in March and April 2020. It turned out that precisely those OSDs were the ones that survived the upgrade. So the surviving drives were created with a different firmware version running on the involved controller. All the other OSDs were created with an older controller firmware. But what difference does this make? Now let s check firmware changelogs. For the 24.21.0-0097 release we found this:

- Cannot create or mount xfs filesystem using xfsprogs 4.19.x kernel 4.20(SCGCQ02027889)
- xfs_info command run on an XFS file system created on a VD of strip size 1M shows sunit and swidth as 0(SCGCQ02056038)

Our XFS problem certainly was related to the controller s firmware. We also recalled that our monitoring system reported different sunit settings for the OSDs that were rebuilt in March and April. For example, OSD 21 was recreated and got different sunit settings:

WARN  server2.example.org  Mount options of /var/lib/ceph/osd/ceph-21      WARN - Missing: sunit=1024, Exceeding: sunit=512

We compared the new OSD 21 with an existing one (OSD 25 on server3):

synpromika@server2 ~ % systemctl show var-lib-ceph-osd-ceph\\x2d21.mount   grep sunit
Options=rw,noatime,attr2,inode64,sunit=512,swidth=512,noquota
synpromika@server3 ~ % systemctl show var-lib-ceph-osd-ceph\\x2d25.mount   grep sunit
Options=rw,noatime,attr2,inode64,sunit=1024,swidth=512,noquota

Thanks to our documentation, we could compare execution logs of their creation:

% diff -u ceph-disk-osd-25.log ceph-disk-osd-21.log
-synpromika@server2 ~ % sudo ceph-disk -v prepare --bluestore /dev/sdj --osd-id 25
+synpromika@server3 ~ % sudo ceph-disk -v prepare --bluestore /dev/sdi --osd-id 21
[...]
-command_check_call: Running command: /sbin/mkfs -t xfs -f -i size=2048 -- /dev/sdj1
-meta-data=/dev/sdj1              isize=2048   agcount=4, agsize=6272 blks
[...]
+command_check_call: Running command: /sbin/mkfs -t xfs -f -i size=2048 -- /dev/sdi1
+meta-data=/dev/sdi1              isize=2048   agcount=4, agsize=6336 blks
          =                       sectsz=4096  attr=2, projid32bit=1
          =                       crc=1        finobt=1, sparse=0, rmapbt=0, reflink=0
-data     =                       bsize=4096   blocks=25088, imaxpct=25
-         =                       sunit=128    swidth=64 blks
+data     =                       bsize=4096   blocks=25344, imaxpct=25
+         =                       sunit=64     swidth=64 blks
 naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
 log      =internal log           bsize=4096   blocks=1608, version=2
          =                       sectsz=4096  sunit=1 blks, lazy-count=1
 realtime =none                   extsz=4096   blocks=0, rtextents=0
[...]

So back then, we even tried to track this down but couldn t make sense of it yet. But now this sounds very much like it is related to the problem we saw with this Ceph/XFS failure. We follow Occam s razor, assuming the simplest explanation is usually the right one, so let s check the disk properties and see what differs:

synpromika@server1 ~ % sudo blockdev --getsz --getsize64 --getss --getpbsz --getiomin --getioopt /dev/sdk
4685545472
2398999281664
512
4096
524288
262144
synpromika@server2 ~ % sudo blockdev --getsz --getsize64 --getss --getpbsz --getiomin --getioopt /dev/sdk
4685545472
2398999281664
512
4096
262144
262144

See the difference between server1 and server2 for identical disks? The getiomin option now reports something different for them:

synpromika@server1 ~ % sudo blockdev --getiomin /dev/sdk            
524288
synpromika@server1 ~ % cat /sys/block/sdk/queue/minimum_io_size
524288
synpromika@server2 ~ % sudo blockdev --getiomin /dev/sdk 
262144
synpromika@server2 ~ % cat /sys/block/sdk/queue/minimum_io_size
262144

It doesn t make sense that the minimum I/O size (iomin, AKA BLKIOMIN) is bigger than the optimal I/O size (ioopt, AKA BLKIOOPT). This leads us to Bug 202127 cannot mount or create xfs on a 597T device, which matches our findings here. But why did this XFS partition work in the past and fails now with the newer kernel version? The XFS behaviour change Now given that we have backups of all the XFS partition, we wanted to track down, a) when this XFS behaviour was introduced, and b) whether, and if so how it would be possible to reuse the XFS partition without having to rebuild it from scratch (e.g. if you would have no working Ceph OSD or backups left). Let s look at such a failing XFS partition with the Grml live system:

root@grml ~ # grml-version
grml64-full 2020.06 Release Codename Ausgehfuahangl [2020-06-24]
root@grml ~ # uname -a
Linux grml 5.6.0-2-amd64 #1 SMP Debian 5.6.14-2 (2020-06-09) x86_64 GNU/Linux
root@grml ~ # grml-hostname grml-2020-06
Setting hostname to grml-2020-06: done
root@grml ~ # exec zsh
root@grml-2020-06 ~ # dpkg -l xfsprogs util-linux
Desired=Unknown/Install/Remove/Purge/Hold
  Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
 / Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
 / Name           Version      Architecture Description
+++-==============-============-============-=========================================
ii  util-linux     2.35.2-4     amd64        miscellaneous system utilities
ii  xfsprogs       5.6.0-1+b2   amd64        Utilities for managing the XFS filesystem

There it s failing, no matter which mount option we try:

root@grml-2020-06 ~ # mount ./sdd1.dd /mnt
mount: /mnt: mount(2) system call failed: Structure needs cleaning.
root@grml-2020-06 ~ # dmesg   tail -30
[...]
[   64.788640] XFS (loop1): SB stripe unit sanity check failed
[   64.788671] XFS (loop1): Metadata corruption detected at xfs_sb_read_verify+0x102/0x170 [xfs], xfs_sb block 0xffffffffffffffff
[   64.788671] XFS (loop1): Unmount and run xfs_repair
[   64.788672] XFS (loop1): First 128 bytes of corrupted metadata buffer:
[   64.788673] 00000000: 58 46 53 42 00 00 10 00 00 00 00 00 00 00 62 00  XFSB..........b.
[   64.788674] 00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[   64.788675] 00000020: 32 b6 dc 35 53 b7 44 96 9d 63 30 ab b3 2b 68 36  2..5S.D..c0..+h6
[   64.788675] 00000030: 00 00 00 00 00 00 40 08 00 00 00 00 00 00 01 00  ......@.........
[   64.788675] 00000040: 00 00 00 00 00 00 01 01 00 00 00 00 00 00 01 02  ................
[   64.788676] 00000050: 00 00 00 01 00 00 18 80 00 00 00 04 00 00 00 00  ................
[   64.788677] 00000060: 00 00 06 48 bd a5 10 00 08 00 00 02 00 00 00 00  ...H............
[   64.788677] 00000070: 00 00 00 00 00 00 00 00 0c 0c 0b 01 0d 00 00 19  ................
[   64.788679] XFS (loop1): SB validate failed with error -117.
root@grml-2020-06 ~ # mount -t xfs -o rw,relatime,attr2,inode64,sunit=1024,swidth=512,noquota ./sdd1.dd /mnt/
mount: /mnt: wrong fs type, bad option, bad superblock on /dev/loop1, missing codepage or helper program, or other error.
32 root@grml-2020-06 ~ # dmesg   tail -1
[   66.342976] XFS (loop1): stripe width (512) must be a multiple of the stripe unit (1024)
root@grml-2020-06 ~ # mount -t xfs -o rw,relatime,attr2,inode64,sunit=512,swidth=512,noquota ./sdd1.dd /mnt/
mount: /mnt: mount(2) system call failed: Structure needs cleaning.
32 root@grml-2020-06 ~ # dmesg   tail -14
[   66.342976] XFS (loop1): stripe width (512) must be a multiple of the stripe unit (1024)
[   80.751277] XFS (loop1): SB stripe unit sanity check failed
[   80.751323] XFS (loop1): Metadata corruption detected at xfs_sb_read_verify+0x102/0x170 [xfs], xfs_sb block 0xffffffffffffffff 
[   80.751324] XFS (loop1): Unmount and run xfs_repair
[   80.751325] XFS (loop1): First 128 bytes of corrupted metadata buffer:
[   80.751327] 00000000: 58 46 53 42 00 00 10 00 00 00 00 00 00 00 62 00  XFSB..........b.
[   80.751328] 00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[   80.751330] 00000020: 32 b6 dc 35 53 b7 44 96 9d 63 30 ab b3 2b 68 36  2..5S.D..c0..+h6
[   80.751331] 00000030: 00 00 00 00 00 00 40 08 00 00 00 00 00 00 01 00  ......@.........
[   80.751331] 00000040: 00 00 00 00 00 00 01 01 00 00 00 00 00 00 01 02  ................
[   80.751332] 00000050: 00 00 00 01 00 00 18 80 00 00 00 04 00 00 00 00  ................
[   80.751333] 00000060: 00 00 06 48 bd a5 10 00 08 00 00 02 00 00 00 00  ...H............
[   80.751334] 00000070: 00 00 00 00 00 00 00 00 0c 0c 0b 01 0d 00 00 19  ................
[   80.751338] XFS (loop1): SB validate failed with error -117.

Also xfs_repair doesn t help either:

root@grml-2020-06 ~ # xfs_info ./sdd1.dd
meta-data=./sdd1.dd              isize=2048   agcount=4, agsize=6272 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=0, rmapbt=0
         =                       reflink=0
data     =                       bsize=4096   blocks=25088, imaxpct=25
         =                       sunit=128    swidth=64 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=1608, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
root@grml-2020-06 ~ # xfs_repair ./sdd1.dd
Phase 1 - find and verify superblock...
bad primary superblock - bad stripe width in superblock !!!
attempting to find secondary superblock...
..............................................................................................Sorry, could not find valid secondary superblock
Exiting now.

With the SB stripe unit sanity check failed message, we could easily track this down to the following commit fa4ca9c:

% git show fa4ca9c5574605d1e48b7e617705230a0640b6da   cat
commit fa4ca9c5574605d1e48b7e617705230a0640b6da
Author: Dave Chinner <dchinner@redhat.com>
Date:   Tue Jun 5 10:06:16 2018 -0700
    
    xfs: catch bad stripe alignment configurations
    
    When stripe alignments are invalid, data alignment algorithms in the
    allocator may not work correctly. Ensure we catch superblocks with
    invalid stripe alignment setups at mount time. These data alignment
    mismatches are now detected at mount time like this:
    
    XFS (loop0): SB stripe unit sanity check failed
    XFS (loop0): Metadata corruption detected at xfs_sb_read_verify+0xab/0x110, xfs_sb block 0xffffffffffffffff
    XFS (loop0): Unmount and run xfs_repair
    XFS (loop0): First 128 bytes of corrupted metadata buffer:
    0000000091c2de02: 58 46 53 42 00 00 10 00 00 00 00 00 00 00 10 00  XFSB............
    0000000023bff869: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
    00000000cdd8c893: 17 32 37 15 ff ca 46 3d 9a 17 d3 33 04 b5 f1 a2  .27...F=...3....
    000000009fd2844f: 00 00 00 00 00 00 00 04 00 00 00 00 00 00 06 d0  ................
    0000000088e9b0bb: 00 00 00 00 00 00 06 d1 00 00 00 00 00 00 06 d2  ................
    00000000ff233a20: 00 00 00 01 00 00 10 00 00 00 00 01 00 00 00 00  ................
    000000009db0ac8b: 00 00 03 60 e1 34 02 00 08 00 00 02 00 00 00 00  ... .4..........
    00000000f7022460: 00 00 00 00 00 00 00 00 0c 09 0b 01 0c 00 00 19  ................
    XFS (loop0): SB validate failed with error -117.
    
    And the mount fails.
    
    Signed-off-by: Dave Chinner <dchinner@redhat.com>
    Reviewed-by: Carlos Maiolino <cmaiolino@redhat.com>
    Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com>
    Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
diff --git fs/xfs/libxfs/xfs_sb.c fs/xfs/libxfs/xfs_sb.c
index b5dca3c8c84d..c06b6fc92966 100644
--- fs/xfs/libxfs/xfs_sb.c
+++ fs/xfs/libxfs/xfs_sb.c
@@ -278,6 +278,22 @@ xfs_mount_validate_sb(
                return -EFSCORRUPTED;
         
        
+       if (sbp->sb_unit)  
+               if (!xfs_sb_version_hasdalign(sbp)  
+                   sbp->sb_unit > sbp->sb_width  
+                   (sbp->sb_width % sbp->sb_unit) != 0)  
+                       xfs_notice(mp, "SB stripe unit sanity check failed");
+                       return -EFSCORRUPTED;
+                 
+         else if (xfs_sb_version_hasdalign(sbp))   
+               xfs_notice(mp, "SB stripe alignment sanity check failed");
+               return -EFSCORRUPTED;
+         else if (sbp->sb_width)  
+               xfs_notice(mp, "SB stripe width sanity check failed");
+               return -EFSCORRUPTED;
+        
+
+       
        if (xfs_sb_version_hascrc(&mp->m_sb) &&
            sbp->sb_blocksize < XFS_MIN_CRC_BLOCKSIZE)  
                xfs_notice(mp, "v5 SB sanity check failed");

This change is included in kernel versions 4.18-rc1 and newer:

% git describe --contains fa4ca9c5574605d1e48
v4.18-rc1~37^2~14

Now let s try with an older kernel version (4.9.0), using old Grml 2017.05 release:

root@grml ~ # grml-version
grml64-small 2017.05 Release Codename Freedatensuppe [2017-05-31]
root@grml ~ # uname -a
Linux grml 4.9.0-1-grml-amd64 #1 SMP Debian 4.9.29-1+grml.1 (2017-05-24) x86_64 GNU/Linux
root@grml ~ # lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 9.0 (stretch)
Release:        9.0
Codename:       stretch
root@grml ~ # grml-hostname grml-2017-05
Setting hostname to grml-2017-05: done
root@grml ~ # exec zsh
root@grml-2017-05 ~ #
root@grml-2017-05 ~ # xfs_info ./sdd1.dd
xfs_info: ./sdd1.dd is not a mounted XFS filesystem
1 root@grml-2017-05 ~ # xfs_repair ./sdd1.dd
Phase 1 - find and verify superblock...
bad primary superblock - bad stripe width in superblock !!!
attempting to find secondary superblock...
..............................................................................................Sorry, could not find valid secondary superblock
Exiting now.
1 root@grml-2017-05 ~ # mount ./sdd1.dd /mnt
root@grml-2017-05 ~ # mount -t xfs
/root/sdd1.dd on /mnt type xfs (rw,relatime,attr2,inode64,sunit=1024,swidth=512,noquota)
root@grml-2017-05 ~ # ls /mnt
activate.monmap  active  block  block_uuid  bluefs  ceph_fsid  fsid  keyring  kv_backend  magic  mkfs_done  ready  require_osd_release  systemd  type  whoami
root@grml-2017-05 ~ # xfs_info /mnt
meta-data=/dev/loop1             isize=2048   agcount=4, agsize=6272 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=1 spinodes=0 rmapbt=0
         =                       reflink=0
data     =                       bsize=4096   blocks=25088, imaxpct=25
         =                       sunit=128    swidth=64 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal               bsize=4096   blocks=1608, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

Mounting there indeed works! Now, if we mount the filesystem with new and proper sunit/swidth settings using the older kernel, it should rewrite them on disk:

root@grml-2017-05 ~ # mount -t xfs -o sunit=512,swidth=512 ./sdd1.dd /mnt/
root@grml-2017-05 ~ # umount /mnt/

And indeed, mounting this rewritten filesystem then also works with newer kernels:

root@grml-2020-06 ~ # mount ./sdd1.rewritten /mnt/
root@grml-2020-06 ~ # xfs_info /root/sdd1.rewritten
meta-data=/dev/loop1             isize=2048   agcount=4, agsize=6272 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=0, rmapbt=0
         =                       reflink=0
data     =                       bsize=4096   blocks=25088, imaxpct=25
         =                       sunit=64    swidth=64 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=1608, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
root@grml-2020-06 ~ # mount -t xfs                
/root/sdd1.rewritten on /mnt type xfs (rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,sunit=512,swidth=512,noquota)

FTR: The sunit=512,swidth=512 from the xfs mount option is identical to xfs_info s output sunit=64,swidth=64 (because mount.xfs s sunit value is given in 512-byte block units, see man 5 xfs, and the xfs_info output reported here is in blocks with a block size (bsize) of 4096, so sunit = 512*512 := 64*4096 ). mkfs uses minimum and optimal sizes for stripe unit and stripe width; you can check this e.g. via (note that server2 with fixed firmware version reports proper values, whereas server3 with broken controller firmware reports non-sense):

synpromika@server2 ~ % for i in /sys/block/sd*/queue/ ; do printf "%s: %s %s\n" "$i" "$(cat "$i"/minimum_io_size)" "$(cat "$i"/optimal_io_size)" ; done
[...]
/sys/block/sdc/queue/: 262144 262144
/sys/block/sdd/queue/: 262144 262144
/sys/block/sde/queue/: 262144 262144
/sys/block/sdf/queue/: 262144 262144
/sys/block/sdg/queue/: 262144 262144
/sys/block/sdh/queue/: 262144 262144
/sys/block/sdi/queue/: 262144 262144
/sys/block/sdj/queue/: 262144 262144
/sys/block/sdk/queue/: 262144 262144
/sys/block/sdl/queue/: 262144 262144
/sys/block/sdm/queue/: 262144 262144
/sys/block/sdn/queue/: 262144 262144
[...]
synpromika@server3 ~ % for i in /sys/block/sd*/queue/ ; do printf "%s: %s %s\n" "$i" "$(cat "$i"/minimum_io_size)" "$(cat "$i"/optimal_io_size)" ; done
[...]
/sys/block/sdc/queue/: 524288 262144
/sys/block/sdd/queue/: 524288 262144
/sys/block/sde/queue/: 524288 262144
/sys/block/sdf/queue/: 524288 262144
/sys/block/sdg/queue/: 524288 262144
/sys/block/sdh/queue/: 524288 262144
/sys/block/sdi/queue/: 524288 262144
/sys/block/sdj/queue/: 524288 262144
/sys/block/sdk/queue/: 524288 262144
/sys/block/sdl/queue/: 524288 262144
/sys/block/sdm/queue/: 524288 262144
/sys/block/sdn/queue/: 524288 262144
[...]

This is the underlying reason why the initially created XFS partitions were created with incorrect sunit/swidth settings. The broken firmware of server1 and server3 was the cause of the incorrect settings they were ignored by old(er) xfs/kernel versions, but treated as an error by new ones. Make sure to also read the XFS FAQ regarding How to calculate the correct sunit,swidth values for optimal performance . We also stumbled upon two interesting reads in RedHat s knowledge base: 5075561 + 2150101 (requires an active subscription, though) and #1835947. Am I affected? How to work around it? To check whether your XFS mount points are affected by this issue, the following command line should be useful:

awk '$3 == "xfs" print $2 ' /proc/self/mounts   while read mount ; do echo -n "$mount " ; xfs_info $mount   awk '$0 ~ "swidth" gsub(/.*=/,"",$2); gsub(/.*=/,"",$3); print $2,$3 '   awk '  if ($1 > $2) print "impacted"; else print "OK" ' ; done

If you run into the above situation, the only known solution to get your original XFS partition working again, is to boot into an older kernel version again (4.17 or older), mount the XFS partition with correct sunit/swidth settings and then boot back into your new system (kernel version wise). Lessons learned

document everything and ensure to have all relevant information available (including actual times of changes, used kernel/package/firmware/ versions. The thorough documentation was our most significant asset in this case, because we had all the data and information we needed during the emergency handling as well as for the post mortem/RCA)
if something changes unexpectedly, dig deeper
know who to ask, a network of experts pays off
including timestamps in your shell makes reconstruction easier (the more people and documentation involved, the harder it gets to wade through it)
keep an eye on changelogs/release notes
apply regular updates and don t forget invisible layers (e.g. BIOS, controller/disk firmware, IPMI/OOB (ILO/RAC/IMM/ ) firmware)
apply regular reboots, to avoid a possible delta becoming bigger (which makes debugging harder)

Thanks: Darshaka Pathirana, Chris Hofstaedtler and Michael Hanscho. Looking for help with your IT infrastructure? Let us know!

The Debian stretch release is going to happen soon (on 2017-06-17) and since our latest Grml release is based on a very recent version of Debian stretch I m taking this as opportunity to announce it also here. So by the end of May we released a new stable release of Grml (the Debian based live system focusing on system administrator s needs), known as version 2017.05 with codename Freedatensuppe. Details about the changes of the new release are available in the official release notes and as usual the ISOs are available via grml.org/download. With this new Grml release we finally made the switch from file-rc to systemd. From a user s point of view this doesn t change that much, though to prevent having to answer even more mails regarding the switch I wrote down some thoughts in Grml s FAQ. There are some things that we still need to improve and sort out, but overall the switch to systemd so far went better than anticipated (thanks a lot to the pkg-systemd folks, especially Felipe Sateler and Michael Biebl!). And last but not least, Darshaka Pathirana helped me a lot with the systemd integration and polishing the release, many thanks! Happy Grml-ing!

Review: Small Gods, by Terry Pratchett

Series:	Discworld #13
Publisher:	Harper
Copyright:	February 1994
Printing:	March 2008
ISBN:	0-06-109217-7
Format:	Mass market
Pages:	357

Small Gods is the thirteenth Discworld novel, but it features new characters and is unrelated to any of the previous books. Some reading order guides show it as following Pyramids in an "ancient civilizations" track, but its only relationship with that book is some minor thematic similarities. You could start here with Discworld if you wanted to. Brutha is a novice in the hierarchy of the church of the Great God Om, and his elders are convinced he'll probably die a novice. He's just not particularly bright, you see. But he is very obedient, and he doesn't mind doing hard work, and there's nothing exactly wrong with him, except that he looks at people with startling intensity when they're talking to him. Almost as if he's listening. All that seems about to change, however, when the Great God Om himself approaches Brutha and starts talking to him. Not that Brutha is at all convinced at first that this is happening, particularly given that Om appears in the form of a small, battered, one-eyed tortoise who was dropped into the church garden by an eagle attempting to break his shell. Small Gods is, as you might have guessed, a parody of religion, at least large, organized religion with fixed hierarchies, organizations called the Quisition that like to torture people, and terrifyingly devout deacons who are certain of themselves in ways that no human ever should be. It's also an interesting bit of Discworld metaphysics: gods gain power from worship (a very old idea in fantasy), and when they don't get enough worship, they end up much diminished and even adrift in the desert. Or trapped in the form of a small tortoise. One might wonder how Om ended up in his present condition given the vast and extremely authoritarian church devoted to his worship, but that's the heart of Pratchett's rather pointed parody: large religious organizations end up being about themselves, rather than about the god they supposedly worship, to such an extent that they don't provide any worship at all. Brutha is not thinking of things like this. Once he's finally convinced that Om is who he claims to be, he provides worship and belief of a very practical but wholehearted and unshakable sort, just as he does everything else in life. That makes him the eighth prophet of Om as prophecy foretold, but it's far from clear how that will be of any practical use. Or how Om will come back into power. And meanwhile, Brutha has come to the attention of Vorbis, the head of the Exquisitors, who does not know about the tortoise (and wouldn't believe if he did), but who has a use for Brutha's other talent: his eidetic memory. In typical Pratchett fashion, the story expands to include a variety of other memorable characters from the neighboring city of Ephebe, a country full of gods and philosophers. Vorbis's aims here are unclear at the start of the book, but Vorbis being who he is, they can't be good. Brutha is drawn along in his wake. Meanwhile, Om is constantly watching for an opportunity to regain his lost power and worshipful following, and also to avoid being eaten. Despite the humorous components, Small Gods is rather serious about religion and about its villain. It's also a touch repetitive; Om's lack of power and constant fretting about it, Brutha's earnest but naive loyalty, and Vorbis's malevolent determination are repeatedly stressed and get a little old. Some bits in Ephebe are quite fun, but the action is a bit disjointed, partly because the protagonist is rarely the motive force in the plot. There are also some extended scenes of trudging through the desert that I thought dragged a bit. But Pratchett hits some powerful notes in his critique of religion, and there are a few bits with Death at the end of the book that I thought were among the better pieces of Discworld philosophy. And when Brutha gets a chance to use his one talent of memory, I greatly enjoyed the resulting scenes. He hits just the right combination of modesty, capability, and earnestness. I know a lot of Pratchett readers really like Small Gods. I'm not one of those; I thought it was about average for the Discworld series (at least among the books I've read so far). But average for Discworld is still pretty good, and its new setting makes it a plausible place to start (or to take a break from the other Discworld plot threads). Followed, in publication order, by Lords and Ladies. I don't believe it has a direct plot sequel. Rating: 7 out of 10

Review: A Man Called Ove, by Fredrik Backman

Translator:	Henning Koch
Publisher:	Washington Square
Copyright:	2012, 2014
Printing:	May 2015
ISBN:	1-4767-3802-5
Format:	Trade paperback
Pages:	337

Ove is 59 years old and drives a Saab. He's grumpy, often taciturn, very particular in how things should be done, and extremely judgmental about how other people do them. He is the sort of person who regularly rants about how poorly things are done these days and how much better they used to be. Ove does not like change. A Man Called Ove opens with him terrorizing the employees of an Apple Store, trying to buy a computer, but this incident is actually foreshadowing, and will only make sense at the very end of the book. The story actually begins in the second chapter, with Ove making his morning rounds of the neighborhood in which he lives, discovering an out-of-place bicycle and a mangy cat, and then starting to put a hook in his ceiling. But just as he's getting started, he's interrupted by new neighbors, who are incapable of backing up a trailer properly without scraping it against his house. Not that motor vehicles are allowed in the area anyway. That inauspicious beginning changes Ove's life, mostly through the sheer persistence of other people's disasters. It's not obvious at first that it will, and at the start A Man Called Ove could be a funny collection of stories about a curmudgeon. But as Backman shows more of Ove's life and tells more of his background and situation, it becomes something so much more, something satisfying and heart-breaking and deeply human. I've been struggling to review this book because I find it hard to capture what makes it so wonderful. Making that even harder, several key plot elements are introduced gradually in the story in ways that add a lot to the rhythm of the plot, and I don't want to spoil them. I think the closest I can get in a spoiler-free review is that A Man Called Ove is about empathy. It's about human connection, even when people seem unlikable, unreachable, or angrily off-putting. And it's a book about seeing the best inside other people, and about finding ways to be persistently oneself while still changing enough to find new connections, and about recognizing those moments when someone is showing you their best without getting caught up on the surface presentation. The man Ove is the center of this story, the subject of tight third person perspective for nearly all of the book. He's 59 when the story opens, but by the end of the book, mostly through flashback chapters, the reader knows his childhood and early adulthood and much of the story of his marriage. At first, he seems to be an obnoxious, surly, angry curmudgeon, the sort of old man who yells at clouds. But the joy of this book is how the reader's perception changes, how one gains sympathy, and then respect, first for Ove's unshakable inner sense of morality that he got from his father and then for his rule-based approach to how the world should work. One never entirely agrees with him, but Backman demystifies and explains Ove's thought process and ties it into a different generation and a different way of interacting with work (although Ove was still uncommon even in his youth, just not as unique). As I write this review, news and opinion in the United States are very focused on the plight of the white working class and how that does or does not explain recent election results. Backman is Swedish (I read this book in translation), so it's not coming from US culture and the cultural fault lines are not quite the same. But I think this book says something deeply valuable and fascinating about the working-class culture of Ove's youth, something that's much less about specific politics and much more about how it feels to make things with one's hands, to build or rebuild one's own house, or to work for thirty years at the same job and not be interested in a promotion to management. Backman does a truly spectacular job conveying the sense of angry frustration at the changes in work and life, the difficulty communicating one's internal feelings meaningfully, and the quiet joy of finding those places in life where one can do things properly. Ove is, of course, not the only character in this book, and every character here is a delight in their own idiosyncratic ways. The main story arc involves the various people in his neighborhood, particularly his new next-door neighbors: a pregnant Iranian woman and her very laid-back husband who is incapable of doing things around the house but keeps trying. I think Parvaneh, the woman, is my favorite character of the story except for Ove, and it's fitting that she's the first to work out the broad outlines of what's happening in Ove's life and the tricky path to effectively helping him. In a way, she's Ove's opposite: fiery, mercurial, talkative, and meddling. But she sees things in Ove that no one else seems to notice. (And the scene between her and Ove when she's learning to drive is a thing of beauty.) This is a book that could have been extremely sad, and yet isn't. It's a book about somber, depressing topics that somehow manages to be delightfully funny. And it's about a curmudgeon who persistently fails to have any sort of stereotyped heart of gold, but is nonetheless one of the most satisfying, fascinating, ethical, and good-willed characters I've ever read about. It manages to treat a collection of very different characters with individualized deep empathy and appreciation, while never pushing them all into the same mold. And the ending is wonderful. I rarely read slice of life stories, but this one is worth making an exception for. It's one of the best books I've ever read. Highly, highly recommended. Rating: 10 out of 10

Review: A Man Called Ove, by Fredrik Backman

Translator:	Henning Koch
Publisher:	Washington Square
Copyright:	2012, 2014
Printing:	May 2015
ISBN:	1-4767-3802-5
Format:	Trade paperback
Pages:	337

As I previously blogged about, I wrote an Annual Review of Anthropology on digital media last year. About a month ago, I found out that anyone can download it thanks to a link provided by the ARA, which we are allowed to put on one institutional web page. So go here (and go to the citation for the link) for those who are interested in a way too short review of some of the ethnographic literature on digital media. Writing the piece left me many psychological wounds and scars, one of which had to do with the fact that I probably overlooked some folks. I have been left out of review type essays and honestly, it sucks. I tried to be as comprehensive as possible: I chose not to massively whittle down the scope (which was an option) and was able to smuggle in more citations than originally allowed and yet I still cut out 200 citations. But in the end I overlooked some folks as I found out about them too late. If I could go back in time, this is who I would include (well there are others but I have chosen these for now). So Shaka McGlotten: not only does he have a cool name, he studies some cool stuff like DIY online porn, race, and zombies. He has published a bunch of articles and a book is forthcoming. Check out his work here. I missed this book Online a Lot of the Time: Ritual, Fetish, and Sign by Ken Hills which looks quite pertinent and a great read. Jonathan Marshall is an anthropologist in Sydney who has been working on digital stuff for a long while now and recently published Living Cybermind, which covers in detail modes of interacting and communicating in a detailed examination of a mailing list by the name of Cybermind. Although this book is not out yet, it will be soon and looks fascinating: Digital Jesus (great cover). From what I understand, and one of the reasons I want to dive into the book , is because it is so longitudinal, at least when measured in Internet years. Rob Howard has studied Christians online since the days of Usenet to the present and thus this ethnography promises to have some real meat to it. I am eagerly anticipating the forthcoming book by Beth Coleman . I had a chance to read a chapter recently and it looks fantastic. While in some respects grounded in virtual worlds, it is far more expansive than that topic, addressing a range of issues from desire, experience, emotion and race. Can t wait. I cite four dissertations in the ARA mostly because I read them, thought they were great, and material based on the dissertation is en route to being published in some form in the next future. The one dissertation I wish I had read is Jenny Cool s thesis on cyberorganic.

I started to write this review last night, and went looking for Wil Wheaton s blog, where many of the stories came from, so I can link to it from my review. It was getting late, I was tired, and so I was a bit disoriented for a few seconds when I saw my own words flash up on the screen. At the time, his most recent story had excerpted my review of paper books. Wow, I thought. This never happens when I m about to review Dickens. And actually, it s never happened before, ever. I ll admit to owning a big grin when I saw that one of my favorite authors liked one of my blog posts. And Wil Wheaton is one of my favorite authors for sure. I enjoy reading others too, of course, but Wil s writing is something I can really identify with like no other. My parents were never in a London debtor s prison like Dickens were; I was never a promising medical student like A. C. Doyle. But I was, and am, a geek, and Wil Wheaton captures that more perfectly than anyone. After I read Just a Geek a few years ago, I gave it to my wife to read, claiming it would help her understand me better. I think it did. In The Happiest Days of Our Lives, Wil recounts memories of his childhood, and of more recent days. He talks of flashbacks to his elementary school days, when he and his classmates tried to have the coolest Star Wars action figures (for me: calculator watches). Or how his aunt introduced him to D&D, which reminded me of how my uncle got me interested in computers. Teaching himself D&D was an escape for the geeky kid that wasn t good at sports, as teaching myself Pascal and C was for me. Between us, the names and activities are different, but the story is the same. I particularly appreciated Wil s reflections on his teenage years. Like him, at that age, I often found myself as the youngest person in a room full of adults. Yet I was still a teenager, and like any teenager, did some things that I look back on with some embarrassment now. Wil was completely honest with himself he admitted crashing a golf cart on the Paramount studio lot, for instance, but also reminds me that he was a teenager then. He recognizes that he didn t always make the best choices and wasn t always successful with what he did, but isn t ashamed of himself either. That s helpful for me to remember; I shouldn t be unreasonably harsh on my 16-year-old self, and need to remember that I had to be a teenager too. I also identify with him as a dad. He wrote of counting the days until he could teach his boys about D&D, about passing on being a geek to his sons. I ve had a similar excitement about being able to help Jacob build his first computer. Already Jacob, who is 3, loves using the manual typewriter I cleaned up for him, and spent an hour using the adding machine I dug out on Sunday while I was watching the boys. (I regret that I didn t have time to take it apart and show him how it worked right then when he asked). And perhaps his 2nd-favorite present of Christmas was the $3.50 large-button calculator with solar cell power I got him as an impulse buy at the pharmacy the other day. He is particularly enamored with the square root button because a single press replaces all the numbers on the screen with completely different numbers! I can t find the exact passage now, but Wil wrote at one point about his transition from a career in acting to a career in writing. He said that he likes the feeling he gets when his writing can touch people. He s been able to redefine himself not as a guy that used to be an actor on Star Trek but a person that is a good author, now. I agree, and think his best work has been done with a keyboard instead of a camera. And that leaves me wondering where my career will take me. Yes, I m an author, but of technical books. Authors of technical books rarely touch people s hearts. There s a reason we read Shakespeare and Dickens in literature classes, but no high school English teacher has ever assigned Newton s Opticks, despite its incredible importance to the world. Newton revolutionized science, mathematics, and philosophy, but Opticks doesn t speak to the modern heart like Romeo and Jiuliet still does. Generations of people have learned more about the world from Shakespeare than from Newton. I don t have Wil s gift for writing such touching stories. I ve only been able to even approach that sort of thing once or twice, and it certainly won t make a career for me. Like Wil, I m rarely the youngest person in the room anymore. His days of being a famous teenage actor on a scifi series are long gone, as are mine of single-handedly defeating entire teams at jr. high programming contests. (OK, that s a stretch, but at the time it sure felt exciting.) But unlike him, I m not completely content with my niche yet. I blog about being a geek in rural Kansas, where there still aren t many. I m a dad, with an incredible family. And I write about programming, volunteer for Debian and a few other causes, and have a surprisingly satisfying job working for a company that builds lawn mowers. And yet, I have this unshakable feeling of unsettledness. That I need to stop and think more about what I really want to do with my life, perhaps cultivate some talents I don t yet have, or perhaps find a way to make my current path more meaningful. So I will take Wil s book as a challenge, to all those that were once sure of what their lives would look like, and are less sure with each passing year: take a chance, and make it yours. And on that score, perhaps I ve done more than I had realized at first. Terah and I took a big chance moving to Kansas, and another one when we bought my grandparents run-down house to fix up and live in. Perhaps it s not a bad idea to pause every few years and ask the question: Do I still like the direction I m heading? Can I change it? Wil Wheaton gives me lots to think about, in the form of easy-to-read reflections on his own life. I heartily recommend both Just a Geek and The Happiest Days of Our Lives. (And that has nothing to do with the fact that the Ubuntu machine he used to write the book probably had installed on it a few pieces of code that I wrote, I promise you.)

Debra and Ian at Tanagra. The Source. The Source at Yahoo. Jaldhar and the Source at Tanagra. Torvalds under two moons, his sky is grey. Git, his arms wide. Jaldhar of Tanagra, Git of Torvalds, at Lungha. Jaldhar and Git in court. The court of silence. Jaldhar, his eyes closed. #debian-devel, its arms wide, its thighs fat. Jaldhar, his eyes open! The Source in Git, its sails unfurled. Update: From Barak A. Pearlmutter who doesn't have a blog.

JOIN US, AS GNULIB HAS.
THERE IS NOTHING TO FEAR.
SUSPEND YOUR DOUBTS.
SUBMIT TO THE GIT.OR.CZ SUPERIORITY.
YOUR REPOSITORY WILL BE CONVERTED.
COMPLEXITY IS BLISS.
ALL CODE WILL BE REPLICATED.
ALL BRANCHES SERVE THE CONSENSUALITY.
YOUR INDIVIDUALITY WILL BE A
TREASURED ASSET OF THE GITHUB.

The short answer is no, but as our status page easily reflects, there has been lots of work going on during the last two weeks, once etch's release unblocked the way to upload new versions to unstable. This post intends to resume the trend of updating on the status of GNOME in Debian, after we ended up deciding we'd ship etch with 2.14 for a number of reasons, most notably some complications with the GTK 2.10 transition at that time. You'll be able to find other related news items in Debian GNOME team's website. What has the Debian GNOME team up to during the last 6 months? Our first priority was to focus on unstable's GNOME 2.14 packages again, in an attempt to fix any outstanding remaining bugs from our packaging, and get them in the best shape possible to deliver a polished GNOME desktop for etch. I think the result is really good, and Debian's default GNOME desktop is both very usable and attractive. In parallel, the preparations for a complete set of GNOME 2.16 packages continued in our Subversion repository and kept appearing, little by little, in experimental. The most visible consequence of our 2.16 efforts translated into nobse's backport of 2.16 for etch, which can be found in the corresponding repository. And then, with etch deep frozen and nearly ready to be released, GNOME 2.18 was released, and of course the GNOME team didn't wait too much to start working on it. Our current status is looking good: the Developer Platform is already available in unstable, although buildd's are fighting the builds on various architectures. When the dust settles (GTK 2.10's landing has generated quite a big cloud; we have a list of packages that still haven't completed the GTK+ 2.10 transition), we'll be able to prepare and upload the more complex Desktop components like the panel, nautilus, evolution or control-center. Unstable users should probably be seeing daily progress on this front, so keep an eye on your package managers! Although Debian 4.0 released with an old version of GNOME, vast amounts of time and work have been invested to release it with the necessary backported fixes and enhancements. The newer GNOME versions have been available in Debian official ftp archives in very reasonable timeframes; this has only been possible thanks to the restless efforts of the (fortunately) growing Debian GNOME team members: giskard, feedback, HE, lool, np237, slomo, shaka, sjoerd, xaiki and not forgetting our incredible bug triager, svena. Thanks! On the behind the scenes department, it's a pleasure to report that Lo c Minier and Jordi Mallach very recently joined the GNOME Foundation's board of advisors in representation of the Debian Project, replacing Matthew Garrett, who has been representing us for the last few years until he left the project. Thanks, Matthew!

Quoting Shakespeare's Sister:

It remains a radical act to be fat and happy [...], especially if you're a woman (for whom "jolly" fatness isn't an option). If you're fat, you're not only meant to be unhappy, but deeply ashamed of yourself, projecting at all times an apologetic nature, indicative of your everlasting remorse for having wrought your monstrous self upon the world. You are certainly not meant to be bold, or assertive, or confident and should you manage to overcome the constant drumbeat of messages that you are ugly and unsexy and have earned equally society's disdain and your own self-hatred, should you forget your place and walk into the world one day with your head held high, you are to be reminded by the cow-calls and contemptuous looks of perfect strangers that you are not supposed to have self-esteem; you don't deserve it. Rare indeed is the fat chick who manages to find contentment in her own skin, because everything around her is designed so that she will not.

Being publicly fat and happy is hard; being publicly, shamelessly, unshakably fat and happy is an act of both will and bravery.

Love yourself, whatever your size is!

Being Sedintary Is Bad For You, Not Being Fat

Looking out over Toronto from the top of the CN Tower, and still running for public office, I realize that I have taken several actions in my life for which I owe public apologies. Once, I blamed the entire history of Middle East politics on a hapless middle-aged retard just because he said that he liked malls. He is currently awaiting trial at the International Criminal Court in rainy The Hague. I would like to apologize to the people of the Netherlands for my part in inflicting an American on them. When I was 55, I strapped a large snowboard to each foot and sat in the middle of a narrow trail in Vermont, just to be a dick. I would like to apologize to all the skiers who injured themselves fatally in attempts to avoid me. Four hours later, I tried the same on a mountain in Utah. I would like to apologize to the American people for failing to injure any Mormons in that endeavor. In order to console myself, I broke into a beer brewery and upped the alcohol content to 4% by weight, in contravention of state law. I would like to apologize for wasting alcohol on the people of Utah. When I was three, my cousin hung up on an imaginary phone call with Mr. President and proceeded to place an imaginary phone call to Mr. Government. I became so overcome by stress from her antics that I cast away her mock aloe, peddled my tricycle to Boston, and pissed all over the First Church of Christ, Scientist. I would like to apologize for not having waited several years in order to piss on some Scientologists instead. I would like to apologize to Uncle Bill for having dragged his name into this blog entry, especially in such close proximity to the Scientologists, who hate him with aplomb. I would like to apologize for having re-enacted this episode every month since then, despite the protests of Mr. "I-am-not-a-Jew" Schwartz et al. Several years later, I defecated on the streets of Hoboken, Jersey City, Atlantic City, and Camden during rush-hour traffic. I would like to apologize for not doing this in every city in New Jersey (Newark in particular). Last month on a flight to Dar-us-Salaam, I noticed that Steve Martin was sitting across the aisle from me. While he was sleepily drooling into his pillow, I swiped his notebook. I would like to apologize to Steve Martin for plagiarizing this blog entry verbatim. I would also like to apologize for having sex with his wife Karen seventeen times. I would like to apologize to Karen for having sex with her three times without her consent, and I would like to apologize to both Steve and Karen for having sex with their children while pretending they were members of the Plimpton family. I would also like to apologize for having sex with their elderly Hawaiian turtle, whom I mistakenly believed was dead. In 1981, I dined at Sammy's Roumanian Steakhouse while wearing a purple zoot suit with a silver see-through mesh tank top. I would like to apologize to the Romanian people, the denizens of Chrystie Street, the City of New Orleans, the poblano farmers, the shiksa that gave me a lapdance, and the marmot named Stu who saw the pictures on the website of a Jewish dog rapist and promptly choked to death on his own vomit. In 1987, I dined at a Benihana with a girl who would later get tattoos and piercings. We sat at a hibachi with a truckload of Italian tourists who spoke no English at all. I would like to apologize for laughing hysterically at them when one asked, Du sprichst Deutsch? , and for getting blood all over the teppanyaki when I slit their throats after another asked, Quel pays ? On New Year's Eve in 1989, I dined at the same Benihana with a different girl whose command of Japanese was incomplete. When a middle-aged drunken British woman jammed her tongue into my mouth, a Brooklyn school bus driver laughed at me. After leaving the restaurant, he offered me a Werther's Original candy to get the taste of old woman out of my mouth. I would like to apologize for following him home and forcing him to watch the same episode of Full House over and over again for 37 hours. I would also like to apologize to Girl Scout Troop 5166 for calling them a bunch of stupid whores. While they still are a bunch of stupid whores, there was no way I could have known that at the time. Furthermore, when I was growing up, the term stupid whore approximated the use of Khidir beneath Momouteh and Shaka, when the walls fell. Finally, I would like to apologize for making a Star Trek reference when I could have said that thing about porchmonkeys instead. Now on with the campaign!

Search Results: "shaka"

12 December 2022

Russ Allbery: Review: The Unbroken

31 May 2022

Russ Allbery: Review: Maskerade

9 April 2021

Michael Prokop: A Ceph war story

14 June 2017

Michael Prokop: Grml 2017.05 Codename Freedatensuppe

28 April 2017

Russ Allbery: Review: Small Gods

25 December 2016

Russ Allbery: Review: A Man Called Ove

Russ Allbery: Review: A Man Called Ove

2 December 2010

Biella Coleman: If I could have, I would have

29 December 2009

John Goerzen: Review: The Happiest Days of Our Lives (by Wil Wheaton)

7 May 2009

Jaldhar Vyas: Shaka, When the Firewall Fell

18 April 2007

Jordi Mallach: Debian's GNOME 2.18: are we there yet?

5 January 2007

Amaya Rodrigo: Love yourself, whatever your size is!

13 August 2006

Clint Adams: Waiting for something to break