Jonathan Dowland: Propaganda A Secret Wish
How can I not have done one of these for Propaganda already?
How can I not have done one of these for Propaganda already?
How can I not have done one of these for Propaganda already?
How can I not have done one of these for Propaganda already?
| Illustrator: | Pauline Baynes |
| Series: | Chronicles of Narnia #7 |
| Publisher: | Collier Books |
| Copyright: | 1956 |
| Printing: | 1978 |
| ISBN: | 0-02-044210-6 |
| Format: | Mass market |
| Pages: | 184 |
"Sir," said Tirian, when he had greeted all these. "If I have read the chronicle aright, there should be another. Has not your Majesty two sisters? Where is Queen Susan?" "My sister Susan," answered Peter shortly and gravely, "is no longer a friend of Narnia." "Yes," said Eustace, "and whenever you've tried to get her to come and talk about Narnia or do anything about Narnia, she says 'What wonderful memories you have! Fancy your still thinking about all those funny games we used to play when we were children.'" "Oh Susan!" said Jill. "She's interested in nothing nowadays except nylons and lipstick and invitations. She always was a jolly sight too keen on being grown-up." "Grown-up indeed," said the Lady Polly. "I wish she would grow up. She wasted all her school time wanting to be the age she is now, and she'll waste all the rest of her life trying to stay that age. Her whole idea is to race on to the silliest time of one's life as quick as she can and then stop there as long as she can."There are so many obvious and dire problems with this passage, and so many others have written about it at length, that I will only add a few points. First, I find it interesting that neither Lucy nor Edmund says a thing. (I would like to think that Edmund knows better.) The real criticism comes from three characters who never interacted with Susan in the series: the two characters introduced after she was no longer allowed to return to Narnia, and a character from the story that predated hers. (And Eustace certainly has some gall to criticize someone else for treating Narnia as a childish game.) It also doesn't say anything good about Lewis that he puts his rather sexist attack on Susan into the mouths of two other female characters. Polly's criticism is a somewhat generic attack on puberty that could arguably apply to either sex (although "silliness" is usually reserved for women), but Jill makes the attack explicitly gendered. It's the attack of a girl who wants to be one of the boys on a girl who embraces things that are coded feminine, and there's a whole lot of politics around the construction of gender happening here that Lewis is blindly reinforcing and not grappling with at all. Plus, this is only barely supported by single sentences in The Voyage of the Dawn Treader and The Horse and His Boy and directly contradicts the earlier books. We're expected to believe that Susan the archer, the best swimmer, the most sensible and thoughtful of the four kids has abruptly changed her whole personality. Lewis could have made me believe Susan had soured on Narnia after the attempted kidnapping (and, although left unstated, presumably eventual attempted rape) in The Horse and His Boy, if one ignores the fact that incident supposedly happens before Prince Caspian where there is no sign of such a reaction. But not for those reasons, and not in that way. Thankfully, after this, the book gets better, starting with the Dwarfs, which is one of the two passages that had a profound influence on me. Except for one Dwarf who allied with Tirian, the Dwarfs reacted to the exposure of Shift's lies by disbelieving both Tirian and Shift, calling a pox on both their houses, and deciding to make their own side. During the last fight in front of the stable, they started killing whichever side looked like they were winning. (Although this is horrific in the story, I think this is accurate social commentary on a certain type of cynicism, even if I suspect Lewis may have been aiming it at atheists.) Eventually, they're thrown through the stable door by the Calormenes. However, rather than seeing the land of beauty and plenty that everyone else sees, they are firmly convinced they're in a dark, musty stable surrounded by refuse and dirty straw. This is, quite explicitly, not something imposed on them. Lucy rebukes Eustace for wishing Tash had killed them, and tries to make friends with them. Aslan tries to show them how wrong their perceptions are, to no avail. Their unwillingness to admit they were wrong is so strong that they make themselves believe that everything is worse than it actually is.
"You see," said Aslan. "They will not let us help them. They have chosen cunning instead of belief. Their prison is only in their own minds, yet they are in that prison; and so afraid of being taken in that they cannot be taken out."I grew up with the US evangelical version of Hell as a place of eternal torment, which in turn was used to justify religious atrocities in the name of saving people from Hell. But there is no Hell of that type in this book. There is a shadow into which many evil characters simply disappear, and there's this passage. Reading this was the first time I understood the alternative idea of Hell as the absence of God instead of active divine punishment. Lewis doesn't use the word "Hell," but it's obvious from context that the Dwarfs are in Hell. But it's not something Aslan does to them and no one wants them there; they could leave any time they wanted, but they're too unwilling to be wrong. You may have to be raised in conservative Christianity to understand how profoundly this rethinking of Hell (which Lewis tackles at greater length in The Great Divorce) undermines the system of guilt and fear that's used as motivation and control. It took me several re-readings and a lot of thinking about this passage, but this is where I stopped believing in a vengeful God who will eternally torture nonbelievers, and thus stopped believing in all of the other theology that goes with it. The second passage that changed me is Emeth's story. Emeth is a devout Calormene, a follower of Tash, who volunteered to enter the stable when Shift and his co-conspirators were claiming Aslan/Tash was inside. Some time after going through, he encounters Aslan, and this is part of his telling of that story (and yes, Lewis still has Calormenes telling stories as if they were British translators of the Arabian Nights):
[...] Lord, is it then true, as the Ape said, that thou and Tash are one? The Lion growled so that the earth shook (but his wrath was not against me) and said, It is false. Not because he and I are one, but because we are opposites, I take to me the services which thou hast done to him. For I and he are of such different kinds that no service which is vile can be done to me, and none which is not vile can be done to him. Therefore if any man swear by Tash and keep his oath for the oath's sake, it is by me that he has truly sworn, though he know it not, and it is I who reward him. And if any man do a cruelty in my name, then, though he says the name Aslan, it is Tash whom he serves and by Tash his deed is accepted. Dost thou understand, Child? I said, Lord, thou knowest how much I understand. But I said also (for the truth constrained me), Yet I have been seeking Tash all my days. Beloved, said the Glorious One, unless thy desire had been for me, thou wouldst not have sought so long and so truly. For all find what they truly seek.So, first, don't ever say this to anyone. It's horribly condescending and, since it's normally said by white Christians to other people, usually explicitly colonialist. Telling someone that their god is evil but since they seem to be a good person they're truly worshiping your god is only barely better than saying yours is the only true religion. But it is better, and as someone who, at the time, was wholly steeped in the belief that only Christians were saved and every follower of another religion was following Satan and was damned to Hell, this passage blew my mind. This was the first place I encountered the idea that someone who followed a different religion could be saved, or that God could transcend religion, and it came with exactly the context and justification that I needed given how close-minded I was at the time. Today, I would say that the Christian side of this analysis needs far more humility, and fobbing off all the evil done in the name of the Christian God by saying "oh, those people were really following Satan" is a total moral copout. But, nonetheless, Lewis opened a door for me that I was able to step through and move beyond to a less judgmental, dismissive, and hostile view of others. There's not much else in the book after this. It's mostly Lewis's charmingly Platonic view of the afterlife, in which the characters go inward and upward to truer and more complete versions of both Narnia and England and are reunited (very briefly) with every character of the series. Lewis knows not to try too hard to describe the indescribable, but it remains one of my favorite visions of an afterlife because it makes so explicit that this world is neither static or the last, but only the beginning of a new adventure. This final section of The Last Battle is deeply flawed, rather arrogant, a little bizarre, and involves more lectures on theology than precise description, but I still love it. By itself, it's not a bad ending for the series, although I don't think it has half the beauty or wonder of the end of The Voyage of the Dawn Treader. It's a shame about the rest of the book, and it's a worse shame that Lewis chose to sacrifice Susan on the altar of his prejudices. Those problems made it very hard to read this book again and make it impossible to recommend. Thankfully, you can read the series without it, and perhaps most readers would be better off imagining their own ending (or lack of ending) to Narnia than the one Lewis chose to give it. But the one redeeming quality The Last Battle will always have for me is that, despite all of its flaws, it was exactly the book that I needed to read when I read it. Rating: 4 out of 10
| Illustrator: | Pauline Baynes |
| Series: | Chronicles of Narnia #6 |
| Publisher: | Collier Books |
| Copyright: | 1955 |
| Printing: | 1978 |
| ISBN: | 0-02-044230-0 |
| Format: | Mass market |
| Pages: | 186 |
All of the faces they could see were certainly nice. Both the men and women looked kind and wise, and they seemed to come of a handsome race. But after the children had gone a few steps down the room they came to faces that looked a little different. These were very solemn faces. You felt you would have to mind your P's and Q's, if you ever met living people who looked like that. When they had gone a little farther, they found themselves among faces they didn't like: this was about the middle of the room. The faces here looked very strong and proud and happy, but they looked cruel. A little further on, they looked crueller. Further on again, they were still cruel but they no longer looked happy. They were even despairing faces: as if the people they belonged to had done dreadful things and also suffered dreadful things.The last statue is of a fierce, proud woman that Digory finds strikingly beautiful. (Lewis notes in an aside that Polly always said she never found anything specially beautiful about her. Here, as in The Silver Chair, the girl is the sensible one and things would have gone better if the boy had listened to her, a theme that I find immensely frustrating because Susan was the sensible one in the first two books of the series but then Lewis threw that away.) There is a bell in the middle of this hall, and the pillar that holds that bell has an inscription on it that I think every kid who grew up on Narnia knows by heart.
Make your choice, adventurous Stranger;Polly has no intention of striking the bell, but Digory fights her and does it anyway, waking Jadis from where she sat as the final statue in the hall and setting off one of the greatest reimaginings of a villain in children's literature. Jadis will, of course, become the White Witch who holds Narnia in endless winter some thousand Narnian years later. But the White Witch was a mediocre villain at best, the sort of obvious and cruel villain common in short fairy tales where the author isn't interested in doing much characterization. She exists to be evil, do bad things, and be defeated. She has a few good moments in conflict with Aslan, but that's about it. Jadis in this book is another matter entirely: proud, brilliant, dangerous, and creative. The death of everything on Charn was Jadis's doing: an intentional spell, used to claim a victory of sorts from the jaws of defeat by her sister in a civil war. (I find it fascinating that Lewis puts aside his normally sexist roles here.) Despite the best attempts of the kids to lose her both in Charn and in the Wood (which is inimical to her, in another nice bit of world-building), she manages to get back to England with them. The result is a remarkably good bit of villain characterization. Jadis is totally out of her element, used to a world-spanning empire run with magic and (from what hints we get) vaguely medieval technology. Her plan to take over their local country and eventually the world should be absurd and is played somewhat for laughs. Her magic, which is her great weapon, doesn't even work in England. But Jadis learns at a speed that the reader can watch. She's observant, she pays attention to things that don't fit her expectations, she changes plans, and she moves with predatory speed. Within a few hours in London she's stolen jewels and a horse and carriage, and the local police seem entirely overmatched. There's no way that one person without magic should be a real danger to England around the turn of the 20th century, but by the time the kids manage to pull her back into the Wood, you're not entirely sure England would have been safe. A chaotic confrontation, plus the ability of the rings to work their magic through transitive human contact, ends up with the kids, Uncle Andrew, Jadis, a taxicab driver and his horse all transported through the Wood to a new world. In this case, literally a new world: Narnia at the point of its creation. Here again, Lewis translates Christian myth, in this case the Genesis creation story, into a more vivid and in many ways more beautiful story than the original. Aslan singing the world into existence is an incredible image, as is the newly-created world so bursting with life that even things that normally could not grow will do so. (Which, of course, is why there is a lamp post burning in the middle of the western forest of Narnia for the Pevensie kids to find later.) I think my favorite part is the creation of the stars, but the whole sequence is great. There's also an insightful bit of human psychology. Uncle Andrew can't believe that a lion is singing, so he convinces himself that Aslan is not singing, and thus prevents himself from making any sense of the talking animals later.
Strike the bell and bide the danger,
Or wonder, till it drives you mad,
What would have followed if you had.
Now the trouble about trying to make yourself stupider than you really are is that you very often succeed.As with a lot in Lewis, he probably meant this as a statement about faith, but it generalizes well beyond the religious context. What disappointed me about the creation story, though, is the animals. I didn't notice this as a kid, but this re-read has sensitized me to how Lewis consistently treats the talking animals as less than humans even though he celebrates them. That happens here too: the newly-created, newly-awakened animals are curious and excited but kind of dim. Some of this is an attempt to show that they're young and are just starting to learn, but it also seems to be an excuse for Aslan to set up a human king and queen over them instead of teaching them directly how to deal with the threat of Jadis who the children inadvertently introduced into the world. The other thing I dislike about The Magician's Nephew is that the climax is unnecessarily cruel. Once Digory realizes the properties of the newly-created world, he hopes to find a way to use that to heal his mother. Aslan points out that he is responsible for Jadis entering the world and instead sends him on a mission to obtain a fruit that, when planted, will ward Narnia against her for many years. The same fruit would heal his mother, and he has to choose Narnia over her. (It's a fairly explicit parallel to the Garden of Eden, except in this case Digory passes.) Aslan, in the end, gives Digory the fruit of the tree that grows, which is still sufficient to heal his mother, but this sequence made me angry when re-reading it. Aslan knew all along that what Digory is doing will let him heal his mother as well, but hides this from him to make it more of a test. It's cruel and mean; Aslan could have promised to heal Digory's mother and then seen if he would help Narnia without getting anything in return other than atoning for his error, but I suppose that was too transactional for Lewis's theology or something. Meh. But, despite that, the only reason why this is not the best Narnia book is because The Voyage of the Dawn Treader is the only Narnia book that also nails the ending. The Magician's Nephew, up through Charn, Jadis's rampage through London, and the initial creation of Narnia, is fully as good, perhaps better. It sags a bit at the end, partly because it tries to hard to make the Narnian animals humorous and partly because of the unnecessary emotional torture of Digory. But this still holds up as the second-best Narnia book, and one I thoroughly enjoyed re-reading. If anything, Jadis and Charn are even better than I remembered. Followed by the last book of the series, the somewhat notorious The Last Battle. Rating: 9 out of 10
It all started with the big bang! We nearly lost 33 of 36 disks on a Proxmox/Ceph Cluster; this is the story of how we recovered them.
At the end of 2020, we eventually had a long outstanding maintenance window for taking care of system upgrades at a customer. During this maintenance window, which involved reboots of server systems, the involved Ceph cluster unexpectedly went into a critical state. What was planned to be a few hours of checklist work in the early evening turned out to be an emergency case; let s call it a nightmare (not only because it included a big part of the night). Since we have learned a few things from our post mortem and RCA, it s worth sharing those with others. But first things first, let s step back and clarify what we had to deal with.
The system and its upgrade
One part of the upgrade included 3 Debian servers (we re calling them server1, server2 and server3 here), running on Proxmox v5 + Debian/stretch with 12 Ceph OSDs each (65.45TB in total), a so-called Proxmox Hyper-Converged Ceph Cluster.
First, we went for upgrading the Proxmox v5/stretch system to Proxmox v6/buster, before updating Ceph Luminous v12.2.13 to the latest v14.2 release, supported by Proxmox v6/buster. The Proxmox upgrade included updating corosync from v2 to v3. As part of this upgrade, we had to apply some configuration changes, like adjust ring0 + ring1 address settings and add a mon_host configuration to the Ceph configuration.
During the first two servers reboots, we noticed configuration glitches. After fixing those, we went for a reboot of the third server as well. Then we noticed that several Ceph OSDs were unexpectedly down. The NTP service wasn t working as expected after the upgrade. The underlying issue is a race condition of ntp with systemd-timesyncd (see #889290). As a result, we had clock skew problems with Ceph, indicating that the Ceph monitors clocks aren t running in sync (which is essential for proper Ceph operation). We initially assumed that our Ceph OSD failure derived from this clock skew problem, so we took care of it. After yet another round of reboots, to ensure the systems are running all with identical and sane configurations and services, we noticed lots of failing OSDs. This time all but three OSDs (19, 21 and 22) were down:
% sudo ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 65.44138 root default -2 21.81310 host server1 0 hdd 1.08989 osd.0 down 1.00000 1.00000 1 hdd 1.08989 osd.1 down 1.00000 1.00000 2 hdd 1.63539 osd.2 down 1.00000 1.00000 3 hdd 1.63539 osd.3 down 1.00000 1.00000 4 hdd 1.63539 osd.4 down 1.00000 1.00000 5 hdd 1.63539 osd.5 down 1.00000 1.00000 18 hdd 2.18279 osd.18 down 1.00000 1.00000 20 hdd 2.18179 osd.20 down 1.00000 1.00000 28 hdd 2.18179 osd.28 down 1.00000 1.00000 29 hdd 2.18179 osd.29 down 1.00000 1.00000 30 hdd 2.18179 osd.30 down 1.00000 1.00000 31 hdd 2.18179 osd.31 down 1.00000 1.00000 -4 21.81409 host server2 6 hdd 1.08989 osd.6 down 1.00000 1.00000 7 hdd 1.08989 osd.7 down 1.00000 1.00000 8 hdd 1.63539 osd.8 down 1.00000 1.00000 9 hdd 1.63539 osd.9 down 1.00000 1.00000 10 hdd 1.63539 osd.10 down 1.00000 1.00000 11 hdd 1.63539 osd.11 down 1.00000 1.00000 19 hdd 2.18179 osd.19 up 1.00000 1.00000 21 hdd 2.18279 osd.21 up 1.00000 1.00000 22 hdd 2.18279 osd.22 up 1.00000 1.00000 32 hdd 2.18179 osd.32 down 1.00000 1.00000 33 hdd 2.18179 osd.33 down 1.00000 1.00000 34 hdd 2.18179 osd.34 down 1.00000 1.00000 -3 21.81419 host server3 12 hdd 1.08989 osd.12 down 1.00000 1.00000 13 hdd 1.08989 osd.13 down 1.00000 1.00000 14 hdd 1.63539 osd.14 down 1.00000 1.00000 15 hdd 1.63539 osd.15 down 1.00000 1.00000 16 hdd 1.63539 osd.16 down 1.00000 1.00000 17 hdd 1.63539 osd.17 down 1.00000 1.00000 23 hdd 2.18190 osd.23 down 1.00000 1.00000 24 hdd 2.18279 osd.24 down 1.00000 1.00000 25 hdd 2.18279 osd.25 down 1.00000 1.00000 35 hdd 2.18179 osd.35 down 1.00000 1.00000 36 hdd 2.18179 osd.36 down 1.00000 1.00000 37 hdd 2.18179 osd.37 down 1.00000 1.00000Our blood pressure increased slightly! Did we just lose all of our cluster? What happened, and how can we get all the other OSDs back? We stumbled upon this beauty in our logs:
kernel: [ 73.697957] XFS (sdl1): SB stripe unit sanity check failed kernel: [ 73.698002] XFS (sdl1): Metadata corruption detected at xfs_sb_read_verify+0x10e/0x180 [xfs], xfs_sb block 0xffffffffffffffff kernel: [ 73.698799] XFS (sdl1): Unmount and run xfs_repair kernel: [ 73.699199] XFS (sdl1): First 128 bytes of corrupted metadata buffer: kernel: [ 73.699677] 00000000: 58 46 53 42 00 00 10 00 00 00 00 00 00 00 62 00 XFSB..........b. kernel: [ 73.700205] 00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ kernel: [ 73.700836] 00000020: 62 44 2b c0 e6 22 40 d7 84 3d e1 cc 65 88 e9 d8 bD+.."@..=..e... kernel: [ 73.701347] 00000030: 00 00 00 00 00 00 40 08 00 00 00 00 00 00 01 00 ......@......... kernel: [ 73.701770] 00000040: 00 00 00 00 00 00 01 01 00 00 00 00 00 00 01 02 ................ ceph-disk[4240]: mount: /var/lib/ceph/tmp/mnt.jw367Y: mount(2) system call failed: Structure needs cleaning. ceph-disk[4240]: ceph-disk: Mounting filesystem failed: Command '['/bin/mount', '-t', u'xfs', '-o', 'noatime,inode64', '--', '/dev/disk/by-parttypeuuid/4fbd7e29-9d25-41b8-afd0-062c0ceff05d.cdda39ed-5 ceph/tmp/mnt.jw367Y']' returned non-zero exit status 32 kernel: [ 73.702162] 00000050: 00 00 00 01 00 00 18 80 00 00 00 04 00 00 00 00 ................ kernel: [ 73.702550] 00000060: 00 00 06 48 bd a5 10 00 08 00 00 02 00 00 00 00 ...H............ kernel: [ 73.702975] 00000070: 00 00 00 00 00 00 00 00 0c 0c 0b 01 0d 00 00 19 ................ kernel: [ 73.703373] XFS (sdl1): SB validate failed with error -117.The same issue was present for the other failing OSDs. We hoped, that the data itself was still there, and only the mounting of the XFS partitions failed. The Ceph cluster was initially installed in 2017 with Ceph jewel/10.2 with the OSDs on filestore (nowadays being a legacy approach to storing objects in Ceph). However, we migrated the disks to bluestore since then (with ceph-disk and not yet via ceph-volume what s being used nowadays). Using ceph-disk introduces these 100MB XFS partitions containing basic metadata for the OSD. Given that we had three working OSDs left, we decided to investigate how to rebuild the failing ones. Some folks on #ceph (thanks T1, ormandj + peetaur!) were kind enough to share how working XFS partitions looked like for them. After creating a backup (via dd), we tried to re-create such an XFS partition on server1. We noticed that even mounting a freshly created XFS partition failed:
synpromika@server1 ~ % sudo mkfs.xfs -f -i size=2048 -m uuid="4568c300-ad83-4288-963e-badcd99bf54f" /dev/sdc1
meta-data=/dev/sdc1 isize=2048 agcount=4, agsize=6272 blks
= sectsz=4096 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=0
= reflink=0
data = bsize=4096 blocks=25088, imaxpct=25
= sunit=128 swidth=64 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=1608, version=2
= sectsz=4096 sunit=1 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
synpromika@server1 ~ % sudo mount /dev/sdc1 /mnt/ceph-recovery
SB stripe unit sanity check failed
Metadata corruption detected at 0x433840, xfs_sb block 0x0/0x1000
libxfs_writebufr: write verifer failed on xfs_sb bno 0x0/0x1000
cache_node_purge: refcount was 1, not zero (node=0x1d3c400)
SB stripe unit sanity check failed
Metadata corruption detected at 0x433840, xfs_sb block 0x18800/0x1000
libxfs_writebufr: write verifer failed on xfs_sb bno 0x18800/0x1000
SB stripe unit sanity check failed
Metadata corruption detected at 0x433840, xfs_sb block 0x0/0x1000
libxfs_writebufr: write verifer failed on xfs_sb bno 0x0/0x1000
SB stripe unit sanity check failed
Metadata corruption detected at 0x433840, xfs_sb block 0x24c00/0x1000
libxfs_writebufr: write verifer failed on xfs_sb bno 0x24c00/0x1000
SB stripe unit sanity check failed
Metadata corruption detected at 0x433840, xfs_sb block 0xc400/0x1000
libxfs_writebufr: write verifer failed on xfs_sb bno 0xc400/0x1000
releasing dirty buffer (bulk) to free list!releasing dirty buffer (bulk) to free list!releasing dirty buffer (bulk) to free list!releasing dirty buffer (bulk) to free list!found dirty buffer (bulk) on free list!bad magic number
bad magic number
Metadata corruption detected at 0x433840, xfs_sb block 0x0/0x1000
libxfs_writebufr: write verifer failed on xfs_sb bno 0x0/0x1000
releasing dirty buffer (bulk) to free list!mount: /mnt/ceph-recovery: wrong fs type, bad option, bad superblock on /dev/sdc1, missing codepage or helper program, or other error.
Ouch. This very much looked related to the actual issue we re seeing. So we tried to execute mkfs.xfs with a bunch of different sunit/swidth settings. Using -d sunit=512 -d swidth=512 at least worked then, so we decided to force its usage in the creation of our OSD XFS partition. This brought us a working XFS partition. Please note, sunit must not be larger than swidth (more on that later!).
Then we reconstructed how to restore all the metadata for the OSD (activate.monmap, active, block_uuid, bluefs, ceph_fsid, fsid, keyring, kv_backend, magic, mkfs_done, ready, require_osd_release, systemd, type, whoami). To identify the UUID, we can read the data from ceph --format json osd dump , like this for all our OSDs (Zsh syntax ftw!):
synpromika@server1 ~ % for f in 0..37 ; printf "osd-$f: %s\n" "$(sudo ceph --format json osd dump jq -r ".osds[] select(.osd==$f) .uuid")" osd-0: 4568c300-ad83-4288-963e-badcd99bf54f osd-1: e573a17a-ccde-4719-bdf8-eef66903ca4f osd-2: 0e1b2626-f248-4e7d-9950-f1a46644754e osd-3: 1ac6a0a2-20ee-4ed8-9f76-d24e900c800c [...]Identifying the corresponding raw device for each OSD UUID is possible via:
synpromika@server1 ~ % UUID="4568c300-ad83-4288-963e-badcd99bf54f" synpromika@server1 ~ % readlink -f /dev/disk/by-partuuid/"$ UUID " /dev/sdc1The OSD s key ID can be retrieved via:
synpromika@server1 ~ % OSD_ID=0 synpromika@server1 ~ % sudo ceph auth get osd."$ OSD_ID " -f json 2>/dev/null jq -r '.[] .key' AQCKFpZdm0We[...]Now we also need to identify the underlying block device:
synpromika@server1 ~ % OSD_ID=0 synpromika@server1 ~ % sudo ceph osd metadata osd."$ OSD_ID " -f json jq -r '.bluestore_bdev_partition_path' /dev/sdc2With all of this, we reconstructed the
keyring, fsid, whoami, block + block_uuid files. All the other files inside the XFS metadata partition are identical on each OSD. So after placing and adjusting the corresponding metadata on the XFS partition for Ceph usage, we got a working OSD hurray! Since we had to fix yet another 32 OSDs, we decided to automate this XFS partitioning and metadata recovery procedure.
We had a network share available on /srv/backup for storing backups of existing partition data. On each server, we tested the procedure with one single OSD before iterating over the list of remaining failing OSDs. We started with a shell script on server1, then adjusted the script for server2 and server3. This is the script, as we executed it on the 3rd server.
Thanks to this, we managed to get the Ceph cluster up and running again. We didn t want to continue with the Ceph upgrade itself during the night though, as we wanted to know exactly what was going on and why the system behaved like that. Time for RCA!
Root Cause Analysis
So all but three OSDs on server2 failed, and the problem seems to be related to XFS. Therefore, our starting point for the RCA was, to identify what was different on server2, as compared to server1 + server3. My initial assumption was that this was related to some firmware issues with the involved controller (and as it turned out later, I was right!). The disks were attached as JBOD devices to a ServeRAID M5210 controller (with a stripe size of 512). Firmware state:
synpromika@server1 ~ % sudo storcli64 /c0 show all grep '^Firmware' Firmware Package Build = 24.16.0-0092 Firmware Version = 4.660.00-8156 synpromika@server2 ~ % sudo storcli64 /c0 show all grep '^Firmware' Firmware Package Build = 24.21.0-0112 Firmware Version = 4.680.00-8489 synpromika@server3 ~ % sudo storcli64 /c0 show all grep '^Firmware' Firmware Package Build = 24.16.0-0092 Firmware Version = 4.660.00-8156This looked very promising, as server2 indeed runs with a different firmware version on the controller. But how so? Well, the motherboard of server2 got replaced by a Lenovo/IBM technician in January 2020, as we had a failing memory slot during a memory upgrade. As part of this procedure, the Lenovo/IBM technician installed the latest firmware versions. According to our documentation, some OSDs were rebuilt (due to the filestore->bluestore migration) in March and April 2020. It turned out that precisely those OSDs were the ones that survived the upgrade. So the surviving drives were created with a different firmware version running on the involved controller. All the other OSDs were created with an older controller firmware. But what difference does this make? Now let s check firmware changelogs. For the 24.21.0-0097 release we found this:
- Cannot create or mount xfs filesystem using xfsprogs 4.19.x kernel 4.20(SCGCQ02027889) - xfs_info command run on an XFS file system created on a VD of strip size 1M shows sunit and swidth as 0(SCGCQ02056038)Our XFS problem certainly was related to the controller s firmware. We also recalled that our monitoring system reported different sunit settings for the OSDs that were rebuilt in March and April. For example, OSD 21 was recreated and got different sunit settings:
WARN server2.example.org Mount options of /var/lib/ceph/osd/ceph-21 WARN - Missing: sunit=1024, Exceeding: sunit=512We compared the new OSD 21 with an existing one (OSD 25 on server3):
synpromika@server2 ~ % systemctl show var-lib-ceph-osd-ceph\\x2d21.mount grep sunit Options=rw,noatime,attr2,inode64,sunit=512,swidth=512,noquota synpromika@server3 ~ % systemctl show var-lib-ceph-osd-ceph\\x2d25.mount grep sunit Options=rw,noatime,attr2,inode64,sunit=1024,swidth=512,noquotaThanks to our documentation, we could compare execution logs of their creation:
% diff -u ceph-disk-osd-25.log ceph-disk-osd-21.log
-synpromika@server2 ~ % sudo ceph-disk -v prepare --bluestore /dev/sdj --osd-id 25
+synpromika@server3 ~ % sudo ceph-disk -v prepare --bluestore /dev/sdi --osd-id 21
[...]
-command_check_call: Running command: /sbin/mkfs -t xfs -f -i size=2048 -- /dev/sdj1
-meta-data=/dev/sdj1 isize=2048 agcount=4, agsize=6272 blks
[...]
+command_check_call: Running command: /sbin/mkfs -t xfs -f -i size=2048 -- /dev/sdi1
+meta-data=/dev/sdi1 isize=2048 agcount=4, agsize=6336 blks
= sectsz=4096 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0
-data = bsize=4096 blocks=25088, imaxpct=25
- = sunit=128 swidth=64 blks
+data = bsize=4096 blocks=25344, imaxpct=25
+ = sunit=64 swidth=64 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=1608, version=2
= sectsz=4096 sunit=1 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
[...]
So back then, we even tried to track this down but couldn t make sense of it yet. But now this sounds very much like it is related to the problem we saw with this Ceph/XFS failure. We follow Occam s razor, assuming the simplest explanation is usually the right one, so let s check the disk properties and see what differs:
synpromika@server1 ~ % sudo blockdev --getsz --getsize64 --getss --getpbsz --getiomin --getioopt /dev/sdk 4685545472 2398999281664 512 4096 524288 262144 synpromika@server2 ~ % sudo blockdev --getsz --getsize64 --getss --getpbsz --getiomin --getioopt /dev/sdk 4685545472 2398999281664 512 4096 262144 262144See the difference between server1 and server2 for identical disks? The
getiomin option now reports something different for them:
synpromika@server1 ~ % sudo blockdev --getiomin /dev/sdk 524288 synpromika@server1 ~ % cat /sys/block/sdk/queue/minimum_io_size 524288 synpromika@server2 ~ % sudo blockdev --getiomin /dev/sdk 262144 synpromika@server2 ~ % cat /sys/block/sdk/queue/minimum_io_size 262144It doesn t make sense that the minimum I/O size (iomin, AKA
BLKIOMIN) is bigger than the optimal I/O size (ioopt, AKA BLKIOOPT). This leads us to Bug 202127 cannot mount or create xfs on a 597T device, which matches our findings here. But why did this XFS partition work in the past and fails now with the newer kernel version?
The XFS behaviour change
Now given that we have backups of all the XFS partition, we wanted to track down, a) when this XFS behaviour was introduced, and b) whether, and if so how it would be possible to reuse the XFS partition without having to rebuild it from scratch (e.g. if you would have no working Ceph OSD or backups left).
Let s look at such a failing XFS partition with the Grml live system:
root@grml ~ # grml-version grml64-full 2020.06 Release Codename Ausgehfuahangl [2020-06-24] root@grml ~ # uname -a Linux grml 5.6.0-2-amd64 #1 SMP Debian 5.6.14-2 (2020-06-09) x86_64 GNU/Linux root@grml ~ # grml-hostname grml-2020-06 Setting hostname to grml-2020-06: done root@grml ~ # exec zsh root@grml-2020-06 ~ # dpkg -l xfsprogs util-linux Desired=Unknown/Install/Remove/Purge/Hold Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend / Err?=(none)/Reinst-required (Status,Err: uppercase=bad) / Name Version Architecture Description +++-==============-============-============-========================================= ii util-linux 2.35.2-4 amd64 miscellaneous system utilities ii xfsprogs 5.6.0-1+b2 amd64 Utilities for managing the XFS filesystemThere it s failing, no matter which mount option we try:
root@grml-2020-06 ~ # mount ./sdd1.dd /mnt mount: /mnt: mount(2) system call failed: Structure needs cleaning. root@grml-2020-06 ~ # dmesg tail -30 [...] [ 64.788640] XFS (loop1): SB stripe unit sanity check failed [ 64.788671] XFS (loop1): Metadata corruption detected at xfs_sb_read_verify+0x102/0x170 [xfs], xfs_sb block 0xffffffffffffffff [ 64.788671] XFS (loop1): Unmount and run xfs_repair [ 64.788672] XFS (loop1): First 128 bytes of corrupted metadata buffer: [ 64.788673] 00000000: 58 46 53 42 00 00 10 00 00 00 00 00 00 00 62 00 XFSB..........b. [ 64.788674] 00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 64.788675] 00000020: 32 b6 dc 35 53 b7 44 96 9d 63 30 ab b3 2b 68 36 2..5S.D..c0..+h6 [ 64.788675] 00000030: 00 00 00 00 00 00 40 08 00 00 00 00 00 00 01 00 ......@......... [ 64.788675] 00000040: 00 00 00 00 00 00 01 01 00 00 00 00 00 00 01 02 ................ [ 64.788676] 00000050: 00 00 00 01 00 00 18 80 00 00 00 04 00 00 00 00 ................ [ 64.788677] 00000060: 00 00 06 48 bd a5 10 00 08 00 00 02 00 00 00 00 ...H............ [ 64.788677] 00000070: 00 00 00 00 00 00 00 00 0c 0c 0b 01 0d 00 00 19 ................ [ 64.788679] XFS (loop1): SB validate failed with error -117. root@grml-2020-06 ~ # mount -t xfs -o rw,relatime,attr2,inode64,sunit=1024,swidth=512,noquota ./sdd1.dd /mnt/ mount: /mnt: wrong fs type, bad option, bad superblock on /dev/loop1, missing codepage or helper program, or other error. 32 root@grml-2020-06 ~ # dmesg tail -1 [ 66.342976] XFS (loop1): stripe width (512) must be a multiple of the stripe unit (1024) root@grml-2020-06 ~ # mount -t xfs -o rw,relatime,attr2,inode64,sunit=512,swidth=512,noquota ./sdd1.dd /mnt/ mount: /mnt: mount(2) system call failed: Structure needs cleaning. 32 root@grml-2020-06 ~ # dmesg tail -14 [ 66.342976] XFS (loop1): stripe width (512) must be a multiple of the stripe unit (1024) [ 80.751277] XFS (loop1): SB stripe unit sanity check failed [ 80.751323] XFS (loop1): Metadata corruption detected at xfs_sb_read_verify+0x102/0x170 [xfs], xfs_sb block 0xffffffffffffffff [ 80.751324] XFS (loop1): Unmount and run xfs_repair [ 80.751325] XFS (loop1): First 128 bytes of corrupted metadata buffer: [ 80.751327] 00000000: 58 46 53 42 00 00 10 00 00 00 00 00 00 00 62 00 XFSB..........b. [ 80.751328] 00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 80.751330] 00000020: 32 b6 dc 35 53 b7 44 96 9d 63 30 ab b3 2b 68 36 2..5S.D..c0..+h6 [ 80.751331] 00000030: 00 00 00 00 00 00 40 08 00 00 00 00 00 00 01 00 ......@......... [ 80.751331] 00000040: 00 00 00 00 00 00 01 01 00 00 00 00 00 00 01 02 ................ [ 80.751332] 00000050: 00 00 00 01 00 00 18 80 00 00 00 04 00 00 00 00 ................ [ 80.751333] 00000060: 00 00 06 48 bd a5 10 00 08 00 00 02 00 00 00 00 ...H............ [ 80.751334] 00000070: 00 00 00 00 00 00 00 00 0c 0c 0b 01 0d 00 00 19 ................ [ 80.751338] XFS (loop1): SB validate failed with error -117.Also xfs_repair doesn t help either:
root@grml-2020-06 ~ # xfs_info ./sdd1.dd
meta-data=./sdd1.dd isize=2048 agcount=4, agsize=6272 blks
= sectsz=4096 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=0, rmapbt=0
= reflink=0
data = bsize=4096 blocks=25088, imaxpct=25
= sunit=128 swidth=64 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=1608, version=2
= sectsz=4096 sunit=1 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
root@grml-2020-06 ~ # xfs_repair ./sdd1.dd
Phase 1 - find and verify superblock...
bad primary superblock - bad stripe width in superblock !!!
attempting to find secondary superblock...
..............................................................................................Sorry, could not find valid secondary superblock
Exiting now.
With the SB stripe unit sanity check failed message, we could easily track this down to the following commit fa4ca9c:
% git show fa4ca9c5574605d1e48b7e617705230a0640b6da cat
commit fa4ca9c5574605d1e48b7e617705230a0640b6da
Author: Dave Chinner <dchinner@redhat.com>
Date: Tue Jun 5 10:06:16 2018 -0700
xfs: catch bad stripe alignment configurations
When stripe alignments are invalid, data alignment algorithms in the
allocator may not work correctly. Ensure we catch superblocks with
invalid stripe alignment setups at mount time. These data alignment
mismatches are now detected at mount time like this:
XFS (loop0): SB stripe unit sanity check failed
XFS (loop0): Metadata corruption detected at xfs_sb_read_verify+0xab/0x110, xfs_sb block 0xffffffffffffffff
XFS (loop0): Unmount and run xfs_repair
XFS (loop0): First 128 bytes of corrupted metadata buffer:
0000000091c2de02: 58 46 53 42 00 00 10 00 00 00 00 00 00 00 10 00 XFSB............
0000000023bff869: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000000cdd8c893: 17 32 37 15 ff ca 46 3d 9a 17 d3 33 04 b5 f1 a2 .27...F=...3....
000000009fd2844f: 00 00 00 00 00 00 00 04 00 00 00 00 00 00 06 d0 ................
0000000088e9b0bb: 00 00 00 00 00 00 06 d1 00 00 00 00 00 00 06 d2 ................
00000000ff233a20: 00 00 00 01 00 00 10 00 00 00 00 01 00 00 00 00 ................
000000009db0ac8b: 00 00 03 60 e1 34 02 00 08 00 00 02 00 00 00 00 ... .4..........
00000000f7022460: 00 00 00 00 00 00 00 00 0c 09 0b 01 0c 00 00 19 ................
XFS (loop0): SB validate failed with error -117.
And the mount fails.
Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Carlos Maiolino <cmaiolino@redhat.com>
Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
diff --git fs/xfs/libxfs/xfs_sb.c fs/xfs/libxfs/xfs_sb.c
index b5dca3c8c84d..c06b6fc92966 100644
--- fs/xfs/libxfs/xfs_sb.c
+++ fs/xfs/libxfs/xfs_sb.c
@@ -278,6 +278,22 @@ xfs_mount_validate_sb(
return -EFSCORRUPTED;
+ if (sbp->sb_unit)
+ if (!xfs_sb_version_hasdalign(sbp)
+ sbp->sb_unit > sbp->sb_width
+ (sbp->sb_width % sbp->sb_unit) != 0)
+ xfs_notice(mp, "SB stripe unit sanity check failed");
+ return -EFSCORRUPTED;
+
+ else if (xfs_sb_version_hasdalign(sbp))
+ xfs_notice(mp, "SB stripe alignment sanity check failed");
+ return -EFSCORRUPTED;
+ else if (sbp->sb_width)
+ xfs_notice(mp, "SB stripe width sanity check failed");
+ return -EFSCORRUPTED;
+
+
+
if (xfs_sb_version_hascrc(&mp->m_sb) &&
sbp->sb_blocksize < XFS_MIN_CRC_BLOCKSIZE)
xfs_notice(mp, "v5 SB sanity check failed");
This change is included in kernel versions 4.18-rc1 and newer:
% git describe --contains fa4ca9c5574605d1e48 v4.18-rc1~37^2~14Now let s try with an older kernel version (4.9.0), using old Grml 2017.05 release:
root@grml ~ # grml-version
grml64-small 2017.05 Release Codename Freedatensuppe [2017-05-31]
root@grml ~ # uname -a
Linux grml 4.9.0-1-grml-amd64 #1 SMP Debian 4.9.29-1+grml.1 (2017-05-24) x86_64 GNU/Linux
root@grml ~ # lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 9.0 (stretch)
Release: 9.0
Codename: stretch
root@grml ~ # grml-hostname grml-2017-05
Setting hostname to grml-2017-05: done
root@grml ~ # exec zsh
root@grml-2017-05 ~ #
root@grml-2017-05 ~ # xfs_info ./sdd1.dd
xfs_info: ./sdd1.dd is not a mounted XFS filesystem
1 root@grml-2017-05 ~ # xfs_repair ./sdd1.dd
Phase 1 - find and verify superblock...
bad primary superblock - bad stripe width in superblock !!!
attempting to find secondary superblock...
..............................................................................................Sorry, could not find valid secondary superblock
Exiting now.
1 root@grml-2017-05 ~ # mount ./sdd1.dd /mnt
root@grml-2017-05 ~ # mount -t xfs
/root/sdd1.dd on /mnt type xfs (rw,relatime,attr2,inode64,sunit=1024,swidth=512,noquota)
root@grml-2017-05 ~ # ls /mnt
activate.monmap active block block_uuid bluefs ceph_fsid fsid keyring kv_backend magic mkfs_done ready require_osd_release systemd type whoami
root@grml-2017-05 ~ # xfs_info /mnt
meta-data=/dev/loop1 isize=2048 agcount=4, agsize=6272 blks
= sectsz=4096 attr=2, projid32bit=1
= crc=1 finobt=1 spinodes=0 rmapbt=0
= reflink=0
data = bsize=4096 blocks=25088, imaxpct=25
= sunit=128 swidth=64 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal bsize=4096 blocks=1608, version=2
= sectsz=4096 sunit=1 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
Mounting there indeed works! Now, if we mount the filesystem with new and proper sunit/swidth settings using the older kernel, it should rewrite them on disk:
root@grml-2017-05 ~ # mount -t xfs -o sunit=512,swidth=512 ./sdd1.dd /mnt/ root@grml-2017-05 ~ # umount /mnt/And indeed, mounting this rewritten filesystem then also works with newer kernels:
root@grml-2020-06 ~ # mount ./sdd1.rewritten /mnt/
root@grml-2020-06 ~ # xfs_info /root/sdd1.rewritten
meta-data=/dev/loop1 isize=2048 agcount=4, agsize=6272 blks
= sectsz=4096 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=0, rmapbt=0
= reflink=0
data = bsize=4096 blocks=25088, imaxpct=25
= sunit=64 swidth=64 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=1608, version=2
= sectsz=4096 sunit=1 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
root@grml-2020-06 ~ # mount -t xfs
/root/sdd1.rewritten on /mnt type xfs (rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,sunit=512,swidth=512,noquota)
FTR: The sunit=512,swidth=512 from the xfs mount option is identical to xfs_info s output sunit=64,swidth=64 (because mount.xfs s sunit value is given in 512-byte block units, see man 5 xfs, and the xfs_info output reported here is in blocks with a block size (bsize) of 4096, so sunit = 512*512 := 64*4096 ).
mkfs uses minimum and optimal sizes for stripe unit and stripe width; you can check this e.g. via (note that server2 with fixed firmware version reports proper values, whereas server3 with broken controller firmware reports non-sense):
synpromika@server2 ~ % for i in /sys/block/sd*/queue/ ; do printf "%s: %s %s\n" "$i" "$(cat "$i"/minimum_io_size)" "$(cat "$i"/optimal_io_size)" ; done [...] /sys/block/sdc/queue/: 262144 262144 /sys/block/sdd/queue/: 262144 262144 /sys/block/sde/queue/: 262144 262144 /sys/block/sdf/queue/: 262144 262144 /sys/block/sdg/queue/: 262144 262144 /sys/block/sdh/queue/: 262144 262144 /sys/block/sdi/queue/: 262144 262144 /sys/block/sdj/queue/: 262144 262144 /sys/block/sdk/queue/: 262144 262144 /sys/block/sdl/queue/: 262144 262144 /sys/block/sdm/queue/: 262144 262144 /sys/block/sdn/queue/: 262144 262144 [...] synpromika@server3 ~ % for i in /sys/block/sd*/queue/ ; do printf "%s: %s %s\n" "$i" "$(cat "$i"/minimum_io_size)" "$(cat "$i"/optimal_io_size)" ; done [...] /sys/block/sdc/queue/: 524288 262144 /sys/block/sdd/queue/: 524288 262144 /sys/block/sde/queue/: 524288 262144 /sys/block/sdf/queue/: 524288 262144 /sys/block/sdg/queue/: 524288 262144 /sys/block/sdh/queue/: 524288 262144 /sys/block/sdi/queue/: 524288 262144 /sys/block/sdj/queue/: 524288 262144 /sys/block/sdk/queue/: 524288 262144 /sys/block/sdl/queue/: 524288 262144 /sys/block/sdm/queue/: 524288 262144 /sys/block/sdn/queue/: 524288 262144 [...]This is the underlying reason why the initially created XFS partitions were created with incorrect sunit/swidth settings. The broken firmware of server1 and server3 was the cause of the incorrect settings they were ignored by old(er) xfs/kernel versions, but treated as an error by new ones. Make sure to also read the XFS FAQ regarding How to calculate the correct sunit,swidth values for optimal performance . We also stumbled upon two interesting reads in RedHat s knowledge base: 5075561 + 2150101 (requires an active subscription, though) and #1835947. Am I affected? How to work around it? To check whether your XFS mount points are affected by this issue, the following command line should be useful:
awk '$3 == "xfs" print $2 ' /proc/self/mounts while read mount ; do echo -n "$mount " ; xfs_info $mount awk '$0 ~ "swidth" gsub(/.*=/,"",$2); gsub(/.*=/,"",$3); print $2,$3 ' awk ' if ($1 > $2) print "impacted"; else print "OK" ' ; doneIf you run into the above situation, the only known solution to get your original XFS partition working again, is to boot into an older kernel version again (4.17 or older), mount the XFS partition with correct sunit/swidth settings and then boot back into your new system (kernel version wise). Lessons learned
| Publisher: | Bantam Spectra |
| Copyright: | May 1988 |
| Printing: | July 1989 |
| ISBN: | 0-553-27903-3 |
| Format: | Mass market |
| Pages: | 552 |
She shimmers, my city, she shimmers. She is said to be the most beautiful of all the cities of the Civilized Worlds, more beautiful even than Parpallaix or the cathedral cities of Vesper. To the west, pushing into the green sea like a huge, jewel-studded sleeve of city, the fragile obsidian cloisters and hospices of the Farsider's Quarter gleamed like black glass mirrors. Straight ahead as we skated, I saw the frothy churn of the Sound and their whitecaps of breakers crashing against the cliffs of North Beach and above the entire city, veined with purple and glazed with snow and ice, Waaskel and Attakel rose up like vast pyramids against the sky. Beneath the half-ring of extinct volcanoes (Urkel, I should mention, is the southernmost peak, and though less magnificent than the others, it has a conical symmetry that some find pleasing) the towers and spires of the Academy scattered the dazzling false winter light so that the whole of the Old City sparkled.That's less than half of that paragraph, and the entire book is written like that, even in the middle of conversations. Endless, constant words piled on words about absolutely everything, whether important or not, whether emotionally significant or not. And much of it isn't even description, but philosophical ponderings that are desperately trying to seem profound. Here's another bit:
Although I knew I had never seen her before, I felt as if I had known her all my life. I was instantly in love with her, not, of course, as one loves another human being, but as a wanderer might love a new ocean or a gorgeous snowy peak he has glimpsed for the first time. I was practically struck dumb by her calmness and her beauty, so I said the first stupid thing which came to mind. "Welcome to Neverness," I told her.Now, I should be fair: some people like this kind of description, or at least have more tolerance for it than I do. But that brings me to the second problem: there isn't a single truly likable character in this entire novel. Ringess, the person telling us this whole story, is a spoiled man-child, the sort of deeply immature and insecure person who attempts to compensate through bluster, impetuousness, and refusing to ever admit that he made a mistake or needed to learn something. He spends a good portion of the book, particularly the deeply bizarre and off-putting sections with the fake Neanderthals, attempting to act out some sort of stereotyped toxic masculinity and wallowing in negative emotions. Soli is an arrogant, abusive asshole from start to finish. Katherine, Ringess's love interest, is a seer who has had her eyes removed to see the future (I cannot express how disturbing I found Zindell's descriptions of this), has bizarre and weirdly sexualized reactions to the future she never explains, and leaves off the ends of all of her sentences, which might be be the most pointlessly irritating dialogue quirk I've seen in a novel. And Ringess's mother is a man-hating feminist from a separatist culture who turns into a master manipulator (I'm starting to see why Card liked this book). I at least really wanted to like Bardo, Ringess's closest friend, who has a sort of crude loyalty and unwillingness to get pulled too deep into the philosophical quicksand lurking underneath everything in this novel. Alas, Zindell insists on constantly describing Bardo's odious eating, belching, and sexual habits every time he's on the page, thus reducing him to the disgusting buffoon who gets drunk a lot and has irritating verbal ticks. About the only person I could stand by the end of the book was Justine, who at least seems vaguely sensible (and who leaves the person who abuses her), but she's too much of a non-entity to carry sustained interest. (There is potential here for a deeply scathing and vicious retelling of this story from Justine's point of view, focusing on the ways she was belittled, abused, and ignored, but I think Zindell was entirely unaware of why that would be so effective.) Oh, and there's lots of gore and horrific injury and lovingly-described torture, because of course there is. And that brings me back to the second half of that St. Louis Post-Dispatch review quote: "... really comes to life among the intrigues of Neverness." I would love to know what was hiding behind the ellipses in this pull quote, because this half-sentence is not wrong. Insofar as Neverness has any real appeal, it's in the intrigues of the city of Neverness and in the political structure that rules it. What this quote omits is that these intrigues start around page 317, more than halfway through the novel. That's about the point where faux-Wolfe starts mixing with late-career Frank Herbert and we get poet-assassins, some revelations about the leader of the Pilot culture, and some more concrete explanations of what this mess of a book is about. Unfortunately, you have to read through the huge and essentially meaningless Neanderthal scenes to get there, scenes that have essentially nothing to do with the interesting content of this book. (Everything that motivates them turns out to be completely irrelevant to the plot and useless for the characters.) The last 40% of the book is almost passable, and characters I cared about might have even made it enjoyable. Still, a couple of remaining problems detract heavily, chief among them the lack of connection of the great revelation of the story to, well, anything in the story. We learn at the very start of the novel that the stars of the Vild are mysteriously exploding, and much of the novel is driven by uncovering an explanation and solution. The characters do find an explanation, but not through any investigation. Ringess is simply told what is happening, in a wad of exposition, as a reward for something else entirely. It's weirdly disconnected from and irrelevant to everything else in the story. (There are some faint connections to the odd technological rules that the Pilot society lives under, but Zindell doesn't even draw attention to those.) The political intrigue in Neverness is similar: it appears out of nowhere more than halfway through the book, with no dramatic foundation for the motives of the person who has been keeping most of the secrets. And the final climax of the political machinations involves a bunch of mystical nonsense masquerading as science, and more of the Neanderthal bullshit that ruins the first half of the book. This is a thoroughly bad book: poorly plotted, poorly written, clotted and pretentious in style, and full of sociopaths and emotionally stunted children. I read the whole thing because I'm immensely stubborn and make poor life choices, but I was saying the eight deadly words ("I don't care what happens to these people") by a hundred pages in. Don't emulate my bad decisions. (Somehow, this novel was shortlisted for the Arthur C. Clarke award in 1990. What on earth could they possibly have been thinking?) Neverness is a stand-alone novel, but the ending sets up a subsequent trilogy that I have no intention of reading. Followed by The Broken God. Rating: 2 out of 10
The second pre-conference day was dedicated to books and beers, with a visit to an exquisite print studio, and a beer tasting session at one of the craft breweries in Canada. In addition we could grab a view into the Canadian lifestyle by visiting Pavneet s beautiful house in the countryside, as well as enjoying traditional style pastries from a bakery.
Although they wont win a prize for typography (besides one page of a coaster there that carried a nice pun), their beers are exquisite. We got five different beers to taste, plus extensive explanations on brewing methods and differences. Now I finally understand why most of the new craft breweries in Japan are making Ales: Ales don t need a long process and are ready for sale in rather short time, compared to e.g. lagers.)
Here is my monthly update covering a large part of what I have been doing in the free software world (previously):
Which gadgets does Bond bring to Switzerland?
On Her Majesty's Secret Service is one of the more memorable Bond movies for its spectacular setting in the Swiss Alps, the location now transformed into a mountain-top revolving restaurant visited by thousands of tourists every day with a comfortable cable car service and hiking trails with breathtaking views that never become boring.
Can you imagine Bond leaving behind his gun and his skis and visiting Switzerland with a smartphone instead? Eating a pizza with one hand while using the fingertips of the other to operate an app for making drone strikes on villains, swiping through Tinder for a new girl to replace the one who died (from boredom) in his previous "adventure" and letting his gelati melt while engrossed in a downhill ski or motorcycle game in all the glory of a 5.7" 24-bit colour display?
Of course its absurd. Would you want to live like that yourself? We see more and more of it in people who are supposedly in Switzerland on the trip of a lifetime. Would you tolerate it in a movie? The mobile phone industry has paid big money to have their technology appear on the silver screen but audience feedback shows people are frustrated with movies that plaster the contents of text messages across the screen every few minutes; hopefully Bond movies will continue to plaster bullets and blood across the screen instead.
Time for freedom
How would you live for a day or a weekend or an entire holiday without your mobile phone? There are many small frustrations you may experience but the biggest one and the indirect cause of many other problems you will experience may be the inability to tell the time.
Many people today have stopped wearing a watch, relying instead upon their mobile phone to tell the time.
Without either a phone or a watch, frustration is not far away.
If you feel apprehension just at the thought of leaving your phone at home, the lack of a watch may be a subconcious factor behind your hesitation.
Trying is better than reading
Many articles and blogs give opinions about how to buy a watch, how much to spend and what you can wear it with. Don't spend a lot of time reading any of it, if you don't know where to start, simply go down to the local high street or mall and try them. Start with the most glamorous and expensive models from Swiss manufacturers, as these are what everything else is compared to and then perhaps proceed to look more widely. While Swiss brands tend to sell through the stores, vendors on Amazon and eBay now distribute a range of watches from manufacturers in Japan, China and other locations, such as Orient and Invicta, at a fraction of the price of those in the stores. You still need to try a few first to identify your preferred style and case size though. Google can also turn up many options for different budgets.
| Series: | Steerswomen #2 |
| Publisher: | Del Rey |
| Copyright: | 1992, 2001 |
| Printing: | 2001 |
| ISBN: | 0-345-46105-3 |
| Format: | Trade paperback |
| Pages: | 389 |
| Series: | Steerswomen #1 |
| Publisher: | Del Rey |
| Copyright: | 1989, 2003 |
| Printing: | 2003 |
| ISBN: | 0-345-46105-3 |
| Format: | Trade paperback |
| Pages: | 264 |
Here s a photo journal of a walk I just completed around the Pot Lake loop of the Bluff Wilderness Hiking Trail. Hope you enjoy it!
I was dissatisfied with my initial post, so have reduced the size to improve load time, changed the gallery software and have rewritten many of the captions.
[slb_group]
Click this photo to start the slideshow.
And Greg Gregory of the Travel Ink Tattoo Studio from UK was there, with his shiny Airstream.
I also spent a while among the records in the Bear Family Records booth. They are a Germany based independent record label specialised in reissues of country and 50s rock'n'roll. Couldn't resist, and I bought a beautiful Sun Records' tshirt.
Just Rockin' and Rollin'. Aka: dance time
After that, it was time to dance. I missed the dance camp of the afternoon, but the DJ sets were fantastic, all 40s and 50s stuff, and I fell in love with Lindy Hop and Boogie Woogie, and well, obviously, Jive.
I could have spent hours watching the people dancing, and clumsily trying the most basic moves myself.
People
And the people, did I mention the people?
Or the amazingly dressed German couple I met in via Carducci.
And this couple too, was pretty cool.
The Prettiest Smile award goes to these lovely ladies!
Cars
Who knows me, can tell that I don't love cars.
Also, the black Cadillac had the terrible effect on me of putting "Santa Claus is Back in Town" in my head (or, more precisely, Elvis tomcatting his way through the song, singing "Got no sleigh with reindeer / No sack on my back / You're gonna see me comin' in a big black Cadillac").
Music!
Sadly, I missed Stray Cat's Slim Jim Phantom but I was just in time for Ben E. King.
And, well, more dancing: the dj sets on the three stages went on until 3 am.
Day 2
The next morning I took advantage of the early opening of Rocca Roveresca to visit it. The Rocca itself is beautiful and very well maintained, and hosts various exhibitions.
But the ones moving me were the pics from "Buddy Holly, The Day The Music Dies": a collection of photos taken by Bill Francis during the (sadly brief) career of Buddy Holly from the very beginnings to his death.
After that, it was time to come back to year 2014, but really I felt like I've walked for a while in another decade and planet. And the cool thing is that I could enjoy the great 40s and 50s music and dances (and clothes!) without the horrible stereotypes and cultural norms of the time period. A total win. :)
So, ehm, that's it. I'm a bit sad to be back, and to cheer myself up I'm
already planning to attend Wanda Jackson gig in Aarburg (CH) next month.
Zoe slept pretty well last night. She only woke up briefly at 4am because
Cowie had fallen out of bed and she couldn't find her.
Today was the last Playgroup of the term. Megan, her little sister and her Dad
came as well to check it out, which was nice, because Zoe then had someone to
actively play with in addition to me.
After Playgroup, we went to the adjacent Bulimba Memorial Park with Megan, and
then had some lunch at Grill'd. Megan's Dad wanted to do some work on their
house while Megan's little sister napped, so I offered to give Megan a play
date at our place.
The plan was to watch a movie and chill out. The girls picked
Ratatouille and I made a batch of popcorn for them. Unfortunately
Megan seemed to be less of a square eyes than Zoe, and she lost interest after
a bit, so we stopped watching the movie and moved out to the balcony to do some
craft.
Zoe had been wanting to make a crown for Mummy's boss for a while, so we made a
couple of crowns with the hot glue gun. I had bought this bag of mixed craft
"jewels" and it's probably the best single craft thing I've bought. Zoe
loves gluing them onto everything.
After that, Zoe pulled out the bag of coloured foil confetti. If the gems were
the best thing I've bought, this would have to be the worst. So far, all it's
done is leak in the drawer it's been stored in, and I've been avoiding using it
because it was going to be messy.
Today, Zoe wanted to glue it onto the outside of her cardboard box, so I
decided to give in and embrace the mess, and boy, did we make a mess.
It probably ended up being the longest bit of cooperative play the girls did.
They'd alternate between handing each other a fistful of confetti while I
applied globs of glue where directed. Probably about 10 percent of each handful
ended up stuck to the rocket, so the balcony looked like quite a mess by the
end of it all, but at least it was a dry mess, so I could just vacuum it all
up. I suspect I'll be encountering dregs for quite a while, because I doubt
it's stuck to the cardboard particularly well.
After that, the girls played indoors for a bit, and watched a bit more of the
movie, but Megan seemed to be scared of Anton Ego, so I think that was why
it wasn't holding her attention.
The other activity that the girls seemed to thoroughly enjoy was tearing around
the living room squealing while they took turns at throwing a grapefruit-sized
beach ball at me, and I threw it back at them.
Jason came back to pick up Megan, and I started dinner. Not that long after
Megan left, Sarah arrived to watch Zoe for me so I could go visit my cousin in
hospital. I had dinner on the table pretty much as soon as she walked in the
door, and headed out.
My yoga teacher was out sick this morning. I had grand plans of instead biking
to the pool and going for a swim, but when my alarm went off, and the weather
outside was grey and miserable, having an extra half an hour lie in seemed more
attractive. I think I made the right choice, because I felt like a million
bucks today.
I did all of the preparation to bake a batch of carrot and kale muffins before
Zoe arrived, and we baked a batch as soon as she arrived and had them out of
the oven in time to drive to Playgroup. I was expecting a larger crowd today on
account of the wet weather, but it turned out quite the opposite. That said,
Zoe still had a good time. There's really no other kids her age though, so it's
still winding up as a "play with Dad in a different environment".
My ABN came through yesterday, so after Playgroup we walked to the bank to give
it to them and sort out a business credit card. Zoe was super well behaved
while I did that, so we grabbed a fresh hot cross bun from Brumby's across the street afterwards.
I needed some more stamps, so I figured we could just walk down to the post
office down the other end of Oxford Street while we were there. That was slow
going, but Zoe was enjoying walking with her umbrella in the rain. We got to
the post office, and I discovered that postage is going up to 70 cents next
week, and they couldn't sell be 70 cent stamps yet, and buying 60 cent stamps
would be pointless after Monday, so I left empty handed. Zoe was eyeing off the
umbrellas they had for sale in the post office.
No sooner had we walked out of the post office and Zoe managed to walk all over
her umbrella and totally destroy it. If I hadn't seen it happen, I'd have said
she did it deliberately to get a new umbrella, but it really was an accident,
so we had to turn around and buy one of the umbrellas from the post office.
We made our way back down Oxford Street, and stopped in the boutique toy shop
there. Zoe was particularly enthralled by the musical jewelery boxes, and
really wanted one. I negotiated with her for it to be a birthday present, and
got it gift wrapped. She seemed fine with the idea of not being able to have it
until her birthday.
We then made our way back to the car and drove home for a rather late lunch.
I'm glad Zoe had the hot cross bun after the bank to keep her going, as she
didn't seem to mind the late lunch at all.
I made fritters with some left over corned beef, and then after we'd had lunch
I thought we might as well get out of the house.
Zoe's Kindergarten has a working bee on Saturday (I'm expecting it'll get
canceled due to the wet weather, though). Incidentally, I have no idea where
the "bee" in "working bee" comes from. Megan's Dad is Welsh, and I was lost for
words when it came to explaining what a working bee was. Anyway, I wanted to
get some gardening gloves, so we went to Bunnings. I managed to find some cute
little kids ones as well, so Zoe can help.
After Bunnings, we went to the pet shop to get some more cat litter. The pet
shop seems to be a great source of entertainment for Zoe. She absolutely loved
playing around with the hutches they had on display, and checking out all the
fish, and the aquarium accessories.
We eventually made it out of the pet shop, and we went around to the adjacent
shopping mall in search of some craft supplies at some of the cheap shops
there. It turns out the cheap shops are also a great source of entertainment.
By the time we were done there, it was time to get home so Sarah could pick Zoe
up, so we headed home. There was enough time for Zoe to watch a little bit of
TV before Sarah arrived.
So despite the weather, and no real plan for the day, we managed to completely
fill the day, and Zoe had a great time. She seemed no worse for wear for
powering through without a nap. There was a few small tantrums at Bunnings, but
that mostly revolved around her tipping over her miniature shopping trolley.
This is a little tutorial on how to create swirl charms for keyrings from plastic bottles."What is PET bottle?What you will need
A domesticated bottle kept for companionship or amusement."
Gulnur Ozdaglar
First of all, you'll need to cut out the swirl from the bottle.
As the edges of the cut are a bit raw, I decided to lightly melt them with the lighter: be careful as
if the plastic melt too much it will ruin our nice swirl. Also, remember to
open a window while doing it: it's not a lot of melting but better safe than sorry.
Step 3
With a pin or a needle punch a hole in the upper part of the swirl, where the jumpring will be set to connect the
swirl with the keychain ring.
Now you need to decide which type of swirl do: for the painted one, follow step 4a; for the
transparent one with the red embroidery, follow step 4b.
Step 4a
Paint your swirl red, or of your favourite colour (I experimented a bit here, having lots of
nail polish, and made also a purple one).
Step 4b
With a pin or a needle, punch a series of holes, in the middle of the swirl, following the swirl shape.
Add the jumpring, if you have a ready made one.
Connect the jumpring and the metal keyring.
Congratulations: you now have a swirl shaped keycharm!
Ordinarily, I m a big believer that it is important to keep up to date with what every piece of software which competes with yours is doing, to remain educated on the latest concepts. Sometimes, there are concepts that get added which are definitely worth ripping off. We ve ripped off plenty of the better design choices from Windows or Mac OS, over the years, for use in the Free Desktop.
So, what about Windows 8, the hip new OS on everyone s lips?
Well, here s the thing I ve been using it on and off for a few months now for running legacy apps, and I can t for the life of me find anything worth stealing.
Let s take the key change Windows 8 has apps built with a new design paradigm which definitely isn t called Metro. Metro apps don t really have windows in the traditional sense they re more modeled on full-screen apps from smartphones or tablets than on Windows 1.0 -> 7. Which is fine, really, if you re running Windows 8 on a tablet or touchscreen device. But what if you re not? What about the normal PC user?
As Microsoft themselves ask:
The answer to that is, well, you sorta don t.
Metro apps can exist in three states fullscreen, almost fullscreen, or vertical stripe. You re allowed to have two apps at most at the same time one mostly full screen, and one vertical stripe. So what happens if you try to *use* that? Let s take a fairly common thing I do watch a video and play Minesweeper. In this example, the video player is the current replacement for Windows Media Player, and ships by default. The Minesweeper game isn t installed by default, but is the only Minesweeper game in the Windows 8 app store which is gratis and by Microsoft Game Studios.
Here s option A:
And for contrast, here s option B:
Which of these does a better job of letting me play Minesweeper and watch a video at the same time?
Oh, here s option C, dumping Microsoft s own software, and using a third-party video player and third party Minesweeper implementation:
It s magical almost as if picking my own window sizes makes the experience better.
So, as you can see above, the old OS is still hiding there, in the form of a Windows 8 app called Desktop . Oh, sorry, didn t I say? Metro apps, and non-Metro apps, are segregated. You can run both (the Desktop app can also be almost-fullscreen or a vertical strip), but they get their own lists of apps when multitasking. Compare the list on the left with the list at the bottom:
And it s even more fun for apps like Internet Explorer, which can be started in both modes (and you often need both modes). Oh, and notice how the Ribbon interface from Office 2007 has invaded Explorer, filling the view with large buttons to do things you never want to do under normal circumstances.
So, that s a short primer on why Windows 8 is terrible.
Is there really nothing here worth stealing? Actually, yes, there is! After much research, I have discovered Windows 8 s shining jewel:
The new Task Manager is lovely. I want it on my Linux systems. But that s it.
So, in 2002 I attended my first SPUG meeting. Also in attendance were Randal, Tim, J.P. (I think) and quite a few of the other usual suspects. The guest speaker was the inestimable Dr. Damian Conway. This night might be the single most important event in my career as a builder of the internets. I was already set in my ways regarding my operating system, and I knew enough perl to get by, but I wasn t in touch with any users groups until then.
The crown jewel of the night was Damian s recital in full of his poem, I Am The Very Model of A Poster To CLPM (sung to the tune of that song from Pirates of Penzance). I must admit that I d never heard of CLPM and that it all sounded like gobble-de-gook at the time. If I recall correctly, this happened just after he took about 10 minutes to teach us all to pronounce the name of then-Pumpking, Jarkko Hietaniemi. This was the same year that the following video was made
<iframe allowfullscreen="allowfullscreen" frameborder="0" height="315" src="http://www.youtube.com/embed/XkU_GQVOCqo" width="420"></iframe>
With that in mind, there s one more thing I need to share with you before disclosing the lyrics. This piece was developed during a time when it was common for members of the perl community to flame one another on news feeds, take their anger and frustration out on newbies and generally be the bane of most of the internet. I like to think that it helped to shape me into a kinder, gentler perl programmer, debian user and general netizen over the years. I would also like to think that it will do the same for future members of our communities.
And so, without further ado, Damian Conway s
Next.
