Russell Coker: Internode NBN500
| Connection | Receive Mbit/s | Send Mbit/s |
|---|---|---|
| 100baseT | 96 | 47 |
| Gigabit | 535 | 46 |
| 2.4GHz Wifi | 172 | 45 |
| Wifi5 | 514 | 41 |
| Wifi5 Over Mesh | 200 | 45 |
| Connection | Receive Mbit/s | Send Mbit/s |
|---|---|---|
| 100baseT | 96 | 47 |
| Gigabit | 535 | 46 |
| 2.4GHz Wifi | 172 | 45 |
| Wifi5 | 514 | 41 |
| Wifi5 Over Mesh | 200 | 45 |
| Publisher: | Princeton University Press |
| Copyright: | 2006, 2008 |
| Printing: | 2008 |
| ISBN: | 0-691-13640-8 |
| Format: | Trade paperback |
| Pages: | 278 |
How are you meant to write about an unfinished emancipation? Bodies of Light is a 2014 book by Glasgow-born Sarah Moss on the stirrings of women's suffrage in an arty clique in nineteenth-century England. Set in the intellectually smoggy cities of Manchester and London, we follow the studious and intelligent Alethia 'Ally' Moberly, who is struggling to gain the acceptance of herself, her mother and the General Medical Council.
'Alethia' may be the Greek goddess of truth, but our Ally is really searching for wisdom. Her strengths are her patience and bookish learning, and she acquires Latin as soon as she learns male doctors will use it to keep women away from the operating theatre. In fact, Ally's acquisition of language becomes a recurring leitmotif: replaying a suggestive dream involving a love interest, for instance, Ally thinks of 'dark, tumbling dreams for which she has a perfectly adequate vocabulary'. There are very few moments of sensuality in the book, and pairing it with Ally's understated wit achieves a wonderful effect.
The amount we learn about a character is adapted for effect as well. There are few psychological insights about Ally's sister, for example, and she thus becomes a fey, mysterious and almost Pre-Raphaelite figure below the surface of a lake to match the artistic movement being portrayed. By contrast, we get almost the complete origin story of Ally's mother, Elizabeth, who also constitutes of those rare birds in literature: an entirely plausible Christian religious zealot. Nothing Ally does is ever enough for her, but unlike most modern portrayals of this dynamic, neither of them are aware of what is going, and it is conveyed in a way that is chillingly... benevolent. This was brought home in the annual 'birthday letters' that Elizabeth writes to her daughter:
Last year's letter said that Ally was nervous, emotional and easily swayed, and that she should not allow her behaviour to be guided by feeling but remember always to assert her reason. Mamma would help her with early hours, plain food and plenty of exercise. Ally looks at the letter, plump in its cream envelope. She hopes Mamma wrote it before scolding her yesterday.The book makes the implicit argument that it is a far more robust argument against pervasive oppression to portray a character in, say, 'a comfortable house, a kind husband and a healthy child', yet they are nonetheless still deeply miserable, for reasons they can't quite put their finger on. And when we see Elizabeth perpetuating some generational trauma with her own children, it is telling that is pattern is not short-circuited by an improvement in their material conditions. Rather, it is arrested only by a kind of political consciousness in Ally's case, the education in a school. In fact, if there is a real hero in Bodies of Light, it is the very concept of female education. There's genuine shading to the book's ideological villains, despite finding their apotheosis in the jibes about 'plump Tories'. These remarks first stuck out to me as cheap thrills by the author; easy and inexpensive potshots that are unbecoming of the pages around them. But they soon prove themselves to be moments of much-needed humour. Indeed, when passages like this are read in their proper context, the proclamations made by sundry Victorian worthies start to serve as deadpan satire:
We have much evidence that the great majority of your male colleagues regard you as an aberration against nature, a disgusting, unsexed creature and a danger to the public.Funny as these remarks might be, however, these moments have a subtler and more profound purpose as well. Historical biography always has the risk of allowing readers to believe that the 'issue' has already been solved hence, perhaps, the enduring appeal of science fiction. But Moss providing these snippets from newspapers 150 years ago should make a clear connection to a near-identical moral panic today. On the other hand, setting your morality tale in the past has the advantage that you can show that progress is possible. And it can also demonstrate how that progress might come about as well. This book makes the argument for collective action and generally repudiates individualisation through ever-fallible martyrs. Ally always needs 'allies' not only does she rarely work alone, but she is helped in some way by almost everyone around her. This even includes her rather problematic mother, forestalling any simplistic proportioning of blame. (It might be ironic that Bodies of Light came out in 2014, the very same year that Sophia Amoruso popularised the term 'girl boss'.) Early on, Ally's schoolteacher is coded as the primary positive influence on her, but Ally's aunt later inherits this decisive role, continuing Ally's education on cultural issues and what appears to be the Victorian version of 'self-care'. Both the aunt and the schoolteacher are, of course, surrogate mother figures. After Ally arrives in the cut-throat capital, you often get the impression you are being shown discussions where each of the characters embodies a different school of thought within first-wave feminism. This can often be a fairly tedious device in fiction, the sort of thing you would find in a Sally Rooney novel, Pilgrim's Progress or some other ponderously polemical tract. Yet when Ally appears to 'win' an argument, it is only in the sense that the narrator continues to follow her, implicitly and lightly endorsing her point. Perhaps if I knew my history better, I might be able to associate names with the book's positions, but perhaps it is better (at least for the fiction-reading experience...) that I don't, as the baggage of real-world personalities can often get in the way. I'm reminded here of Regina King's One Night in Miami... (2020), where caricatures of Malcolm X, Muhammad Ali, Jim Brown and Sam Cooke awkwardly replay various arguments within an analogous emancipatory struggle. Yet none of the above will be the first thing a reader will notice. Each chapter begins with a description of an imaginary painting, providing a title and a date alongside a brief critical exegesis. The artworks serve a different purpose in each chapter: a puzzle to be unlocked, a fear to be confirmed, an unsolved enigma. The inclusion of (artificial) provenances is interesting as well, not simply because they add colour and detail to the chapter to come, but because their very inclusion feels reflective of how we see art today.
Orphelia (1852) by Sir John Everett Millais.
I won't reveal precisely how many books I read in 2020, but it was definitely an improvement on 74 in 2019, 53 in 2018 and 50 in 2017. But not only did I read more in a quantitative sense, the quality seemed higher as well. There were certainly fewer disappointments: given its cultural resonance, I was nonplussed by Nick Hornby's Fever Pitch and whilst Ian Fleming's The Man with the Golden Gun was a little thin (again, given the obvious influence of the Bond franchise) the booked lacked 'thinness' in a way that made it interesting to critique. The weakest novel I read this year was probably J. M. Berger's Optimal, but even this hybrid of Ready Player One late-period Black Mirror wasn't that cringeworthy, all things considered. Alas, graphic novels continue to not quite be my thing, I'm afraid.
I perhaps experienced more disappointments in the non-fiction section. Paul Bloom's Against Empathy was frustrating, particularly in that it expended unnecessary energy battling its misleading title and accepted terminology, and it could so easily have been an 20-minute video essay instead). (Elsewhere in the social sciences, David and Goliath will likely be the last Malcolm Gladwell book I voluntarily read.) After so many positive citations, I was also more than a little underwhelmed by Shoshana Zuboff's The Age of Surveillance Capitalism, and after Ryan Holiday's many engaging reboots of Stoic philosophy, his Conspiracy (on Peter Thiel and Hulk Hogan taking on Gawker) was slightly wide of the mark for me.
Anyway, here follows a selection of my favourites from 2020, in no particular order:
The disease stiffened and carried off three or four patients who were expected to recover. These were the unfortunates of the plague, those whom it killed when hope was highIt somehow captured the nostalgic yearning for high-definition videos of cities and public transport; one character even visits the completely deserted railway station in Oman simply to read the timetables on the wall.
Small, podgy, and at best middle-aged, Smiley was by appearance one of London's meek who do not inherit the earth. His legs were short, his gait anything but agile, his dress costly, ill-fitting, and extremely wet.Almost a direct rebuttal to Ian Fleming's 007, Tinker, Tailor has broken-down cars, bad clothes, women with their own internal and external lives (!), pathetically primitive gadgets, and (contra Mad Men) hangovers that significantly longer than ten minutes. In fact, the main aspect that the mostly excellent 2011 film adaption doesn't really capture is the smoggy and run-down nature of 1970s London this is not your proto-Cool Britannia of Austin Powers or GTA:1969, the city is truly 'gritty' in the sense there is a thin film of dirt and grime on every surface imaginable. Another angle that the film cannot capture well is just how purposefully the novel does not mention the United States. Despite the US obviously being the dominant power, the British vacillate between pretending it doesn't exist or implying its irrelevance to the matter at hand. This is no mistake on Le Carr 's part, as careful readers are rewarded by finding this denial of US hegemony in metaphor throughout --pace Ian Fleming, there is no obvious Felix Leiter to loudly throw money at the problem or a Sheriff Pepper to serve as cartoon racist for the Brits to feel superior about. By contrast, I recall that a clever allusion to "dusty teabags" is subtly mirrored a few paragraphs later with a reference to the installation of a coffee machine in the office, likely symbolic of the omnipresent and unavoidable influence of America. (The officer class convince themselves that coffee is a European import.) Indeed, Le Carr communicates a feeling of being surrounded on all sides by the peeling wallpaper of Empire. Oftentimes, the writing style matches the graceless and inelegance of the world it depicts. The sentences are dense and you find your brain performing a fair amount of mid-flight sentence reconstruction, reparsing clauses, commas and conjunctions to interpret Le Carr 's intended meaning. In fact, in his eulogy-cum-analysis of Le Carr 's writing style, William Boyd, himself a ventrioquilist of Ian Fleming, named this intentional technique 'staccato'. Like the musical term, I suspect the effect of this literary staccato is as much about the impact it makes on a sentence as the imperceptible space it generates after it. Lastly, the large cast in this sprawling novel is completely believable, all the way from the Russian spymaster Karla to minor schoolboy Roach the latter possibly a stand-in for Le Carr himself. I got through the 500-odd pages in just a few days, somehow managing to hold the almost-absurdly complicated plot in my head. This is one of those classic books of the genre that made me wonder why I had not got around to it before.
Perhaps his life might have veered elsewhere if the US government had opened the country to colored advancement like they opened the army. But it was one thing to allow someone to kill for you and another to let him live next door.Sardonic aper us of this kind are pretty relentless throughout the book, but it never tips its hand too far into on nihilism, especially when some of the visual metaphors are often first-rate: "An American flag sighed on a pole" is one I can easily recall from memory. In general though, The Nickel Boys is not only more world-weary in tenor than his previous novel, the United States it describes seems almost too beaten down to have the energy conjure up the Swiftian magical realism that prevented The Underground Railroad from being overly lachrymose. Indeed, even we Whitehead transports us a present-day New York City, we can't indulge in another kind of fantasy, the one where America has solved its problems:
The Daily News review described the [Manhattan restaurant] as nouveau Southern, "down-home plates with a twist." What was the twist that it was soul food made by white people?It might be overly reductionist to connect Whitehead's tonal downshift with the racial justice movements of the past few years, but whatever the reason, we've ended up with a hard-hitting, crushing and frankly excellent book.
"Earlier tonight I gave some thought to stealing a kiss from you, though you are very young, and sick and unattractive to boot, but now I am of a mind to give you five or six good licks with my belt." "One would be as unpleasant as the other."Perhaps this should be unsurprising. Maddie, a fourteen-year-old girl from Yell County, Arkansas, can barely fire her father's heavy pistol, so she can only has words to wield as her weapon. Anyway, it's not just me who treasures this book. In her encomium that presages most modern editions, Donna Tartt of The Secret History fame traces the novels origins through Huckleberry Finn, praising its elegance and economy: "The plot of True Grit is uncomplicated and as pure in its way as one of the Canterbury Tales". I've read any Chaucer, but I am inclined to agree. Tartt also recalls that True Grit vanished almost entirely from the public eye after the release of John Wayne's flimsy cinematic vehicle in 1969 this earlier film was, Tartt believes, "good enough, but doesn't do the book justice". As it happens, reading a book with its big screen adaptation as a chaser has been a minor theme of my 2020, including P. D. James' The Children of Men, Kazuo Ishiguro's Never Let Me Go, Patricia Highsmith's Strangers on a Train, James Ellroy's The Black Dahlia, John Green's The Fault in Our Stars, John le Carr 's Tinker, Tailor Soldier, Spy and even a staged production of Charles Dicken's A Christmas Carol streamed from The Old Vic. For an autodidact with no academic background in literature or cinema, I've been finding this an effective and enjoyable means of getting closer to these fine books and films it is precisely where they deviate (or perhaps where they are deficient) that offers a means by which one can see how they were constructed. I've also found that adaptations can also tell you a lot about the culture in which they were made: take the 'straightwashing' in the film version of Strangers on a Train (1951) compared to the original novel, for example. It is certainly true that adaptions rarely (as Tartt put it) "do the book justice", but she might be also right to alight on a legal metaphor, for as the saying goes, to judge a movie in comparison to the book is to do both a disservice.
We're accustomed to worrying about AI systems being built that will either "go rogue" and attack us, or succeed us in a bizarre evolution of, um, evolution what we didn't reckon on is the sheer inscrutability of these manufactured minds. And minds is not a misnomer. How else should we think about the neural network Google has built so its translator can model the interrelation of all words in all languages, in a kind of three-dimensional "semantic space"?New Dark Age also turns its attention to the weird, algorithmically-derived products offered for sale on Amazon as well as the disturbing and abusive videos that are automatically uploaded by bots to YouTube. It should, by rights, be a mess of disparate ideas and concerns, but Bridle has a flair for introducing topics which reveals he comes to computer science from another discipline altogether; indeed, on a four-part series he made for Radio 4, he's primarily referred to as "an artist". Whilst New Dark Age has rather abstract section topics, Adam Greenfield's Radical Technologies is a rather different book altogether. Each chapter dissects one of the so-called 'radical' technologies that condition the choices available to us, asking how do they work, what challenges do they present to us and who ultimately benefits from their adoption. Greenfield takes his scalpel to smartphones, machine learning, cryptocurrencies, artificial intelligence, etc., and I don't think it would be unfair to say that starts and ends with a cynical point of view. He is no reactionary Luddite, though, and this is both informed and extremely well-explained, and it also lacks the lazy, affected and Private Eye-like cynicism of, say, Attack of the 50 Foot Blockchain. The books aren't a natural pair, for Bridle's writing contains quite a bit of air in places, ironically mimics the very 'clouds' he inveighs against. Greenfield's book, by contrast, as little air and much lower pH value. Still, it was more than refreshing to read two technology books that do not limit themselves to platitudinal booleans, be those dangerously naive (e.g. Kevin Kelly's The Inevitable) or relentlessly nihilistic (Shoshana Zuboff's The Age of Surveillance Capitalism). Sure, they are both anti-technology screeds, but they tend to make arguments about systems of power rather than specific companies and avoid being too anti-'Big Tech' through a narrower, Silicon Valley obsessed lens for that (dipping into some other 2020 reading of mine) I might suggest Wendy Liu's Abolish Silicon Valley or Scott Galloway's The Four. Still, both books are superlatively written. In fact, Adam Greenfield has some of the best non-fiction writing around, both in terms of how he can explain complicated concepts (particularly the smart contract mechanism of the Ethereum cryptocurrency) as well as in the extremely finely-crafted sentences I often felt that the writing style almost had no need to be that poetic, and I particularly enjoyed his fictional scenarios at the end of the book.
A better proxy for your life isn't your first home, but your last. Where you draw your last breath is more meaningful, as it's a reflection of your success and, more important, the number of people who care about your well-being. Your first house signals the meaningful your future and possibility. Your last home signals the profound the people who love you. Where you die, and who is around you at the end, is a strong signal of your success or failure in life.Nir Eyal's Indistractable, however, is a totally different kind of 'self-help' book. The important background story is that Eyal was the author of the widely-read Hooked which turned into a secular Bible of so-called 'addictive design'. (If you've ever been cornered by a techbro wielding a Wikipedia-thin knowledge of B. F. Skinner's behaviourist psychology and how it can get you to click 'Like' more often, it ultimately came from Hooked.) However, Eyal's latest effort is actually an extended mea culpa for his previous sin and he offers both high and low-level palliative advice on how to avoid falling for the tricks he so studiously espoused before. I suppose we should be thankful to capitalism for selling both cause and cure. Speaking of markets, there appears to be a growing appetite for books in this 'anti-distraction' category, and whilst I cannot claim to have done an exhausting study of this nascent field, Indistractable argues its points well without relying on accurate-but-dry "studies show..." or, worse, Gladwellian gotchas. My main criticism, however, would be that Eyal doesn't acknowledge the limits of a self-help approach to this problem; it seems that many of the issues he outlines are an inescapable part of the alienation in modern Western society, and the only way one can really avoid distraction is to move up the income ladder or move out to a 500-acre ranch.
| Publisher: | Redhook Books |
| Copyright: | October 2020 |
| ISBN: | 0-316-42202-9 |
| Format: | Kindle |
| Pages: | 515 |
Once upon a time there were three sisters. They were born in a forgotten kingdom that smelled of honeysuckle and mud, where the Big Sandy ran wide and the sycamores shone white as knuckle-bones on the banks. The sisters had no mother and a no-good father, but they had each other; it might have been enough. But the sisters were banished from their kingdom, broken and scattered.The Once and Future Witches opens with Juniper, the youngest, arriving in the city of New Salem. The year is 1893, but not in our world, not quite; Juniper has witch-ways in her pocket and a few words of power. That's lucky for her because the wanted posters arrived before she did. Unbeknownst to her or to each other, her sisters, Agnes and Bella, are already in New Salem. Agnes works in a cotton mill after having her heart broken one too many times; the mill is safer because you can't love a cotton mill. Bella is a junior librarian, meek and nervous and uncertain but still fascinated by witch-tales and magic. It's Bella who casts the spell, partly by accident, partly out of wild hope, but it was Juniper arriving in the city who provided the final component that made it almost work. Not quite, not completely, but briefly the lost tower of Avalon appears in St. George's Square. And, more importantly, the three sisters are reunited. The world of the Eastwood sisters has magic, but the people in charge of that world aren't happy about it. Magic is a female thing, contrary to science and, more importantly, God. History has followed a similar course to our world in part because magic has been ruthlessly suppressed. Inquisitors are a recent memory and the cemetery has a witch-yard, where witches are buried unnamed and their ashes sown with salt. The city of New Salem is called New Salem because Old Salem, that stronghold of witchcraft, was burned to the ground and left abandoned, fit only for tourists to gawk at the supposedly haunted ruins. The women's suffrage movement is very careful to separate itself from any hint of witchcraft or scandal, making its appeals solely within the acceptable bounds of the church. Juniper is the one who starts to up-end all of that in New Salem. Juniper was never good at doing what she was told. This is an angry book that feels like something out of another era, closer in tone to a Sheri S. Tepper or Joanna Russ novel than the way feminism is handled in recent work. Some of that is the era of the setting, before women even had the right to vote. But primarily it's because Harrow, like those earlier works, is entirely uninterested in making excuses or apologies for male behavior. She takes an already-heated societal conflict and gives the underdogs magic, which turns it into a war. There is likely a better direct analogy from the suffrage movement, but the comparison that came to my mind was if Martin Luther King, Jr. proved ineffective or had not existed, and instead Malcolm X or the Black Panthers became the face of the Civil Rights movement. It's also an emotionally exhausting book. The protagonists are hurt and lost and shattered. Their moments of victory are viciously destroyed. There is torture and a lot of despair. It works thematically; all the external solutions and mythical saviors fail, but in the process the sisters build their own strength and their own community and rescue themselves. But it's hard reading at times if you're emotionally invested in the characters (and I was very invested). Harrow does try to balance the losses with triumphs and that becomes more effective and easier to read in the back half of the book, but I struggled with the grimness at the start. One particular problem for me was that the sisters start the book suspicious and distrustful of each other because of lies and misunderstandings. This is obvious to the reader, but they don't work through it until halfway through the book. I can't argue with this as a piece of characterization it made sense to me that they would have reacted to their past the way that they did. But it was still immensely frustrating to read, since in the meantime awful things were happening and I wanted them to band together to fight. They also worry over the moral implications of the fate of their father, whereas I thought the only problem was that the man couldn't die more than once. There too, it makes sense given the moral framework the sisters were coerced into, but it is not my moral framework and it was infuriating to see them stay trapped in it for so long. The other thing that I found troubling thematically is that Harrow personalizes evil. I thought the more interesting moral challenge posed in this book is a society that systematically abuses women and suppresses their power, but Harrow gradually supplants that systemic conflict with a villain who has an identity and a backstory. It provides a more straightforward and satisfying climax, and she does avoid the trap of letting triumph over one character solve all the broader social problems, but it still felt too easy. Worse, the motives of the villain turn out to be at right angles to the structure of the social oppression. It's just a tool he's using, and while that's also believable, it means the transfer of the narrative conflict from the societal to the personal feels like a shying away from a sharper political point. Harrow lets the inhabitants of New Salem off too easily by giving them the excuse of being manipulated by an evil mastermind. What I thought Harrow did handle well was race, and it feels rare to be able to say this about a book written by and about white women. There are black women in New Salem as well, and they have their own ways and their own fight. They are suspicious of the Eastwood sisters because they're worried white women will stir up trouble and then run away and leave the consequences to fall on black women... and they're right. An alliance only forms once the white women show willingness to stay for the hard parts. Black women are essential to the eventual success of the protagonists, but the opposite is not necessarily true; they have their own networks, power, and protections, and would have survived no matter what the Eastwoods did. The book is the Eastwoods' story, so it's mostly concerned with white society, but I thought Harrow avoided both making black women too magical or making white women too central. They instead operate in parallel worlds that can form the occasional alliance of mutual understanding. It helps that Cleopatra Quinn is one of the best characters of the book. This was hard, emotional reading. It's the sort of book where everything has a price, even the ending. But I'm very glad I read it. Each of the three sisters gets their own, very different character arc, and all three of those arcs are wonderful. Even Agnes, who was the hardest character for me to like at the start of the book and who I think has the trickiest story to tell, becomes so much stronger and more vivid by the end of the book. Sometimes the descriptions are trying a bit too hard and sometimes the writing is not quite up to the intended goal, but some of the descriptions are beautiful and memorable, and Harrow's way of weaving the mythic and the personal together worked for me. This is a more ambitious book than The Ten Thousand Doors of January, and while I think the ambition exceeded Harrow's grasp in a few places and she took a few thematic short-cuts, most of it works. The characters felt like living and changing people, which is not easy given how heavily the story structure leans on maiden, mother, and crone archetypes. It's an uncompromising and furious book that turns the anger of 1970s feminist SF onto themes that are very relevant in 2021. You will have to brace yourself for heartbreak and loss, but I think it's fantasy worth reading. Recommended. Rating: 8 out of 10

Summer heat
(Note, those memory sizes are RSS, minus shared )
It also was taking an unexpected long time, but then, I hadn t cloned a large repository like mozilla-central from scratch in a while, so I wasn t sure if it was just related to its recent growth in size or otherwise. So I collected data on 0.4.0 as well.
Less time spent, less memory usage ok. There s definitely something wrong on master. But wait a minute, that slope from ~2GB to ~4GB on the git-remote-hg process doesn t actually make any kind of sense. I mean, I d understand it if it were starting and finishing with the Import manifest phase, but it starts in the middle of it, and ends long before it finishes. WTH?
First things first, since RSS can be a variety of things, I checked /proc/$pid/smaps and confirmed that most of it was, indeed, the heap.
That s the point where you reach for Google, type something like python memory profile and find various tools. One from the results that I remembered having used in the past is guppy s heapy.
Armed with pdb, I broke execution in the middle of the slope, and tried to get memory stats with heapy. SIGSEGV. Ouch.
Let s try something else. I reached out to objgraph and pympler. SIGSEGV. Ouch again.
Tried working around the crashes for a while (too long while, retrospectively, hindsight is 20/20), and was somehow successful at avoiding them by peaking at a smaller set of objects. But whatever I did, despite being attached to a process that had 2.6GB RSS, I wasn t able to find more than 1.3GB of data. This wasn t adding up.
It surely didn t help that getting to that point took close to an hour each time. Retrospectively, I wish I had investigated using something like Checkpoint/Restore in Userspace.
Anyways, after a while, I decided that I really wanted to try to see the whole picture, not smaller peaks here and there that might be missing something. So I resolved myself to look at the SIGSEGV I was getting when using pympler, collecting a core dump when it happened.
Guess what? The Debian python-dbg package does not contain the debug symbols for the python package. The core dump was useless.
Since I was expecting I d have to fix something in python, I just downloaded its source and built it. Ran the command again, waited, and finally got a backtrace. First Google hit for the crashing function? The exact (unfixed) crash reported on the python bug tracker. No patch.
Crashing code is doing:
((f)->f_builtins != (f)->f_tstate->interp->builtins)
And (f)->f_tstate is NULL. Classic NULL deref.
Added a guard (assessing it wouldn t break anything). Ran the command again. Waited. Again. SIGSEGV.
Facedesk. Another crash on the same line. Did I really use the patched python? Yes. But this time (f)->f_tstate->interp is NULL. Sigh.
Same player, shoot again.
Finally, no crash but still stuck on only 1.3GB accounted for. Ok, I know not all python memory profiling tools are entirely reliable, let s try heapy again. SIGSEGV. Sigh. No debug info on the heapy module, where the crash happens. Sigh. Rebuild the module with debug info, try again. The backtrace looks like heapy is recursing a lot. Look at %rsp, compare with the address space from /proc/$pid/maps. Confirmed. A stack overflow. Let s do ugly things and increase the stack size in brutal ways.
Woohoo! Now heapy tells me there s even less memory used than the 1.3GB I found so far. Like, half less. Yeah, right.
I m not clear on how I got there, but that s when I found gdb-heap, a tool from Red Hat s David Malcolm, and the associated talk Dude, where s my RAM? A deep dive into how Python uses memory (slides).
With a gdb attached, I would finally be able to rip python s guts out and find where all the memory went. Or so I thought. The gdb-heap tool only found about 600MB. About as much as heapy did, for that matter, but it could be coincidental. Oh. Kay.
I don t remember exactly what went through my mind then, but, since I was attached to a running process with gdb, I typed the following on the gdb prompt:
gdb> call malloc_stats()
And that s when the truth was finally unvealed: the memory allocator was just acting up the whole time. The ouput was something like:
Arena 0:
system bytes = some number above (but close to) 2GB
in use bytes = some number above (but close to) 600MB
Yes, the glibc allocator was just telling it had allocated 600MB of memory, but was holding onto 2GB. I must have found a really bad allocation pattern that causes massive fragmentation.
One thing that David Malcolm s talk taught me, though, is that python uses its own allocator for small sizes, so the glibc allocator doesn t know about them. And, roughly, adding the difference between RSS and what glibc said it was holding to to the use bytes it reported somehow matches the 1.3GB I had found so far.
So it was time to see how those things evolved in time, during the entire clone process. I grabbed some new data, tracking the evolution of system bytes and in use bytes .
There are two things of note on this data:
malloc() instead of mmap() for its arenas.
Aha! I thought. That definitely looks much better. Less gap between what glibc says it requested from the system and the RSS size. And, more importantly, no runaway increase of memory usage in the middle of nowhere.
I was preparing myself to write a post about how mixing allocators could have unintended consequences. As a comparison point, I went ahead and ran another test, with the python allocator entirely disabled, this time.
Heh. It turns out glibc was acting up all alone. So much for my (plausible) theory. (I still think mixing allocators can have unintended consequences.)
(Note, however, that the reason why the python allocator exists is valid: without it, the overall clone took almost 10 more minutes)
And since I had been getting all this data with 0.4.0, I gathered new data without the python allocator with the master branch.
This paints a rather different picture than the original data on that branch, with much less memory use regression than one would think. In fact, there isn t much difference, except for the spike at the end, which got worse, and some of the noise during the Import manifests phase that got bigger, implying larger amounts of temporary memory used. The latter may contribute to the allocation patterns that throw glibc s memory allocator off.
It turns out tracking memory usage in python 2.7 is rather painful, and not all the tools paint a complete picture of it. I hear python 3.x is somewhat better in that regard, and I hope it s true, but at the moment, I m stuck with 2.7. The most reliable tool I ve used here, it turns out, is pympler. Or rebuilding the python interpreter without its allocator, and asking the system allocator what is allocated.
With all this data, I now have some defined problems to tackle, some easy (the spike at the end of the clone), and some less easy (working around glibc allocator s behavior). I have a few hunches as to what kind of allocations are causing the runaway increase of RSS. Coincidentally, I m half-way through a refactor of the code dealing with manifests, and it should help dealing with the issue.
But that will be the subject of a subsequent post.
Here is my monthly update covering what I have been doing in the free software world (previous month):
diffoscope is our "diff on steroids" that will not only recursively unpack archives but will transform binary formats into human-readable forms in order to compare them.
disorderfs is our FUSE filesystem that deliberately introduces nondeterminism into the results of system calls such as readdir(3).
strip-nondeterminism is our tool to remove specific information from a completed build.
I attended the Debian Bug Squashing Party (BSP) in Salzburg, Austria where 70+ "release-critical" bugs were fixed. Sincere thanks to Bernd Zeimetz (bzed) for organising and Conova for sponsoring/hosting. The event was covered by the Salzburg Cityguide.
I attended NixOS user group in London, England.
Niels Thykier granted me commit access to the Lintian Git repository and I added myself to the debian/copyright there.
Filed an ITP for the roughtime secure time synchronisation client and server. This is blocked on packaging the Bazel build system. (#838416)
Preparation for the release of TeX Live 2016 have started some time ago with the freeze of updates in TeX Live 2015. Yesterday we announced the official start of the pretest period. That means that we invite people to test the new release and help fixing bugs. At the same time I have uploaded the first set of packages of TeX Live 2016 for Debian to the experimental suite.
Concerning the binaries we do expect a few further changes, but hopefully nothing drastic. The most invasive change on the tlmgr side is that cryptographic signatures are now verified to guarantee authenticity of the packages downloaded, but this is rather irrelevant for Debian users (though I will look into how that works in user mode).
Other than that, many packages have been updated or added since the last Debian packages, here is the unified list:
acro, animate, appendixnumberbeamer, arabluatex, asapsym, asciilist, babel-belarusian, bibarts, biblatex-bookinarticle, biblatex-bookinother, biblatex-caspervector, biblatex-chicago, biblatex-gost, biblatex-ieee, biblatex-morenames, biblatex-opcit-booktitle, bibtexperllibs, bxdvidriver, bxenclose, bxjscls, bxnewfont, bxpapersize, chemnum, cjk-ko, cochineal, csplain, cstex, datetime2-finnish, denisbdoc, dtx, dvipdfmx-def, ejpecp, emisa, fithesis, fnpct, font-change-xetex, forest, formation-latex-ul, gregoriotex, gzt, hausarbeit-jura, hyperxmp, imakeidx, jacow, l3, l3kernel, l3packages, latex2e, latex2e-help-texinfo-fr, latex-bib2-ex, libertinust1math, lollipop, lt3graph, lua-check-hyphen, lualibs, luamplib, luatexja, mathalfa, mathastext, mcf2graph, media9, metrix, nameauth, ndsu-thesis, newtx, normalcolor, noto, nucleardata, nwejm, ocgx2, pdfcomment, pdfpages, pkuthss, polyglossia, proposal, qcircuit, reledmac, rmathbr, savetrees, scanpages, stex, suftesi, svrsymbols, teubner, tex4ebook, tex-ini-files, tikzmark, tikzsymbols, titlesec, tudscr, typed-checklist, ulthese, visualtikz, xespotcolor, xetex-def, xetexko, ycbook, yinit-otf.
Enjoy.
At this moment, some 153 Tamil refugees, fleeing the same type of instability that brought a horrible death to the passengers of MH17, have been locked up in the hull of a customs ship on the high seas. Windowless cabins and a supply of food not fit for a dog are part of the Government's strategy to brutalize these people for simply trying to avoid the risk of enhanced imprisonment(TM) in their own country.
Under international protocol for rescue at sea and political asylum, these people should be taken to the nearest port and given a humanitarian visa on arrival. Australia, however, is trying to lie and cheat their way out of these international obligations while squealing like a stuck pig about the plight of Australians in the hands of Putin. If Prime Minister Tony Abbott wants to encourage Putin to co-operate with the international community, shouldn't he try to lead by example? How can Australians be safe abroad if our country systematically abuses foreigners in their time of need?
I just released PyGObject 3.3.90, for GNOME 3.5.90.
This is now working correctly on big-endian 64 bit machines such as powerpc64, and fixes marshalling for GParamSpec attributes and return values, as well as a few small bug fixes.
Thanks to all contributors!
Complete list of changes:
One of the most frustrating things about Copyright Expansion is that it s being driven through at international levels, far away from most of the people that it affects. When we try to contact our international representatives, such as national government ministers or officers, or Members of the European Parliament (MEPs), it usually takes me at least two attempts to convince them that I actually understand the issue that I fell they re about to bodge.
So, here we go again:-
The European Parliament is being asked to nearly double the term of copyright afforded to sound recordings. Industry lobbyists suggest that extending copyright term will help increase the welfare of performers and session musicians. But the Term Extension Directive, which will be voted on by the Legal Affairs Committee in a few weeks time, will do no such thing.Follow developments on EDRI s Copyright page and Write to your MEP, like Stuart Langridge did. I ll contact Neil Parish MEP directly and hopefully he ll remember me, so I won t have to do that stop sending me copy-pasta from Malcolm Harbour dance. I ll also see if I can get nice responses out of the other parties this time.
This is a continuation from part 1 but can be enjoyed on its own too, like a good red wine. It s a bit early for me for red wine, though.
Alex Gaynor writes about "timeline views" in Django. The abstract problem is to merge already-sorted lazily evaluated lists by some specified attribute.
My solution is similar to Malcolm's suggestion in the comments and improves upon Alex's in that has superior space considerations, as well as supporting reverse ordering with the usual Django syntax:
import heapq def merge_querysets(*querysets, **kwargs): order_by = kwargs.get('order_by', None) if order_by is None: raise ValueError, "Must specify order_by in keyword arguments" # Are we sorting in reverse? reverse = False if order_by.startswith('-'): order_by = order_by[1:] reverse = True class Wrapper(object): """ Hacky wrapper class for custom sorting. """ def __init__(self, sortval, val): self.sortval = sortval self.val = val def __cmp__(self, other): if reverse: return cmp(other.sortval, self.sortval) else: return cmp(self.sortval, other.sortval) iterables = map(iter, querysets) heap = [] while True: for it in iterables: try: val = it.next() sortval = val.getattr(order_by) heapq.heappush(heap, Wrapper(sortval, val)) except StopIteration: # This QuerySet has been exhausted iterables.remove(it) continue if not iterables: # All QuerySets have been exhausted; just yield from the heap # to save cost of insertion. for item in heap: yield item.val return yield heapq.heappop(heap).val def my_view(request): tickets = Ticket.objects.order_by('-create_date') wikis = WikiEdit.objects.order_by('-create_date') changesets = Changeset.objects.order_by('-create_date') objs = merge_querysets(tickets, wikis, changesets, order_by='-create_date') return render_to_response('my_app/template.html', 'objects': objs, )
Guess it's really been longer than a year that haskell has been on my
mind, though not much lately. Things are aligning again. I'll shortly be
visiting the Bay Area again -- last time I piled up haskell documentation
for the plane trip. This time I'm looking forward to the Real World Haskell
book waiting in the mailbox when I get back.
I read and commented on the first several draft chapters, hoping my
ignorance would be useful, and I know one of the authors (though if I'm not
mistaken I've never met him). So I'm looking forward to reading it, but
also feeling guilty that I haven't managed to do anything serious with
haskell yet. No new project that I dared, or had the patience, to attempt
in it. But that's what the book's supposed to solve. Getting over the gap
from a basic understanding to being able to add the language as another
tool in the kit.
And hey, it's better than seriously learning javascript would be, right?
Here are some tricks that you can use to ensure that your announcement to
debian-devel-announce has maximal negative utility.
hide who's talking
Make sure that your announcement does not start off by making clear who's
speaking. The announcement will have a From line with your name on it, and
that's good enough. Everyone knows who you are, what groups you're a member
of, and can guess what group you're speaking for ... or whether you're just
speaking for yourself.
Throughout the annoucement, be use to use terms like "We plan", "let us
describe", "we are considering", "This is where our proposal comes in.", etc.
Maybe in a footnote include a clue such as a mention of "the NM-Committee",
but be sure to not sign the announcement with the name of any group or
team or set of people, so that the reader is left guessing about who this
"we" is all the way to the end and beyond.
(And hey, "cabal" is a fun word.)
weasel words
Scatter throughout the announcement some vaguely worded digs at things you
disagree with. The key here is to make clear what you really mean, while
providing enough plausible deniability that you can deny having said it
later.
So you might say ...
Some time ago a few Developers thus went and pushed forward the "Debian Maintainer" status.... Thus implying that this was a few malcontents who should be ignored. Sure, the whole project voted for this, and so you should be sure to allude to that vote somewhere else. Maybe in a different paragraph entirely, you could refer to "Debian Maintainers (DM) [GR-DM]". (It would be excessive to have that footnote refer to a bogus url like http://vote.debian.org/something.) the big lie Make sure that your announcement starts off all bright and cheery on the surface, with just a hint of deadly steel jaws underneath. Something like:
If you are an existing Debian Developer or Debian Maintainer, don't be afraid, we are not going to take anything away from you.Natter on for many, many pages of trivilities, good ideas, bad ideas, half-baked ideas, conflicting acronyms, and useless footnotes. Then spring the trap:
In future this will be a list maintained by the NM-committee. At the time of migrating from the old to the new way, Ftpmaster will convert the existing DM-Upload-Allowed fields into that list, so there should be no interruption in your ability to upload.And ability to upload is all that matters. Who cares about the regression back to the tyranny of unix permissions? Who cares that DDs won't be able to decide when they trust a DM to upload their package, or that DMs won't be able to sign up quickly and easily, or that DMs will have to petition a disinterested committe to change anything? Nothing will have changed, nothing will be taken away, nothing to see here, move along: The big lie.
or “how did we do Batting Against Three Strikes through the Back Door?”
In the latest EDRI-gram, there’s a report on the telecoms package EuroParl vote which is mostly positive. Thankfully both the Harbour and Trautmann reports were amended in vital ways to protect user privacy.
So, just to tie up my last loose end from my last report: Neil Parish MEP never did bring any replies from Malcolm Harbour back to me. Conservative MEPs, eh? At least they make the right noises on privacy and freedom now, even if the actions are still old-fashioned and confusingly unconvincing.
One of the reasons that I didn’t post much last week was that I was contacting some of my MEPs as part of the Telecom Packet Action in order to help defend cooperative software distribution and use. This is a bit longer than my usual posts and no software is announced below, but I’m posting it here in the hope that more people will take part in this campaign that affects every EU internet user.
In short, some MEPs that I remember from the Software Patent campaign proposed amendments that would enable the unpopular three-strikes regime, where internet users could be disconnected and fined if they had three allegations of copyright infringement made (not necessarily proved AIUI) against them, and ISPs could be required to block legal peer-to-peer cooperative distribution systems like BitTorrent. ISPs would also have to distribute official advice about proper internet use and I remember how good that often is. Note that three-strikes wasn’t explicit in the amendments, but there did appear to be loopholes which could be used to allow it, such as only prohibiting technical feature requirements. For longer descriptions, see articles by La Quadrature du Net (Squaring the Net) and The Open Rights Group.
The UK Independence Party MEPs replied quickly that they would vote against, mainly because of the source of the legislation. I don’t completely agree with that, but at least UKIP does what it says on the tin.
Neil Parish MEP of the Conservatives sent me back a reply which referred to unknown numbering (H1, K1 and so on) and generally seemed like a mish-mash of replies that other people had received. I replied with questions about several aspects, including asking where his numbering came from, and I’ve not seen an answer yet. I’ll email him again in a few days because it’s been over a week.
I was really annoyed by the reported complaints by Malcolm Harbour (Conservatives lead MEP on this issue) about how voters have reacted to these amendments, where he called it “scaremongering” amongst other things. No mongering was required - those amendments are damn scary. See comments from a media solicitor at the end of a BBC report:-
“The amendment will cause several problems, firstly, many broadband users routinely transfer large files which are encrypted. “Many of these are acting quite legitimately and in order to determine whether or not such large files are or are not the produce of illicit file sharing the ISP will have to carry out an unprecedented degree of analysis of its customers’ traffic. “Furthermore, computers are frequently shared - within offices, within homes, within educational institutions and inadvertently, where wrong-doers “piggy back” on an inadequately secured Wi-Fi connection. “All this raises the spectre of people losing internet access - for reasons which are no fault of their own.”The main thing that these actions over the last few years have taught me is the broken state of European Union decision-making. I know I’m sometimes dissatisfied with problems in local and civil democracy, but the EU is a good idea whose democracy and accountability seems broken on a massive scale. There are just too many disconnects, as far as the ordinary voter can see. We put things in to elections and consultations and get bizarre law-making like this. If we act to correct the bizarre law-making, some MEPs start criticising us quite severely. And what kind of stupid representative system allows the representatives to repeatedly blame the voters but still be safely elected because of their high position on their party’s list? I’d love a simple, reformist constitution that’s simple enough for people to actually understand, approve in referendums and participate in the EU effectively. I doubt it will happen in my lifetime, though.
Next.