Search Results: "cas"

29 July 2020

Dima Kogan: An awk corner case?

So even after years and years of experience, core tools still find ways to surprise me. Today I tried to do some timestamp comparisons with mawk (vnl-filter, to be more precise), and ran into a detail of the language that made it not work. Not a bug, I guess, since both mawk and gawk are affected. I'll claim "language design flaw", however. Let's say I'm processing data with unix timestamps in it (seconds since the epoch). gawk and recent versions of mawk have strftime() for that: 
$ date
Wed Jul 29 15:31:13 PDT 2020
$ date +"%s"
1596061880
$ date +"%s"   mawk ' print strftime("%H",$1) '
15
And let's say I want to do something conditional on them. I want only data after 9:00 each day: 
$ date +"%s"   mawk 'strftime("%H",$1) >= 9  print "Yep. After 9:00" '
That's right. No output. But it is 15:31 now, and I confirmed above that strftime() reports the right time, so it should know that it's after 9:00, but it doesn't. What gives? As we know, awk (and perl after it) treat numbers and strings containing numbers similarly: 5+5 and ="5"+5= both work the same, which is really convenient. This can only work if it can be inferred from context whether we want a number or a string; it knows that addition takes two numbers, so it knows to convert ="5"= into a number in the example above. But what if an operator is ambiguous? Then it picks a meaning based on some internal logic that I don't want to be familiar with. And apparently awk implements string comparisons with the same < and > operators, as numerical comparisons, creating the ambiguity I hit today. strftime returns strings, and you get silent, incorrect behavior that then demands debugging. How to fix? By telling awk to treat the output of strftime() as a number: 
$ date +"%s"   mawk '0+strftime("%H",$1) >= 9  print "Yep. After 9:00" '
Yep. After 9:00
With the benefit of hindsight, they really should not have reused any operators for both number and string operations. Then these ambiguities wouldn't occur, and people wouldn't be grumbling into their blogs decades after these decisions were made.

28 July 2020

Chris Lamb: Pop culture matters

Many people labour under the assumption that pop culture is trivial and useless while only 'high' art can grant us genuine and eternal knowledge about the world. Given that we have a finite time on this planet, we are all permitted to enjoy pop culture up to a certain point, but we should always minimise our interaction with it, and consume more moral and intellectual instruction wherever possible. Or so the theory goes. What these people do not realise is that pop and mass culture can often provide more information about the world, humanity in general and what is even more important ourselves. This is not quite the debate around whether high art is artistically better, simply that pop culture can be equally informative. Jeremy Bentham argued in the 1820s that "prejudice apart, the game of push-pin is of equal value with the arts and sciences of music and poetry", that it didn't matter where our pleasures come from. (John Stuart Mill, Bentham's intellectual rival, disagreed.) This fundamental question of philosophical utilitarianism will not be resolved here. However, what might begin to be resolved is our instinctive push-back against pop culture. We all share an automatic impulse to disregard things we do not like and to pretend they do not exist, but this wishful thinking does not mean that these cultural products do not continue to exist when we aren't thinking about them and, more to our point, continue to influence others and even ourselves. Take, for example, the recent trend for 'millennial pink'. With its empty consumerism, faux nostalgia, reductive generational stereotyping, objectively ugly sthetics and tedious misogyny (photographed with Rose Gold iPhones), the very combination appears to have been deliberately designed to annoy me, curiously providing circumstantial evidence in favour of intelligent design. But if I were to immediately dismiss millennial pink and any of the other countless cultural trends I dislike simply because I find them disagreeable, I would be willingly keeping myself blind to their underlying ideology, their significance and their effect on society at large. If I had any ethical or political reservations I might choose not to engage with them economically or to avoid advertising them to others, but that is a different question altogether. Even if we can't notice this pattern within ourselves we can first observe it in others. We can all recall moments where someone has brushed off a casual reference to pop culture, be it Tiger King, TikTok, team sports or Taylor Swift; if you can't, simply look for the abrupt change of tone and the slightly-too-quick dismissal. I am not suggesting you attempt to dissuade others or even to point out this mental tic, but merely seeing it in action can be highly illustrative in its own way. In summary, we can simultaneously say that pop culture is not worthy of our time relative to other pursuits while consuming however much of it we want, but deliberately dismissing pop culture doesn't mean that a lot of other people are not interacting with it and is therefore undeserving of any inquiry. And if that doesn't convince you, just like the once-unavoidable millennial pink, simply sticking our collective heads in the sand will not mean that wider societal-level ugliness is going to disappear anytime soon. Anyway, that's a very long way of justifying why I plan to re-watch TNG.

Steve Kemp: I'm a bit of a git (hacker?)

Sometimes I enjoy reading the source code to projects I like, use, or am about to install for the first time. This was something I used to do on a very regular basis, looking for security issues to report. Nowadays I don't have so much free time, but I still like to inspect the source code to new applications I install, and every now and again I'll find the time to look at the source to random projects. Reading code is good. Reading code is educational. One application I've looked at multiple times is redis, which is a great example of clean and well-written code. That said when reading the redis codebase I couldn't help noticing that there were a reasonably large number of typos/spelling mistakes in the comments, so I submitted a pull-request: Sadly that particular pull-request didn't receive too much attention, although a previous one updating the configuration file was accepted. I was recently reminded of these pull-requests when I was when I was doing some other work. So I figured I'd have a quick scan of a couple of other utilities. In the past I'd just note spelling mistakes when I came across them, usually I'd be opening each file in a project one by one and reading them from top to bottom. (Sometimes I'd just open files in emacs and run "M-x ispell-comments-and-strings", but more often I'd just notice them with my eyes). It did strike me that if I were to do this in a more serious fashion it would be good to automate it. So this time round I hacked up a simple "dump comments" utility, which would scan named files and output the contents of any comments (be they single-line, or multi-line). Once I'd done that I could spell-check easily: 
 $ go run dump-comments.go *.c > comments
 $ aspell -c comments
Anyway the upshot of that was a pull-request against git: We'll see if that makes its way live sometime. In case I get interested in doing this again I've updated my sysbox-utility collection to have a comments sub-command. That's a little more robust and reliable than my previous hack: 
$ sysbox comments -pretty=true $(find . -name '*.c')
..
..
The comments sub-command has support for: Adding new support would be trivial, I just need a start and end pattern to search against. Pull-requests welcome:

Russ Allbery: Review: The City in the Middle of the Night

Review: The City in the Middle of the Night, by Charlie Jane Anders
Publisher: Tor
Copyright: February 2019
Printing: February 2020
ISBN: 1-4668-7113-X
Format: Kindle
Pages: 366
January is a tidally-locked planet divided between permanent night and permanent day, an unfortunate destination for a colony starship. Now, humans cling to a precarious existence along the terminator, huddling in two wildly different cities and a handful of smaller settlements, connected by a road through the treacherous cold. The novel opens with Sophie, a shy university student from the dark side of the city of Xiosphant. She has an overwhelming crush on Bianca, her high-class, self-confident roommate and one of the few people in her life to have ever treated her with compassion and attention. That crush, and her almost non-existent self-esteem, lead her to take the blame for Bianca's petty theft, resulting in what should have been a death sentence. Sophie survives only because she makes first contact with a native intelligent species of January, one that the humans have been hunting for food and sport. Sadly, I think this is enough Anders for me. I've now bounced off two of her novels, both for structural reasons that I think go deeper than execution and indicate a fundamental mismatch between what Anders wants to do as an author and what I'm looking for as a reader. I'll talk more about what this book is doing in a moment, but I have to start with Bianca and Sophie. It's difficult for me to express how much I loathed this relationship and how little I wanted to read about it. It took me about five pages to peg Bianca as a malignant narcissist and Sophie's all-consuming crush as dangerous codependency. It took the entire book for Sophie to figure out how awful Bianca is to her, during which Bianca goes through the entire abusive partner playbook of gaslighting, trivializing, contingent affection, jealous rage, and controlling behavior. And meanwhile Sophie goes back to her again, and again, and again, and again. If I hadn't been reading this book on a Kindle, I think it would have physically hit a wall after their conversation in the junkyard. This is truly a matter of personal taste and preference. This is not an unrealistic relationship; this dynamic happens in life all too often. I'm sure there is someone for whom reading about Sophie's spectacularly poor choices is affirming or cathartic. I've not personally experienced this sort of relationship, which doubtless matters. But having empathy for someone who is making awful and self-destructive life decisions and trusting someone they should not be trusting and who is awful to them in every way is difficult work. Sophie is the victim of Bianca's abuse, but she does so many stupid and ill-conceived things in support of this twisted relationship that I found it very difficult to not get angry at her. Meanwhile, Anders writes Sophie as so clearly fragile and uncertain and devoid of a support network that getting angry at her is like kicking a puppy. The result for me was spending nearly an entire book in a deeply unpleasant state of emotional dissonance. I may be willing to go through that for a close friend, but in a work of fiction it's draining and awful and entirely not fun. The other viewpoint character had the opposite problem for me. Mouth starts the book as a traveling smuggler, the sole survivor of a group of religious travelers called the Citizens. She's practical, tough, and guarded. Beneath that, I think the intent was to show her as struggling to come to terms with the loss of her family and faith community. Her first goal in the book is to recover a recording of Citizen sacred scripture to preserve it and to reconnect with her past. This sounds interesting on the surface, but none of it gelled. Mouth never felt to me like someone from a faith community. She doesn't act on Citizen beliefs to any meaningful extent, she rarely talks about them, and when she does, her attitude is nostalgia without spirituality. When Mouth isn't pursuing goals that turn out to be meaningless, she aimlessly meanders through the story. Sophie at least has agency and makes some important and meaningful decisions. Mouth is just there, even when Anders does shattering things to her understanding of her past. Between Sophie and Bianca putting my shoulders up around my ears within the first few pages of the first chapter and failing to muster any enthusiasm for Mouth, I said the eight deadly words ("I don't care what happens to these people") about a hundred pages in and the book never recovered. There are parts of the world-building I did enjoy. The alien species that Sophie bonds with is not stunningly original, but it's a good (and detailed) take on one of the alternate cognitive and social models that science fiction has dreamed up. I was comparing the strangeness and dislocation unfavorably to China Mi ville's Embassytown while I was reading it, but in retrospect Anders's treatment is more decolonialized. Xiosphant's turn to Circadianism as their manifestation of order is a nicely understated touch, a believable political overreaction to the lack of a day/night cycle. That touch is significantly enhanced by Sophie's time working in a salon whose business model is to help Xiosphant residents temporarily forget about time. And what glimmers we got of politics on the colony ship and their echoing influence on social and political structures were intriguing. Even with the world-building, though, I want the author to be interested in and willing to expand the same bits of world-building that I'm engaged with. Anders didn't seem to be. The reader gets two contrasting cities along a road, one authoritarian and one libertine, which makes concrete a metaphor for single-axis political classification. But then Anders does almost nothing with that setup; it's just the backdrop of petty warlord politics, and none of the political activism of Bianca's student group seems to have relevance or theoretical depth. It's a similar shallowness as the religion of Mouth's Citizens: We get a few fragments of culture and religion, but without narrative exploration and without engagement from any of the characters. The way the crew of the Mothership was assembled seems to have led to a factional and racial caste system based on city of origin and technical expertise, but I couldn't tell you more than that because few of the characters seem to care. And so on. In short, the world-building that I wanted to add up to a coherent universe that was meaningful to the characters and to the plot seemed to be little more than window-dressing. Anders tosses in neat ideas, but they don't add up to anything. They're just background scenery for Bianca and Sophie's drama. The one thing that The City in the Middle of the Night does well is Sophie's nervous but excited embrace of the unknown. It was delightful to see the places where a typical protagonist would have to overcome a horror reaction or talk themselves through tradeoffs and where Sophie's reaction was instead "yes, of course, let's try." It provided an emotional strength to an extended first-contact exploration scene that made it liberating and heart-warming without losing the alienness. During that part of the book (in which, not coincidentally, Bianca does not appear), I was able to let my guard down and like Sophie for the first time, and I suspect that was intentional on Anders's part. But, overall, I think the conflict between Anders's story-telling approach and my preferences as a reader are mostly irreconcilable. She likes to write about people who make bad decisions and compound their own problems. In one of the chapters of her non-fiction book about writing that's being serialized on Tor.com she says "when we watch someone do something unforgivable, we're primed to root for them as they search desperately for an impossible forgiveness." This is absolutely not true for me; when I watch a character do something unforgivable, I want to see repudiation from the protagonists and ideally some clear consequences. When that doesn't happen, I want to stop reading about them and find something more enjoyable to do with my time. I certainly don't want to watch a viewpoint character insist that the person who is doing unforgivable things is the center of her life. If your preferences on character and story arc are closer to Anders's than mine, you may like this book. Certainly lots of people did; it was nominated for multiple awards and won the Locus Award for Best Science Fiction Novel. But despite the things it did well, I had a truly miserable time reading it and am not anxious to repeat the experience. Rating: 4 out of 10

27 July 2020

Matthew Garrett: Filesystem deduplication is a sidechannel

First off - nothing I'm going to talk about in this post is novel or overly surprising, I just haven't found a clear writeup of it before. I'm not criticising any design decisions or claiming this is an important issue, just raising something that people might otherwise be unaware of.

With that out of the way: Automatic deduplication of data is a feature of modern filesystems like zfs and btrfs. It takes two forms - inline, where the filesystem detects that data being written to disk is identical to data that already exists on disk and simply references the existing copy rather than, and offline, where tooling retroactively identifies duplicated data and removes the duplicate copies (zfs supports inline deduplication, btrfs only currently supports offline). In a world where disks end up with multiple copies of cloud or container images, deduplication can free up significant amounts of disk space.

What's the security implication? The problem is that deduplication doesn't recognise ownership - if two users have copies of the same file, only one copy of the file will be stored[1]. So, if user a stores a file, the amount of free space will decrease. If user b stores another copy of the same file, the amount of free space will remain the same. If user b is able to check how much free space is available, user b can determine whether the file already exists.

This doesn't seem like a huge deal in most cases, but it is a violation of expected behaviour (if user b doesn't have permission to read user a's files, user b shouldn't be able to determine whether user a has a specific file). But we can come up with some convoluted cases where it becomes more relevant, such as law enforcement gaining unprivileged access to a system and then being able to demonstrate that a specific file already exists on that system. Perhaps more interestingly, it's been demonstrated that free space isn't the only sidechannel exposed by deduplication - deduplication has an impact on access timing, and can be used to infer the existence of data across virtual machine boundaries.

As I said, this is almost certainly not something that matters in most real world scenarios. But with so much discussion of CPU sidechannels over the past couple of years, it's interesting to think about what other features also end up leaking information in ways that may not be obvious.

(Edit to add: deduplication isn't enabled on zfs by default and is explicitly triggered on btrfs, so unless it's something you've enabled then this isn't something that affects you)

[1] Deduplication is usually done at the block level rather than the file level, but given zfs's support for variable sized blocks, identical files should be deduplicated even if they're smaller than the maximum record size

comment count unavailable comments

Martin Michlmayr: ledger2beancount 2.4 released

I released version 2.4 of ledger2beancount, a ledger to beancount converter. There are two notable changes in this release:
  1. I fixed two regressions introduced in the last release. Sorry about the breakage!
  2. I improved support for hledger. I believe all syntax differences in hledger are supported now.
Here are the changes in 2.4: Thanks to Kirill Goncharov for pointing out one regressions, to Taylor R Campbell for for a patch, to Stefano Zacchiroli for some input, and finally to Simon Michael for input on hledger! You can get ledger2beancount from GitHub

Russ Allbery: Review: Rise of the Warrior Cop

Review: Rise of the Warrior Cop, by Radley Balko
Publisher: PublicAffairs
Copyright: 2013
ISBN: 1-61039-212-4
Format: Kindle
Pages: 336
As the United States tries, in fits and starts, to have a meaningful discussion about long-standing police racism, brutality, overreach, corruption, and murder, I've realized that my theoretical understanding of the history of and alternative frameworks for law enforcement is woefully lacking. Starting with a book by a conservative white guy is not the most ideal of approaches, but it's what I already had on hand, and it won't be the last book I read and review on this topic. (Most of my research so far has been in podcast form. I don't review those here, but I can recommend Ezra Klein's interviews with Ta-Nehisi Coates, Paul Butler, and, most strongly, sujatha baliga.) Rise of the Warrior Cop is from 2013 and has had several moments of fame, no doubt helped by Balko's connections to the conservative and libertarian right. One of the frustrating facts of US politics is that critiques of the justice system from the right (and from white men) get more media attention than critiques from the left. That said, it's a generally well-respected book on the factual history of the topic, and police brutality and civil rights are among the points on which I have stopped-clock agreements with US libertarians. This book is very, very libertarian. In my callow youth, I was an ardent libertarian, so I've read a lot of US libertarian literature. It's a genre with its own conventions that become obvious when you read enough of it, and Rise of the Warrior Cop goes through them like a checklist. Use the Roman Republic (never the Roman Empire) as the starting point for any political discussion, check. Analyze the topic in the context of pre-revolutionary America, check. Spend considerable effort on discerning the opinions of the US founders on the topic since their opinions are always relevant to the modern world, check. Locate some point in the past (preferably before 1960) where the political issue was as good as it has ever been, check. Frame all changes since then as an erosion of rights through government overreach, check. Present your solution as a return to a previous era of respect for civil rights, check. Once you start recognizing the genre conventions, their prevalence in libertarian writing is almost comical. The framing chapters therefore leave a bit to be desired, but the meat of the book is a useful resource. Starting with the 1970s and its use as a campaigning tool by Nixon, Balko traces a useful history of the war on drugs. And starting with the 1980s, the number of cites to primary sources and the evidence of Balko's own research increases considerably. If you want to know how US police turned into military cosplayers with body armor, heavy weapons, and armored vehicles, this book provides a lot of context and history. One of the reasons why I view libertarians as allies of convenience on this specific issue is that drug legalization and disgust with the war on drugs have been libertarian issues for decades. Ideologically honest libertarians (and Balko appears to be one) are inherently skeptical of the police, so when the police overreach in an area of libertarian interest, they notice. Balko makes a solid argument, backed up with statistics, specific programs, legislation, and court cases, that the drug war and its accompanying lies about heavily-armed drug dealers and their supposed threat to police officers was the fuel for the growth of SWAT teams, no-knock search warrants, erosion of legal protections for criminal defendants, and de facto license for the police to ignore the scope and sometimes even the existence of warrants. This book is useful support for the argument that fears for the safety of officers underlying the militarization of police forces are imaginary. One telling point that Balko makes repeatedly and backs with statistical and anecdotal evidence is that the police generally do not use raid tactics on dangerous criminals. On the contrary, aggressive raids are more likely to be used on the least dangerous criminals because they're faster, they're fun for the police (they provide an adrenaline high and let them play with toys), and they're essentially risk-free. If the police believe someone is truly dangerous, they're more likely to use careful surveillance and to conduct a quiet arrest at an unexpected moment. The middle-of-the-night armed break-ins with battering rams, tear gas, and flash-bangs are, tellingly, used against the less dangerous suspects. This is part of Balko's overall argument that police equipment and tactics have become untethered from any realistic threat and have become cultural. He traces an acceleration of that trend to 9/11 and the resulting obsession with terrorism, which further opened the spigot of military hardware and "special forces" training. This became a point of competition between police departments, with small town forces that had never seen a terrorist and had almost no chance of a terrorist incident demanding their own armored vehicles. I've encountered this bizarre terrorism justification personally; one of the reasons my local police department gave in a public hearing for not having a policy against shooting at moving vehicles was "but what if terrorism?" I don't believe there has ever been a local terrorist attack. SWAT in such places didn't involve the special training or dedicated personnel of large city forces; instead, it was a part-time duty for normal police officers, and frequently they were encouraged to practice SWAT tactics by using them at random for some otherwise normal arrest or search. Balko argues that those raids were more exciting than normal police work, leading to a flood of volunteers for that duty and a tendency to use them as much as possible. That in turn normalizes disconnecting police tactics from the underlying crime or situational risk. So far, so good. But despite the information I was able to extract from it, I have mixed feelings about Rise of the Warrior Cop as a whole. At the least, it has substantial limitations. First, I don't trust the historical survey of policing in this book. Libertarian writing makes for bad history. The constraints of the genre require overusing only a few points of reference, treating every opinion of the US founders as holy writ, and tying forward progress to a return to a previous era, all of which interfere with good analysis. Balko also didn't do the research for the historical survey, as is clear from the footnotes. The citations are all to other people's histories, not to primary sources. He's summarizing other people's histories, and you'll almost certainly get better history by finding well-respected historians who cover the same ground. (That said, if you're not familiar with Peel's policing principles, this is a good introduction.) Second, and this too is unfortunately predictable in a libertarian treatment, race rarely appears in this book. If Balko published the same book today, I'm sure he would say more about race, but even in 2013 its absence is strange. I was struck while reading by how many examples of excessive police force were raids on west coast pot farms; yes, I'm sure that was traumatic, but it's not the demographic I would name as the most vulnerable to or affected by police brutality. West coast pot growers are, however, mostly white. I have no idea why Balko made that choice. Perhaps he thought his target audience would be more persuaded by his argument if he focused on white victims. Perhaps he thought it was an easier and less complicated story to tell. Perhaps, like a lot of libertarians, he doesn't believe racism has a significant impact on society because it would be a market failure. Perhaps those were the people who more readily came to mind. But to talk about police militarization, denial of civil rights, and police brutality in the United States without putting race at the center of both the history and the societal effects leaves a gaping hole in the analysis. Given that lack of engagement, I also am dubious of Balko's policy prescriptions. His reform suggestions aren't unreasonable, but they stay firmly in the centrist and incrementalist camp and would benefit white people more than black people. Transparency, accountability, and cultural changes are all fine and good, but the cultural change Balko is focused on is less aggressive arrest tactics, more use of mediation, and better physical fitness. I would not object to those things (well, maybe the last, which seemed odd), but we need to have a discussion about police white supremacist organizations, the prevalence of spousal abuse, and the police tendency to see themselves not as public servants but as embattled warriors who are misunderstood by the naive sheep they are defending. And, of course, you won't find in Rise of the Warrior Cop any thoughtful wrestling with whether there are alternative approaches to community safety, whether punitive rather than restorative justice is effective, or whether crime is a symptom of deeper societal problems we could address but refuse to. The most radical suggestion Balko has is to legalize drugs, which is both the predictable libertarian position and, as we have seen from recent events in the United States, far from the only problem of overcriminalization. I understand why this book is so frequently mentioned on-line, and its author's political views may make it more palatable to some people than a more race-centered or radical perspective. But I don't think this is the best or most useful book on police violence that one could read today. I hope to find a better one in upcoming reviews. Rating: 6 out of 10

26 July 2020

Russ Allbery: Review: Paladin's Grace

Review: Paladin's Grace, by T. Kingfisher
Publisher: Red Wombat Studio
Copyright: 2020
ASIN: B0848Q8JVW
Format: Kindle
Pages: 399
Stephen was a paladin. Then his god died. He was a berserker, an unstoppable warrior in the service of his god. Now, well, he's still a berserker, but going berserk when you don't have a god to control the results is not a good idea. He and his brothers were taken in by the Temple of the Rat, where they serve as guards, watch out for each other, and try to get through each day with an emptiness in their souls where a god should be. Stephen had just finished escorting a healer through some of the poorer parts of town when a woman runs up to him and asks him to hide her. Their awkward simulated tryst is sufficient to fool the two Motherhood priests who were after her for picking flowers from the graveyard. Stephen then walks her home and that would have been the end of it, except that neither could get the other out of their mind. Despite first appearances, and despite being set in the same world and sharing a supporting character, this is not the promised sequel to Swordheart (which is apparently still coming). It's an entirely different paladin story. T. Kingfisher (Ursula Vernon's nom de plume when writing for adults) has a lot of things to say about paladins! And, apparently, paladin-involved romances. On the romance front, Kingfisher clearly has a type. The general shape of the story will be familiar from Swordheart and The Wonder Engine: An independent and occasionally self-confident woman with various quirks, a hunky paladin who is often maddeningly dense, and a lot of worrying on both sides about whether the other person is truly interested in them and if their personal liabilities make a relationship a horrible idea. This is not my preferred romance formula (it provokes the occasional muttered "for the love of god just talk to each other"), but I liked this iteration of it better than the previous two, mostly because of Grace. Grace is a perfumer, a trade she went into by being picked out of a lineup of orphans by a master perfumer for her sense of smell. One of Kingfisher's strengths as a writer is showing someone get lost in their routine day-to-day competence. When mixed with an inherently fascinating profession, this creates a great reading experience. Grace is also an abuse survivor, which made the communication difficulties with Stephen more interesting and subtle. Grace has created space and a life for herself, and her unwillingness to take risks on changes is a deep part of her sense of self and personal safety. As her past is slowly revealed, Kingfisher puts the reader in a position to share Stephen's anger and protectiveness, but then consistently puts Grace's own choices, coping mechanisms, and irritated refusal to be protected back into the center of the story. She has to accept some help as she gets entangled in the investigation of a highly political staged assassination attempt, but both that help and the relationship come on her own terms. It's very well-done. The plot was enjoyable enough, although it involved a bit too much of constantly rising stakes and turns for the worst for my taste, and the ending had a touch of deus ex machina. Like Kingfisher's other books, though, the delight is in the unexpected details. Stephen knitting socks. Grace's frustrated obsession with why he smells like gingerbread. The beautifully practical and respectful relationship between the Temple of the Rat and Stephen's band of former paladins. (After only two books in which they play a major role, the Temple of the Rat is already one of my favorite fantasy religions.) Everything about Bishop Beartongue. Grace's friend Marguerite. And a truly satisfying ending. The best part of this book, though, is the way Grace is shown as a complete character in a way that even most books with well-rounded characterization don't manage. Some things she does make the reader's heart ache because of the hints they provide about her past, but they're also wise and effective safety mechanisms for her. Kingfisher gives her space to be competent and prickly and absent-minded. She has a complete life: friends, work, goals, habits, and little rituals. Grace meets someone and falls in love, but one can readily imagine her not falling in love and going on with her life and that result wouldn't be tragic. In short, she feels like a grown adult who has made her own peace with where she came from and what she is doing. The book provides her an opportunity for more happiness and more closure without undermining her independence. I rarely see this in a novel, and even more rarely done this well. If you haven't read any of Kingfisher's books and are in the mood for faux-medieval city romance involving a perfumer and a bit of political skulduggery, this is a great place to start. If you liked Swordheart, you'll probably like Paladin's Grace; like me, you may even like it a bit more. Recommended, particularly if you want something light and heart-warming. Rating: 8 out of 10

23 July 2020

Enrico Zini: Build Qt5 cross-builder with raspbian sysroot: compiling with the sysroot (continued)

Lite extra ball, from https://www.flickr.com/photos/st3f4n/143623902 This is part of a series of posts on compiling a custom version of Qt5 in order to develop for both amd64 and a Raspberry Pi. The previous rounds of attempts ended in one issue too many to investigate in the allocated hourly budget. Andreas Gruber wrote:
Long story short, a fast solution for the issue with EGLSetBlobFuncANDROID is to remove libraspberrypi-dev from your sysroot and do a full rebuild. There will be some changes to the configure results, so please review them - if they are relevant for you - before proceeding with your work.
That got me unstuck! dpkg --purge libraspberrypi-dev in the sysroot, and we're back in the game. While Qt5's build has proven extremely fragile, I was surprised that some customization from Raspberry Pi hadn't yet broken something. In the end, they didn't disappoint. More i386 issues The run now stops with a new 32bit issue related to v8 snapshots: 
qt-everywhere-src-5.15.0/qtwebengine/src/core/release$ /usr/bin/g++ -pie -Wl,--fatal-warnings -Wl,--build-id=sha1 -fPIC -Wl,-z,noexecstack -Wl,-z,relro -Wl,-z,now -Wl,-z,defs -Wl,--as-needed -m32 -pie -Wl,--disable-new-dtags -Wl,-O2 -Wl,--gc-sections -o "v8_snapshot/mksnapshot" -Wl,--start-group @"v8_snapshot/mksnapshot.rsp"  -Wl,--end-group  -ldl -lpthread -lrt -lz
/usr/bin/ld: skipping incompatible //usr/lib/x86_64-linux-gnu/libz.so when searching for -lz
/usr/bin/ld: skipping incompatible //usr/lib/x86_64-linux-gnu/libz.a when searching for -lz
/usr/bin/ld: cannot find -lz
collect2: error: ld returned 1 exit status
Attempted solution: apt install zlib1g-dev:i386. Alternative solution (untried): configure Qt5 with -no-webengine-v8-snapshot. It builds! Installation paths Now it tries to install files into debian/tmp/home/build/sysroot/opt/qt5custom-armhf/. I realise that I now need to package the sysroot itself, both as a build-dependency of the Qt5 cross-compiler, and as a runtime dependency of the built cross-builder. Conclusion The current work in progress, patches, and all, is at https://github.com/Truelite/qt5custom/tree/master/debian-cross-qtwebengine It blows my mind how ridiculously broken is the Qt5 cross-compiler build, for a use case that, looking at how many people are trying, seems to be one of the main ones for the cross-builder.

20 July 2020

Shirish Agarwal: Hearing loss, pandemic, lockdown

Sorry for not being on blog for sometime, the last few months have been brutal. While I am externally ok, because of the lockdown I sensed major hearing loss. First, I thought it may be a hallucination or something but as it persisted for days, I got myself checked and found out that I got 80% hearing loss in my right ear. How and why I don t know. Is this NIHL or some other kind of hearing loss is yet to be ascertained. I do live what is and used to be one of the busiest roads in the city, now for last few months not so much. On top of it, you have various other noises.

Tinnitus I also experienced Tinnitus which again I perceived to be a hallucination but found it s not. I have no clue if my eiplepsy has anything to do with hearing loss or both are different. I did discover that while today we know that something like Tinnitus exists, just 10-15 years back, people might mistake it for madness. In a way it is madness because you are constantly hearing sound, music etc. 24 7 , that is enough to drive anybody mad. During this brief period, did learn what an Otoscope is . I did get audiometry tests done but need to get at least a second or if possible also a third opinion but those will have to wait as the audio clinics are about 8-10 kms. away. In the open-close-open-close environment just makes it impossible to figure out the time, date and get it done. After that is done then probably get a hearing device, probably a Siemens Signia hearing aid. The hearing aids are damn expensive, almost 50k per piece and they probably have a lifetime of about 5-6 years, so it s a bit of a expensive proposition. I also need a second or/and third opinion on the audiometry profile so I know things are correct. All of these things are gonna take time.

Pandemic Situation in India and Testing Coincidentally, was talking to couple of people about this. It is sad to see that we have the third highest number of covid cases at 1/10th the tests we are doing vis-a-vis U.S.A. According to statistical site ourworldindata , we seem to be testing 0.22 per thousand people compared to 2.28 people per thousand done by United States. Sadly it doesn t give the breakup of the tests, from what I read the PCR tests are better than the antibody tests, a primer shares the difference between the two tests. IIRC, the antibody tests are far cheaper than the swab tests but swab tests are far more accurate as it looks for the virus s genetic material (RNA) . Anyways coming to the numbers, U.S. has a population of roughly 35 crores taking a little bit liberty from numbers given at popclock . India meanwhile has 135 crore or almost four times the population of U.S. and the amount of testing done is 1/10th as shared above. Just goes to share where the GOI priorities lie . We are running out of beds, ventilators and whatever else there is. Whatever resources are there are being used for covid patients and they are being charged a bomb. I have couple of hospitals near my place and the cost of a bed in an isolation ward is upward of INR 100k and if you need a ventilator then add another 50k . And in moment of rarity, the differences between charges of private and public are zero. Meaning there is immense profiteering happening it seems in the medical world. Heck, even the Govt. is on the act where they are charging 18% GST on sanitizers. If this is not looting then I dunno what is.
Example of Medical Bills people have to pay.

China, Nepal & Diplomacy While everybody today knows how China has intruded and captured quite a part of Ladakh, this wasn t the case when they started in April. That time Ajai Shukla had shared this with the top defence personnel but nothing came of it. Then on May 30th he broke/shared the news with the rest of the world and was immediately branded anti-national, person on Chinese payroll and what not. This is when he and Pravin Sawnhey of Force Magazine had both been warning of the same from last year itself. Pravin, has a youtube channel and had been warning India against Chinese intentions from 2015 and even before that. He had warned repeatedly that our obsession with the Pakistan border meant that we were taking eyes of the border with China which spans almost 2300 odd kms. going all the way to Arunachal Pradesh. A good map which shows the conflict can be found at dw.com which I am sharing/reproducing below
India-China Border Areas Copyright DW.com 2020
Note:- I am sharing a neutral party s rendering of the border disputes or somebody who doesn t have much at stake as the two countries have so that things could be looked at little objectively. The Prime Minister on the other hand, made the comment which made galvanising a made-up word into verb . It means to go without coming in. In fact, several news sites shared the statement told by the Prime Minister and the majority of people were shocked. In fact, there had been reports that he gave the current CDS, General Rawat, a person of his own choosing, a peace of his mind. But what lead to this confrontation in the first place ? I think many pieces are part of that puzzle, one of the pieces are surely the cutting of defense budget for the last 6 years, Even this year, if you look at the budget slashes done in the earlier part of the year when he shared how HAL had to raise loans from the market to pay salaries of its own people. Later he shared how the Govt. was planning to slash the defence budget. Interestingly, he had also shared some of the reasons which reaffirm that it is the only the Govt. which can solve some of the issues/conundrum

First, it must recognize that our firms competing for global orders are up against rivals that are being supported by their home governments with tax and export incentives and infrastructure that almost invariably surpasses India s. Our government must provide its aerospace firms with a level playing field, if not a competitive advantage. The greatest deterrent to growth our companies face is the high cost of capital and lack of access to funds. In several cases, Indian MSMEs have had to turn down offers to build components and assemblies for global OEM supply chains simply because the cost of capital to create the shop floor and train the personnel was too high. This resulted in a loss of business and a missed opportunity for creating jobs and skills. To overcome this, the government could create a sector specific A&D Fund to provide low cost capital quickly to enable our MSMEs to grab fleeting business opportunities. Ajai Shukla, blogpost 13th March 2020 . And then reporting on 11th May 2020 itself, CDS Gen. Rawat himself commented on saving the budget, they were in poor taste but still he shared what he thought about it. So, at the end of it one part of the story. The other part of the story probably lies in India s relations with its neighbors and lack of numbers in diplomats and diplomacy. So let me cover both the things one by one .

Diplomats, lack of numbers and hence the hands we are dealth with When Mr. Modi started his first term, he used the term Maximum Governance, Minimum Government but sadly cut those places where it indeed needs more people, one of which is diplomacy. A slightly dated 2012 article/opinion shared writes that India needs to engage with the rest of the world and do with higher number. Cut to 2020 and the numbers more or less remain the same . What Mr. Modi tried to do is instead of using diplomats, he tried to use his charm and hugopolicy for lack of a better term. 6 years later, here we are. After 200 trips abroad, not a single trade agreement to show what he done. I could go on but both time and energy are not on my side hence now switching to Nepal

Nepal, once friend, now enemy ? Nepal had been a friend of India for 70 odd years, what changed in the last few years that it changed from friend to enemy ? There had been two incidents in recent memory that changed the status quo. The first is the 2015 Nepal blockade . Now one could argue it either way but the truth is that Nepal understood that it is heavily dependent on India hence as any sovereign country would do in its interest it also started courting China for imports so there is some balance. The second one though is one of our own making. On December 16, 2014 RBI allowed Nepali citizens to have cash upto INR 25,000/- . Then in 2016 when demonetization was announced, they said that people could exchange only upto INR 4,500/- which was far below the limit shared above. And btw, before people start blaming just RBI for the decision, FEMA decisions are taken jointly by the finance ministry (FE) as well as ministry of external affairs (MEA) . So without them knowing the decision could not have been taken when announcing it. The result of lowering of demonetization is what made Nepal move more into Chinese hands and this has been shared by number of people in numerous articles in different websites. The wire interview with the vice-chairman of Niti Ayog is pretty interesting. The argument that Nepal show give an estimate of how much old money is there falls flat when in demonetization itself, it was thought of that around 30-40% was black money and would not be returned but by RBI s own admissions all 99.3% of the money was returned. Perhaps they should have consulted Prof. Arun Kumar of JNU who has extensively written and studied the topic before doing that fool-hardy step. It is the reason that since then, an economy which was searing at 9% has been contracting ever since, I could give a dozen articles stating that, but for the moment, just one will suffice. The slowing economy and the sharp divisions between people based on either outlook, religion or whatever also encouraged China to attack us. This year is not good for India. The only thing I hope Indians and people all over do is just maintain physical distances, masks and somehow survive till middle of next year without getting infected when probably most of the vaccine candidates have been trialed, results are in and we have a ready vaccine. I do hope that at least for once, ICMR shares data even after the vaccine is approved, whichever vaccine. Till later.

19 July 2020

Russ Allbery: PGP::Sign 1.01

This is mostly a test-suite fix for my Perl module to automate creation and verification of detached signatures. The 1.00 release of PGP::Sign added support for GnuPG v2 and changed the default to assume that gpg is GnuPG v2, but this isn't the case on some older operating systems (particularly non-Linux ones). That in turn caused chaos for automated testing. This release fixes the test suite to detect when gpg is GnuPG v1 and run the test suite accordingly, trying to exercise as much of the module as possible in that case. It also fixes a few issues found by Perl::Critic::Freenode. You can get the latest release from CPAN or from the PGP::Sign distribution page.

18 July 2020

Chris Lamb: The comedy is over

By now everyone must have seen the versions of comedy shows with the laugh track edited out. The removal of the laughter doesn't just reveal the artificial nature of television and how it conscripts the viewer into laughing along; by subverting key conversational conventions, it reveals some of the myriad and subtle ways humans communicate with one another:
Although the show's conversation is ostensibly between two people, the viewer serves as a silent third actor through which they and therefore we are meant to laugh along with. Then, when this third character is forcibly muted, viewers not only have to endure the stilted gaps, they also sense an uncanny loss of familiarity by losing their 'own' part in the script. A similar phenomenon can be seen in other art forms. In Garfield Minus Garfield, the forced negative spaces that these pauses introduce are discomfiting, almost to the level of performance art:
But when the technique is applied to other TV shows such as The Big Bang Theory, it is unsettling in entirely different ways, exposing the dysfunctional relationships and the adorkable mysogny at the heart of the show:
Once you start to look for it, the ur-elements of the audience, response and timing in the way we communicate are everywhere, from the gaps we leave so that others instinctively know when you have finished speaking, to the myriad of ways you can edit a film. These components are always present, it is only when one of them is taken away that they become more apparent. Today, the small delays added by videoconferencing adds an uncanny awkwardness to many of our everyday interactions too. It is said that "comedy is tragedy plus timing", so it is unsurprising that Zoom's undermining of timing leads, by this simple calculus of human interactions, to feelings of... tragedy.

Leaving aside the usual comments about Pavlovian conditioning and the shows that are the exceptions, complaints against canned laughter are the domain of the pub bore. I will therefore only add two brief remarks. First, rather than being cynically added to artificially inflate the lack of 'real' comedy, laugh tracks were initially added to replicate the live audience of existing shows. In other words, without a laugh track, these new shows might have ironically appeared almost as eerie as the fan edits cited above are today. Secondly, although laugh tracks are described as "false", this is not entirely correct. After all, someone did actually laugh, even if it was for an entirey different joke. In his Simulacra and Simulation, cultural theorist Jean Baudrillard might have poetically identified canned laughter as a "reflection of a profound reality", rather than an outright falsehood. One day, when this laughter becomes entirely algorithmically generated, Baudrillard would describe it as "an order of sorcery", placing it metaphysically on the same level as the entirely pumpkin-free Pumpkin Spiced Latte.

For a variety of reasons I recently decided to try interacting with various social media platforms in new ways. One way of loosening my addiction to this pornography of the amygdala was to hide the number of replies, 'likes' and related numbers:
The effect of installing this extension was immediate. I caught my eyes darting to where the numbers had been and realised I had been subconsciously looking for the input and perhaps even the outright validation of the masses. To be sure, these numbers can be relevant and sometimes useful, but they do implicitly involve delegating part of your responsibility of thinking for yourself to the vox populi, or the Greek chorus of the 21st century. Like many of you reading this, I am sure I told myself that the number of 'likes' has no bearing on whether I should agree with something, but hiding the numbers reveals much of this might have been a convenient fiction; as an entire century of discoveries in behavioural economics has demonstrated, all the pleasingly-satisfying arguments for rational free-market economics stand no chance against our inherent buggy mammalian brains.

Tying a few things together, when attempting to doomscroll through social media without these numbers, I realised that social media without the scorecard of engagement is almost exactly like watching these shows without the laugh track. Without the number of 'retweets', the lazy prompts to remind you exactly when, how and for how much to respond are removed, and replaced with the same stilted silences of those edited scenes from Friends. At times, the existential loneliness of Garfield Minus Garfield creeps in too, and there is more than enough of the dysfunctional, validation-seeking and parasocial 'conversations' of The Big Bang Theory. Most of all, the whole exercise permits a certain level of detached, critical analysis, allowing one to observe that the platforms often feel like a pre-written script with your 'friends' cast as actors, all perpetuated on the heady fumes of rows INSERT-ed into a database on the other side of the world. I'm not quite sure how this will affect my usage of the platforms, and any time spent away from these sites may mean fewer online connections at a time when we all need them the most. But as the Karal Marling, professor at the University of Minnesota wrote about artificial audiences: "Let me be the laugh track."

Reproducible Builds (diffoscope): diffoscope 152 released

The diffoscope maintainers are pleased to announce the release of diffoscope version 152. This version includes the following changes: 
[ Chris Lamb ]
* Bug fixes:
  - Don't require zipnote(1) to determine differences in a .zip file as we
    can use libarchive directly.
* Reporting improvements:
  - Don't emit "javap not found in path" if it is available in the path but
    it did not result in any actual difference.
  - Fix "... not available in path" messages when looking for Java
    decompilers; we were using the Python class name (eg. "<class
    'diffoscope.comparators.java.Javap'>") over the actual command we looked
    for (eg. "javap").
* Code improvements:
  - Replace some simple usages of str.format with f-strings.
  - Tidy inline imports in diffoscope.logging.
  - In the RData comparator, always explicitly return a None value in the
    failure cases as we return a non-None value in the "success" one.
[ Jean-Romain Garnier ]
* Improve output of side-by-side diffs, detecting added lines better.
  (MR: reproducible-builds/diffoscope!64)
* Allow passing file with list of arguments to ArgumentParser (eg.
  "diffoscope @args.txt"). (MR: reproducible-builds/diffoscope!62)
You find out more by visiting the project homepage.

17 July 2020

Sven Hoexter: Debian and exFAT

As some might have noticed we now have Linux 5.7 in Debian/unstable and subsequently the in kernel exFAT implementation created by Samsung available. Thus we now have two exFAT implementations, the exfat fuse driver and the Linux based one. Since some comments and mails I received showed minor confusions, especially around the available tooling, it might help to clarify a bit what is required when. Using the Samsung Linux Kernel Implementation Probably the most common use case is that you just want to use the in kernel implementation. Easy, install a Linux 5.7 kernel package for your architecture and either remove the exfat-fuse package or make sure you've version 1.3.0-2 or later installed. Then you can just run mount /dev/sdX /mnt and everything should be fine. Your result will look something like this: 
$ mount grep sdb
/dev/sdb on /mnt type exfat (rw,relatime,fmask=0022,dmask=0022,iocharset=utf8,errors=remount-ro
In the past this basic mount invocation utilized the mount.exfat helper, which was just a symlink to the helper which is shipped as /sbin/mount.exfat-fuse. The link was dropped from the package in 1.3.0-2. If you're running a not so standard setup, and would like to keep an older version of exfat-fuse installed you must invoke mount -i to prevent mount from loading any helper to mount the filesystem. See also man 8 mount. For those who care, mstone@ and myself had a brief discussion about this issue in #963752, which quickly brought me to the conclusion that it's in the best interest of a majority of users to just drop the symlink from the package. Sticking to the Fuse Driver If you would like to stick to the fuse driver you can of course just do it. I plan to continue to maintain all packages for the moment. Just keep the exfat-fuse package installed and use the mount.exfat-fuse helper directly. E.g. 
$ sudo mount.exfat-fuse /dev/sdb /mnt
FUSE exfat 1.3.0
$ mount grep sdb
/dev/sdb on /mnt type fuseblk (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other,blksize=4096)
In case this is something you would like to make permanent, I would recommend that you create yourself a mount.exfat symlink pointing at the mount.exfat-fuse helper. mkfs and fsck - exfat-utils vs exfatprogs Beside of the filesystem access itself we now also have two implementations of tooling, which support filesystem creation, fscking and label adjustment. The older one is exfat-utils by the creator of the fuse driver, which is also part of Debian since the fuse driver was packaged in 2012. New in Debian is the exfatprogs package written by the Samsung engineers. And here the situation starts to get a bit more complicated. Both packages ship a mkfs.exfat and fsck.exfat tool, so we can not co-install them. In the end both packages declare a conflict with each other at the moment. As outlined in this thread I do not plan to overcomplicate the situation by using the alternatives system. I do feel strongly that this would just create more confusion without a real benefit. Since the tools do not have matching cli options, that could also cause additional issues and confusion. I plan to keep that as is, at least for the bullseye release. Afterwards it's possible, depending on how the usage evolves, to drop the mkfs.exfat and fsck.exfat from exfat-utils, they are in fact again only symlinks. pain point might be tools interfacing with the differing implementations. Currently I see only three reverse depedencies, so that should be manageable to consolidate if required. Last but not least it might be relevant to mention that the exfat-utils package also contains a dumpexfat tool which could be helpful if you're more into forensics, or looking into other lower level analysis of an exFAT filsystem. Thus there is a bit of an interest to have those tools co-installed in some - I would say - niche cases. buster-backports Well if you use buster with a backports kernel you're a bit on your own. In case you want to keep the fuse driver installed, but would still like to mount, e.g. for testing, with the kernel exFAT driver, you must use mount -i. I do not plan any uploads to buster-backports. If you need a mkfs.exfat on buster, I would recommend to just use the one from exfat-utils for now. It has been good enough for the past years, should not get sour before the bullseye release, which ships exfatprogs for you. Kudos My sincere kudos go to:

16 July 2020

Louis-Philippe V ronneau: DebConf Videoteam Sprint Report -- DebConf20@Home

DebConf20 starts in about 5 weeks, and as always, the DebConf Videoteam is working hard to make sure it'll be a success. As such, we held a sprint from July 9th to 13th to work on our new infrastructure. A remote sprint certainly ain't as fun as an in-person one, but we nonetheless managed to enjoy ourselves. Many thanks to those who participated, namely: We also wish to extend our thanks to Thomas Goirand and Infomaniak for providing us with virtual machines to experiment on and host the video infrastructure for DebConf20. Advice for presenters For DebConf20, we strongly encourage presenters to record their talks in advance and send us the resulting video. We understand this is more work, but we think it'll make for a more agreeable conference for everyone. Video conferencing is still pretty wonky and there is nothing worse than a talk ruined by a flaky internet connection or hardware failures. As such, if you are giving a talk at DebConf this year, we are asking you to read and follow our guide on how to record your presentation. Fear not: we are not getting rid of the Q&A period at the end of talks. Attendees will ask their questions either on IRC or on a collaborative pad and the Talkmeister will relay them to the speaker once the pre-recorded video has finished playing. New infrastructure, who dis? Organising a virtual DebConf implies migrating from our battle-tested on-premise workflow to a completely new remote one. One of the major changes this means for us is the addition of Jitsi Meet to our infrastructure. We normally have 3 different video sources in a room: two cameras and a slides grabber. With the new online workflow, directors will be able to play pre-recorded videos as a source, will get a feed from a Jitsi room and will see the audience questions as a third source. This might seem simple at first, but is in fact a very major change to our workflow and required a lot of work to implement. 
               == On-premise ==                                          == Online ==
                                                      
              Camera 1                                                 Jitsi
                                                                          
                 v                 ---> Frontend                         v                 ---> Frontend
                                                                                            
    Slides -> Voctomix -> Backend -+--> Frontend         Questions -> Voctomix -> Backend -+--> Frontend
                                                                                            
                 ^                 ---> Frontend                         ^                 ---> Frontend
                                                                          
              Camera 2                                           Pre-recorded video
In our tests, playing back pre-recorded videos to voctomix worked well, but was sometimes unreliable due to inconsistent encoding settings. Presenters will thus upload their pre-recorded talks to SReview so we can make sure there aren't any obvious errors. Videos will then be re-encoded to ensure a consistent encoding and to normalise audio levels. This process will also let us stitch the Q&As at the end of the pre-recorded videos more easily prior to publication. Reducing the stream latency One of the pitfalls of the streaming infrastructure we have been using since 2016 is high video latency. In a worst case scenario, remote attendees could get up to 45 seconds of latency, making participation in events like BoFs arduous. In preparation for DebConf20, we added a new way to stream our talks: RTMP. Attendees will thus have the option of using either an HLS stream with higher latency or an RTMP stream with lower latency. Here is a comparative table that can help you decide between the two protocols:
HLS RTMP
Pros
  • Can be watched from a browser
  • Auto-selects a stream encoding
  • Single URL to remember
  • Lower latency (~5s)
Cons
  • Higher latency (up to 45s)
  • Requires a dedicated video player (VLC, mpv)
  • Specific URLs for each encoding setting
Live mixing from home with VoctoWeb Since DebConf16, we have been using voctomix, a live video mixer developed by the CCC VOC. voctomix is conveniently divided in two: voctocore is the backend server while voctogui is a GTK+ UI frontend directors can use to live-mix. Although voctogui can connect to a remote server, it was primarily designed to run either on the same machine as voctocore or on the same LAN. Trying to use voctogui from a machine at home to connect to a voctocore running in a datacenter proved unreliable, especially for high-latency and low bandwidth connections. Inspired by the setup FOSDEM uses, we instead decided to go with a web frontend for voctocore. We initially used FOSDEM's code as a proof of concept, but quickly reimplemented it in Python, a language we are more familiar with as a team. Compared to the FOSDEM PHP implementation, voctoweb implements A / B source selection (akin to voctogui) as well as audio control, two very useful features. In the following screen captures, you can see the old PHP UI on the left and the new shiny Python one on the right. The old PHP voctowebThe new Python3 voctoweb Voctoweb is still under development and is likely to change quite a bit until DebConf20. Still, the current version seems to works well enough to be used in production if you ever need to. Python GeoIP redirector We run multiple geographically-distributed streaming frontend servers to minimize the load on our streaming backend and to reduce overall latency. Although users can connect to the frontends directly, we typically point them to live.debconf.org and redirect connections to the nearest server. Sadly, 6 months ago MaxMind decided to change the licence on their GeoLite2 database and left us scrambling. To fix this annoying issue, Stefano Rivera wrote a Python program that uses the new database and reworked our ansible frontend server role. Since the new database cannot be redistributed freely, you'll have to get a (free) license key from MaxMind if you to use this role. Ansible & CI improvements Infrastructure as code is a living process and needs constant care to fix bugs, follow changes in DSL and to implement new features. All that to say a large part of the sprint was spent making our ansible roles and continuous integration setup more reliable, less buggy and more featureful. All in all, we merged 26 separate ansible-related merge request during the sprint! As always, if you are good with ansible and wish to help, we accept merge requests on our ansible repository :)

15 July 2020

Evgeni Golov: Scanning with a Brother MFC-L2720DW on Linux without any binary blobs

Back in 2015, I've got a Brother MFC-L2720DW for the casual "I need to print those two pages" and "I need to scan these receipts" at home (and home-office ;)). It's a rather cheap (I paid less than 200 in 2015) monochrome laser printer, scanner and fax with a (well, two, wired and wireless) network interface. In those five years I've never used the fax or WiFi functions, but printed a scanned a few pages. Brother offers Linux drivers, but those are binary blobs which I never really liked to run. The printer part works just fine with a "Generic PCL 6/PCL XL" driver in CUPS or even "driverless" via AirPrint on Linux. You can also feed it plain PostScript, but I found it rather slow compared to PCL. On recent Androids it works using the built in printer service or Mopria Printer Service for older ones - I used to joke "why would you need a printer on your phone?!", but found it quite useful after a few tries. However, for the scanner part I had to use Brother's brscan4 driver on Linux and their iPrint&Scan app on Android - Mopria Scan wouldn't support it. Until, last Friday, I've seen a NEW package being uploaded to Debian: sane-airscan. And yes, monitoring the Debian NEW queue via Twitter is totally legit! sane-airscan is an implementation of Apple's AirScan (eSCL) and Microsoft's WSD/WS-Scan protocols for SANE. I've never heard of those before - only about AirPrint, but thankfully this does not mean nobody has reverse-engineered them and created something that works beautifully on Linux. As of today there are no packages in the official Fedora repositories and the Debian ones are still in NEW, however the upstream documentation refers to an openSUSE OBS repository that works like a charm in the meantime (on Fedora 32). The only drawback I've seen so far: the scanner only works in "Color" mode and there is no way to scan in "Grayscale", making scanning a tad slower. This has been reported upstream and might or might not be fixable, as it seems the device does not announce any mode besides "Color". Interestingly, SANE has an eSCL backend on its own since 1.0.29, but it's disabled in Fedora in favor of sane-airscan even though the later isn't available in Fedora yet. However, it might not even need separate packaging, as SANE upstream is planning to integrate it into sane-backends directly.

Utkarsh Gupta: GSoC Phase 2

Hello, In early May, I got selected as a Google Summer of Code student for Debian to work on a project which is to write a linter (an extension to RuboCop).
This tool is mostly to help the Debian Ruby team. And that is the best part, I love working in/for/with the Ruby team!
(I ve been an active part of the team for 19 months now :)) More details about the project can be found here, on the wiki.
And also, I have got the best mentors I could ve possibly asked for: Antonio Terceiro and David Rodr guez So, the program began on 1st June and I ve been working since then. I log my daily updates at gsocwithutkarsh2102.tk.
The blog for the first part of phase 1 can be found here and that of second part of phase 1 can be found here. Whilst the daily updates are available at the above site^, I ll breakdown the important parts here: Well, the best part yet?
rubocop-packaging is being used by batalert, arbre, rspec-stubbed_env, rspec-pending_for, ISO8601, get_root ( ), gir_ffi, linter, and cucumber-rails. Whilst it has been a lot of fun so far, my plate has started to almost overflow. It seems that I ve got a lot of things to work on (and already things that are due!).
From my major project, college *stuff to my GSoC project, Debian (E)LTS, and a lot *more. Thanks to Antonio for helping me out with *other things (which maps back to his sayings in Paris \o/).
Until next time.
:wq for today.

14 July 2020

Ian Jackson: MessagePack vs CBOR (RFC7049)

tl;dr: Use MessagePack, rather than CBOR. Introduction I recently wanted to choose a binary encoding. This was for a project using Rust serde, so I looked at the list of formats there. I ended up reading about CBOR and MessagePack. Both of these are binary formats for a JSON-like data model. Both of them are "schemaless", meaning you can decode them without knowing the structure. (This also provides some forwards compatibility.) They are, in fact, quite similar (although they are totally incompatible). This is no accident: CBOR is, effectively, a fork of MessagePack. Both formats continue to exist and both are being used in new programs. I needed to make a choice but lacked enough information. I thought I would try to examine the reasons and nature of the split, and to make some kind of judgement about the situation. So I did a lot of reading [11]. Here are my conclusions. History and politics Between about 2010 and 2013 there was only MessagePack. Unfortunately, MessagePack had some problems. The biggest of these was that it lacked a separate string type. Strings were to be encoded simply as byte blocks. This caused serious problems for many MessagePack library implementors: for example, when decoding a MessagePack file the Python library wouldn't know whether to produce a Python bytes object, or a string. Straightforward data structures wouldn't round trip through MessagePack. [1] [2] It seems that in late 2012 this came to the attention to someone with an IETF background. According to them, after unsatisfactory conversations with MessagePack upstream, they decided they would have to fork. They submitted an Internet Draft for a partially-incompatible protocol [3] [4]. Little seemed to happen in the IETF until soon before the Orlando in-person IETF meeting in February 2013.[5] These conversations sparked some discussion in the MessagePack issue tracker. There were long threads including about process [1,2,4 ibid]. But there was also a useful technical discussion, about proposed backward compatible improves to the MessagePack spec.[5] The prominent IETF contributor provided some helpful input in these discussions in the MessagePack community - but also pushed quite hard for a "tagging" system, which suggestion was not accepted (see my technical analysis, below). An improved MessagePack spec resulted, with string support, developed largely by the MessagePack community. It seems to have been available in useable form since mid-2013 and was officially published as canonical in August 2013. Meanwhile a parallel process was pursued in the IETF, based on the IETF contributor's fork, with 11 Internet-Drafts from February[7] to September[8]. This seems to have continued even though the original technical reason for the fork - lack of string vs binary distinction - no longer applied. The IETF proponent expressed unhappiness about MessagePack's stewardship and process as much as they did about the technical details [4, ibid]. The IETF process culminated in the CBOR RFC[9]. The discussion on process questions between the IETF proponent and MessagePack upstream, in the MessagePack issue tracker [4, ibid] should make uncomfortable reading for IETF members. The IETF acceptance of CBOR despite clear and fundamental objections from MessagePack upstream[13] and indeed other respected IETF members[14], does not reflect well on the IETF. The much vaunted openness of the IETF process seems to have been rather one-sided. The IETF proponent here was an IETF Chair. Certainly the CBOR author was very well-spoken and constantly talks about politeness and cooperation and process; but what they actually did was very hostile. They accused the MessagePack community of an "us and them" attitude while simultaneously pursuing a forked specification! The CBOR RFC does mention MessagePack in Appendix E.2. But not to acknowledge that CBOR was inspired by MessagePack. Rather, it does so to make a set of tendentious criticisms of MessagePack. Perhaps these criticisms were true when they were first written in an I-D but they were certainly false by the time the RFC was actually published, which occurred after the MessagePack improvement process was completely concluded, with a formal spec issued. Since then both formats have existed in parallel. Occasionally people discuss which one is better, and sometimes it is alleged that "yes CBOR is the successor to MessagePack", which is not really fair.[9][10] Technical differences The two formats have a similar arrangement: initial byte which can encode small integers, or type and length, or type and specify a longer length encoding. But there are important differences. Overall, MessagePack is very significantly simpler. Floating point CBOR supports five floating point formats! Not only three sizes of IEEE754, but also decimal floating point, and bigfloats. This seems astonishing for a supposedly-simple format. (Some of these are supported via the semi-optional tag mechanism - see below.) Indefinite strings and arrays Like MessagePack, CBOR mostly precedes items with their length. But CBOR also supports "indefinite" strings, arrays, and so on, where the length is not specified at the beginning. The object (array, string, whatever) is terminated by a special "break" item. This seems to me to be a mistake. If you wanted the kind of application where MessagePack or CBOR would be useful, streaming sub-objects of unknown length is not that important. This possibility considerably complicates decoders. CBOR tagging system CBOR has a second layer of sort-of-type which can be attached to each data item. The set of possible tags is open-ended and extensible, but the CBOR spec itself gives tag values for: two kinds of date format; positive and negative bignums; decimal floats (see above); binary but expected to be encoded if converted to JSON (in base64url, base64, or base16); nestedly encoded CBOR; URIs; base64 data (two formats); regexps; MIME messages; and a special tag to make file(1) work. In practice it is not clear how many of these are used, but a decoder must be prepared to at least discard them. The amount of additional spec complexity here is quite astonishing. IMO binary formats like this will (just like JSON) be used by a next layer which always has an idea of what the data means, including (where the data is a binary blob) what encoding it is in etc. So these tags are not useful. These tags might look like a middle way between (i) extending the binary protocol with a whole new type such as an extension type (incompatible with old readers) and encoding your new kind data in a existing type (leaving all readers who don't know the schema to print it as just integers or bytes or string). But I think they are more trouble than they are worth. The tags are uncomfortably similar to the ASN.1 tag system, which is widely regarded as one of ASN.1's unfortunate complexities. MessagePack extension mechanism MessagePack explicitly reserves some encoding space for users and for future extensions: there is an "extension type". The payload is an extension type byte plus some more data bytes; the data bytes are in a format to be defined by the extension type byte. Half of the possible extension byte values are reserved for future specification, and half are designated for application use. This is pleasingly straightforward. (There is also one unused primary initial byte value, but that would be rejected by existing decoders and doesn't seem like a likely direction for future expansion.) Minor other differences in integer encoding The encodings of integers differ. In MessagePack, signed and unsigned integers have different typecodes. In CBOR, signed and unsigned positive integers have the same typecodes; negative integers have a different set of typecodes. This means that a CBOR reader which knows it is expecting a signed value will have to do a top-bit-set check on the actual data value! And a CBOR writer must check the value to choose a typecode. MessagePack reserves fewer shortcodes for small negative integers, than for small positive integers. Conclusions and lessons MessagePack seems to have been prompted into fixing the missing string type problem, but only by the threat of a fork. However, this fork went ahead even after MessagePack clearly accepted the need for a string type. MessagePack had a fixed protocol spec before the IETF did. The continued pursuit of the IETF fork was ostensibly been motivated by a disapproval of the development process and in particular a sense that the IETF process was superior. However, it seems to me that the IETF process was abused by CBOR's proponent, who just wanted things their own way. I have seen claims by IETF proponents that the open decisionmaking system inherently produces superior results. However, in this case the IETF process produced a bad specification. To the extent that other IETF contributors had influence over the ultimate CBOR RFC, I don't think they significantly improved it. CBOR has been described as MessagePack bikeshedded by the IETF. That would have been bad enough, but I think it's worse than that. To a large extent CBOR is one person's NIH-induced bad design rubber stamped by the IETF. CBOR's problems are not simply matters of taste: it's significantly overcomplicated. One lesson for the rest of us is that although being the upstream and nominally in charge of a project seems to give us a lot of power, it's wise to listen carefully to one's users and downstreams. Once people are annoyed enough to fork, the fork will have a life of its own. Another lesson is that many of us should be much warier of the supposed moral authority of the IETF. Many IETF standards are awful (Oauth 2 [12]; IKE; DNSSEC; the list goes on). Sometimes (especially when network adoption effects are weak, as with MessagePack vs CBOR) better results can be obtained from a smaller group, or even an individual, who simply need the thing for their own uses. Finally, governance systems of public institutions like the IETF need to be robust in defending the interests of outsiders (and hence of society at large) against eloquent insiders who know how to work the process machinery. Any institution which nominally serves the public good faces a constant risk of devolving into self-servingness. This risk gets worse the more powerful and respected the institution becomes. References
  1. #13: First-class string type in serialization specification (MessagePack issue tracker, June 2010 - August 2013)
  2. #121: Msgpack can't differentiate between raw binary data and text strings (MessagePack issue tracker, November 2012 - February 2013)
  3. draft-bormann-apparea-bpack-00: The binarypack JSON-like representation format (IETF Internet-Draft, October 2012)
  4. #129: MessagePack should be developed in an open process (MessagePack issue tracker, February 2013 - March 2013)
  5. Re: JSON mailing list and BoF (IETF apps-discuss mailing list message from Carsten Bormann, 18 February 2013)
  6. #128: Discussions on the upcoming MessagePack spec that adds the string type to the protocol (MessagePack issue tracker, February 2013 - August 2013)
  7. draft-bormann-apparea-bpack-01: The binarypack JSON-like representation format (IETF Internet-Draft, February 2013)
  8. draft-bormann-cbor: Concise Binary Object Representation (CBOR) (IETF Internet-Drafts, May 2013 - September 2013)
  9. RFC 7049: Concise Binary Object Representation (CBOR) (October 2013)
  10. "MessagePack should be replaced with [CBOR] everywhere ..." (floatboth on Hacker News, 8th April 2017)
  11. Discussion with very useful set of history links (camgunz on Hacker News, 9th April 2017)
  12. OAuth 2.0 and the Road to Hell (Eran Hammer, blog posting from 2012, via Wayback Machine)
  13. Re: [apps-discuss] [Json] msgpack/binarypack (Re: JSON mailing list and BoF) (IETF list message from Sadyuki Furuhashi, 4th March 2013)
  14. "no apologies for complaining about this farce" (IETF list message from Phillip Hallam-Baker, 15th August 2013)
    Edited 2020-07-14 18:55 to fix a minor formatting issue, and 2020-07-14 22:54 to fix two typos


comment count unavailable comments

13 July 2020

Antoine Beaupr : Not recommending Purism

This is just a quick note to mention that I have updated my hardware documentation on the Librem 13v4 laptop. It has unfortunately turned into a rather lengthy (and ranty) piece about Purism. Let's just say that waiting weeks for your replacement laptop (yes, it died again) does wonders for creativity. To quote the full review:
TL;DR: I recommend people avoid the Purism brand and products. I find they have questionable politics, operate in a "libre-washing" fashion, and produce unreliable hardware. Will not buy again.
People who have read the article might want to jump directly to the new sections: I have also added the minor section of the missing mic jack. I realize that some folks (particularly at Debian) might still work at Purism, and that this article might be demoralizing for their work. If that is the case, I am sorry this article triggered you in any way and I hope this can act as a disclaimer. But I feel it is my duty to document the issues I am going through, as a user, and to call bullshit when I see it (let's face it, the anti-interdiction stuff and the Purism 5 crowd-funding campaign were total bullshit). I also understand that the pandemic makes life hard for everyone, and probably makes a bad situation at Purism worse. But those problems existed before the pandemic happened. They were issues I had identified in 2019 and that I simply never got around to document. I wish that people wishing to support the free software movement would spend their energy towards organisations that actually do honest work in that direction, like System76 and Pine64. And if you're going to go crazy with an experimental free hardware design, why not go retro with the MNT Reform. In the meantime, if you're looking for a phone, I recommend you give the Fairphone a fair chance. It really is a "fair" (as in, not the best, but okay) phone that you can moderately liberate, and it actually frigging works. See also my hardware review of the FP2. Update: this kind of blew up, for my standards: 10k visitors in ~24h while I usually get about 1k visitors after a week on any regular blog post. There were more discussions on the subject here: Trigger warning: some of those threads include personal insults and explicitly venture into the free speech discussion, with predictable (sad) consequences...

12 July 2020

Antoine Beaupr : On contact tracing apps

I have strong doubts about the efficiency of any tracing app of the sort, and even less in the context where it is unlikely that a majority of the population will use it. There's also the problem that this app would need to work on Apple phones, or be incompatible with them, and cause significant "fracture" between those who have access to technology, and those who haven't. See this text for more details. Such an app would be a security and privacy liability at no benefit to public health. There are better options, see for this research on hardware tokens. But I doubt any contact tracing app or hardware will actually work anyways. I am a computer engineer with more than 20 years of experience in the domain, and I have been following this question closely. Please don't do this.
I wrote the above in a response to the Qu bec government's survey about a possible tracing app. Update: a previous version of this article was titled plainly "on contact tracing". In case that was not obvious, I definitely do not object to contact tracing per se. I believe it's a fundamental, critical, and important part of fighting the epidemic and I think we should do it. I do not believe any engineer has found a proper way of doing it with "apps" so far, but I do not deny the utility and importance of "contact tracing" itself. Apologies for the confusion.
Pour une raison que je m'explique mal, le sondage m' t envoy en anglais, et j'ai donc crit ma r ponse dans la langue de Shakespeare au lieu de celle de moli re... Je serai heureux de fournir une traduction fran aise ceux ou celles qui en ont besoin...

Next.

Previous.