Search Results: "rul"

9 August 2021

Russ Allbery: Review: The Last Battle

Review: The Last Battle, by C.S. Lewis
Illustrator: Pauline Baynes
Series: Chronicles of Narnia #7
Publisher: Collier Books
Copyright: 1956
Printing: 1978
ISBN: 0-02-044210-6
Format: Mass market
Pages: 184
The Last Battle is the seventh and final book of the Chronicles of Narnia in every reading order. It ties together (and spoils) every previous Narnia book, so you do indeed want to read it last (or skip it entirely, but I'll get into that). In the far west of Narnia, beyond the Lantern Waste and near the great waterfall that marks Narnia's western boundary, live a talking ape named Shift and a talking donkey named Puzzle. Shift is a narcissistic asshole who has been gaslighting and manipulating Puzzle for years, convincing the poor donkey that he's stupid and useless for anything other than being Shift's servant. At the start of the book, a lion skin washes over the waterfall and into the Cauldron Pool. Shift, seeing a great opportunity, convinces Puzzle to retrieve it. The king of Narnia at this time is Tirian. I would tell you more about Tirian except, despite being the protagonist, that's about all the characterization he gets. He's the king, he's broad-shouldered and strong, he behaves in a correct kingly fashion by preferring hunting lodges and simple camps to the capital at Cair Paravel, and his close companion is a unicorn named Jewel. Other than that, he's another character like Rilian from The Silver Chair who feels like he was taken from a medieval Arthurian story. (Thankfully, unlike Rilian, he doesn't talk like he's in a medieval Arthurian story.) Tirian finds out about Shift's scheme when a dryad appears at Tirian's camp, calling for justice for the trees of Lantern Waste who are being felled. Tirian rushes to investigate and stop this monstrous act, only to find the beasts of Narnia cutting down trees and hauling them away for Calormene overseers. When challenged on why they would do such a thing, they reply that it's at Aslan's orders. The Last Battle is largely the reason why I decided to do this re-read and review series. It is, let me be clear, a bad book. The plot is absurd, insulting to the characters, and in places actively offensive. It is also, unlike the rest of the Narnia series, dark and depressing for nearly all of the book. The theology suffers from problems faced by modern literature that tries to use the Book of Revelation and related Christian mythology as a basis. And it is, most famously, the site of one of the most notorious authorial betrayals of a character in fiction. And yet, The Last Battle, probably more than any other single book, taught me to be a better human being. It contains two very specific pieces of theology that I would now critique in multiple ways but which were exactly the pieces of theology that I needed to hear when I first understood them. This book steered me away from a closed, judgmental, and condemnatory mindset at exactly the age when I needed something to do that. For that, I will always have a warm spot in my heart for it. I'm going to start with the bad parts, though, because that's how the book starts. MAJOR SPOILERS BELOW. First, and most seriously, this is a second-order idiot plot. Shift shows up with a donkey wearing a lion skin (badly), only lets anyone see him via firelight, claims he's Aslan, and starts ordering the talking animals of Narnia to completely betray their laws and moral principles and reverse every long-standing political position of the country... and everyone just nods and goes along with this. This is the most blatant example of a long-standing problem in this series: Lewis does not respect his animal characters. They are the best feature of his world, and he treats them as barely more intelligent than their non-speaking equivalents and in need of humans to tell them what to do. Furthermore, despite the assertion of the narrator, Shift is not even close to clever. His deception has all the subtlety of a five-year-old who doesn't want to go to bed, and he offers the Narnians absolutely nothing in exchange for betraying their principles. I can forgive Puzzle for going along with the scheme since Puzzle has been so emotionally abused that he doesn't know what else to do, but no one else has any excuse, especially Shift's neighbors. Given his behavior in the book, everyone within a ten mile radius would be so sick of his whining, bullying, and lying within a month that they'd never believe anything he said again. Rishda and Ginger, a Calormene captain and a sociopathic cat who later take over Shift's scheme, do qualify as clever, but there's no realistic way Shift's plot would have gotten far enough for them to get involved. The things that Shift gets the Narnians to do are awful. This is by far the most depressing book in the series, even more than the worst parts of The Silver Chair. I'm sure I'm not the only one who struggled to read through the first part of this book, and raced through it on re-reads because everything is so hard to watch. The destruction is wanton and purposeless, and the frequent warnings from both characters and narration that these are the last days of Narnia add to the despair. Lewis takes all the beautiful things that he built over six books and smashes them before your eyes. It's a lot to take, given that previous books would have treated the felling of a single tree as an unspeakable catastrophe. I think some of these problems are due to the difficulty of using Christian eschatology in a modern novel. An antichrist is obligatory, but the animals of Narnia have no reason to follow an antichrist given their direct experience with Aslan, particularly not the aloof one that Shift tries to give them. Lewis forces the plot by making everyone act stupidly and out of character. Similarly, Christian eschatology says everything must become as awful as possible right before the return of Christ, hence the difficult-to-read sections of Narnia's destruction, but there's no in-book reason for the Narnians' complicity in that destruction. One can argue about whether this is good theology, but it's certainly bad storytelling. I can see the outlines of the moral points Lewis is trying to make about greed and rapacity, abuse of the natural world, dubious alliances, cynicism, and ill-chosen prophets, but because there is no explicable reason for Tirian's quiet kingdom to suddenly turn to murderous resource exploitation, none of those moral points land with any force. The best moral apocalypse shows the reader how, were they living through it, they would be complicit in the devastation as well. Lewis does none of that work, so the reader is just left angry and confused. The book also has several smaller poor authorial choices, such as the blackface incident. Tirian, Jill, and Eustace need to infiltrate Shift's camp, and use blackface to disguise themselves as Calormenes. That alone uncomfortably reveals how much skin tone determines nationality in this world, but Lewis makes it far worse by having Tirian comment that he "feel[s] a true man again" after removing the blackface and switching to Narnian clothes. All of this drags on and on, unlike Lewis's normally tighter pacing, to the point that I remembered this book being twice the length of any other Narnia book. It's not; it's about the same length as the rest, but it's such a grind that it feels interminable. The sum total of the bright points of the first two-thirds of the book are the arrival of Jill and Eustace, Jill's one moment of true heroism, and the loyalty of a single Dwarf. The rest is all horror and betrayal and doomed battles and abject stupidity. I do, though, have to describe Jill's moment of glory, since I complained about her and Eustace throughout The Silver Chair. Eustace is still useless, but Jill learned forestcraft during her previous adventures (not that we saw much sign of this previously) and slips through the forest like a ghost to steal Puzzle and his lion costume out from the under the nose of the villains. Even better, she finds Puzzle and the lion costume hilarious, which is the one moment in the book where one of the characters seems to understand how absurd and ridiculous this all is. I loved Jill so much in that moment that it makes up for all of the pointless bickering of The Silver Chair. She doesn't get to do much else in this book, but I wish the Jill who shows up in The Last Battle had gotten her own book. The end of this book, and the only reason why it's worth reading, happens once the heroes are forced into the stable that Shift and his co-conspirators have been using as the stage for their fake Aslan. Its door (for no well-explained reason) has become a door to Aslan's Country and leads to a reunion with all the protagonists of the series. It also becomes the frame of Aslan's final destruction of Narnia and judging of its inhabitants, which I suspect would be confusing if you didn't already know something about Christian eschatology. But before that, this happens, which is sufficiently and deservedly notorious that I think it needs to be quoted in full.
"Sir," said Tirian, when he had greeted all these. "If I have read the chronicle aright, there should be another. Has not your Majesty two sisters? Where is Queen Susan?" "My sister Susan," answered Peter shortly and gravely, "is no longer a friend of Narnia." "Yes," said Eustace, "and whenever you've tried to get her to come and talk about Narnia or do anything about Narnia, she says 'What wonderful memories you have! Fancy your still thinking about all those funny games we used to play when we were children.'" "Oh Susan!" said Jill. "She's interested in nothing nowadays except nylons and lipstick and invitations. She always was a jolly sight too keen on being grown-up." "Grown-up indeed," said the Lady Polly. "I wish she would grow up. She wasted all her school time wanting to be the age she is now, and she'll waste all the rest of her life trying to stay that age. Her whole idea is to race on to the silliest time of one's life as quick as she can and then stop there as long as she can."
There are so many obvious and dire problems with this passage, and so many others have written about it at length, that I will only add a few points. First, I find it interesting that neither Lucy nor Edmund says a thing. (I would like to think that Edmund knows better.) The real criticism comes from three characters who never interacted with Susan in the series: the two characters introduced after she was no longer allowed to return to Narnia, and a character from the story that predated hers. (And Eustace certainly has some gall to criticize someone else for treating Narnia as a childish game.) It also doesn't say anything good about Lewis that he puts his rather sexist attack on Susan into the mouths of two other female characters. Polly's criticism is a somewhat generic attack on puberty that could arguably apply to either sex (although "silliness" is usually reserved for women), but Jill makes the attack explicitly gendered. It's the attack of a girl who wants to be one of the boys on a girl who embraces things that are coded feminine, and there's a whole lot of politics around the construction of gender happening here that Lewis is blindly reinforcing and not grappling with at all. Plus, this is only barely supported by single sentences in The Voyage of the Dawn Treader and The Horse and His Boy and directly contradicts the earlier books. We're expected to believe that Susan the archer, the best swimmer, the most sensible and thoughtful of the four kids has abruptly changed her whole personality. Lewis could have made me believe Susan had soured on Narnia after the attempted kidnapping (and, although left unstated, presumably eventual attempted rape) in The Horse and His Boy, if one ignores the fact that incident supposedly happens before Prince Caspian where there is no sign of such a reaction. But not for those reasons, and not in that way. Thankfully, after this, the book gets better, starting with the Dwarfs, which is one of the two passages that had a profound influence on me. Except for one Dwarf who allied with Tirian, the Dwarfs reacted to the exposure of Shift's lies by disbelieving both Tirian and Shift, calling a pox on both their houses, and deciding to make their own side. During the last fight in front of the stable, they started killing whichever side looked like they were winning. (Although this is horrific in the story, I think this is accurate social commentary on a certain type of cynicism, even if I suspect Lewis may have been aiming it at atheists.) Eventually, they're thrown through the stable door by the Calormenes. However, rather than seeing the land of beauty and plenty that everyone else sees, they are firmly convinced they're in a dark, musty stable surrounded by refuse and dirty straw. This is, quite explicitly, not something imposed on them. Lucy rebukes Eustace for wishing Tash had killed them, and tries to make friends with them. Aslan tries to show them how wrong their perceptions are, to no avail. Their unwillingness to admit they were wrong is so strong that they make themselves believe that everything is worse than it actually is.
"You see," said Aslan. "They will not let us help them. They have chosen cunning instead of belief. Their prison is only in their own minds, yet they are in that prison; and so afraid of being taken in that they cannot be taken out."
I grew up with the US evangelical version of Hell as a place of eternal torment, which in turn was used to justify religious atrocities in the name of saving people from Hell. But there is no Hell of that type in this book. There is a shadow into which many evil characters simply disappear, and there's this passage. Reading this was the first time I understood the alternative idea of Hell as the absence of God instead of active divine punishment. Lewis doesn't use the word "Hell," but it's obvious from context that the Dwarfs are in Hell. But it's not something Aslan does to them and no one wants them there; they could leave any time they wanted, but they're too unwilling to be wrong. You may have to be raised in conservative Christianity to understand how profoundly this rethinking of Hell (which Lewis tackles at greater length in The Great Divorce) undermines the system of guilt and fear that's used as motivation and control. It took me several re-readings and a lot of thinking about this passage, but this is where I stopped believing in a vengeful God who will eternally torture nonbelievers, and thus stopped believing in all of the other theology that goes with it. The second passage that changed me is Emeth's story. Emeth is a devout Calormene, a follower of Tash, who volunteered to enter the stable when Shift and his co-conspirators were claiming Aslan/Tash was inside. Some time after going through, he encounters Aslan, and this is part of his telling of that story (and yes, Lewis still has Calormenes telling stories as if they were British translators of the Arabian Nights):
[...] Lord, is it then true, as the Ape said, that thou and Tash are one? The Lion growled so that the earth shook (but his wrath was not against me) and said, It is false. Not because he and I are one, but because we are opposites, I take to me the services which thou hast done to him. For I and he are of such different kinds that no service which is vile can be done to me, and none which is not vile can be done to him. Therefore if any man swear by Tash and keep his oath for the oath's sake, it is by me that he has truly sworn, though he know it not, and it is I who reward him. And if any man do a cruelty in my name, then, though he says the name Aslan, it is Tash whom he serves and by Tash his deed is accepted. Dost thou understand, Child? I said, Lord, thou knowest how much I understand. But I said also (for the truth constrained me), Yet I have been seeking Tash all my days. Beloved, said the Glorious One, unless thy desire had been for me, thou wouldst not have sought so long and so truly. For all find what they truly seek.
So, first, don't ever say this to anyone. It's horribly condescending and, since it's normally said by white Christians to other people, usually explicitly colonialist. Telling someone that their god is evil but since they seem to be a good person they're truly worshiping your god is only barely better than saying yours is the only true religion. But it is better, and as someone who, at the time, was wholly steeped in the belief that only Christians were saved and every follower of another religion was following Satan and was damned to Hell, this passage blew my mind. This was the first place I encountered the idea that someone who followed a different religion could be saved, or that God could transcend religion, and it came with exactly the context and justification that I needed given how close-minded I was at the time. Today, I would say that the Christian side of this analysis needs far more humility, and fobbing off all the evil done in the name of the Christian God by saying "oh, those people were really following Satan" is a total moral copout. But, nonetheless, Lewis opened a door for me that I was able to step through and move beyond to a less judgmental, dismissive, and hostile view of others. There's not much else in the book after this. It's mostly Lewis's charmingly Platonic view of the afterlife, in which the characters go inward and upward to truer and more complete versions of both Narnia and England and are reunited (very briefly) with every character of the series. Lewis knows not to try too hard to describe the indescribable, but it remains one of my favorite visions of an afterlife because it makes so explicit that this world is neither static or the last, but only the beginning of a new adventure. This final section of The Last Battle is deeply flawed, rather arrogant, a little bizarre, and involves more lectures on theology than precise description, but I still love it. By itself, it's not a bad ending for the series, although I don't think it has half the beauty or wonder of the end of The Voyage of the Dawn Treader. It's a shame about the rest of the book, and it's a worse shame that Lewis chose to sacrifice Susan on the altar of his prejudices. Those problems made it very hard to read this book again and make it impossible to recommend. Thankfully, you can read the series without it, and perhaps most readers would be better off imagining their own ending (or lack of ending) to Narnia than the one Lewis chose to give it. But the one redeeming quality The Last Battle will always have for me is that, despite all of its flaws, it was exactly the book that I needed to read when I read it. Rating: 4 out of 10

2 August 2021

Colin Watson: Launchpad now runs on Python 3!

After a very long porting journey, Launchpad is finally running on Python 3 across all of our systems. I wanted to take a bit of time to reflect on why my emotional responses to this port differ so much from those of some others who ve done large ports, such as the Mercurial maintainers. It s hard to deny that we ve had to burn a lot of time on this, which I m sure has had an opportunity cost, and from one point of view it s essentially running to stand still: there is no single compelling feature that we get solely by porting to Python 3, although it s clearly a prerequisite for tidying up old compatibility code and being able to use modern language facilities in the future. And yet, on the whole, I found this a rewarding project and enjoyed doing it. Some of this may be because by inclination I m a maintenance programmer and actually enjoy this sort of thing. My default view tends to be that software version upgrades may be a pain but it s much better to get that pain over with as soon as you can rather than trying to hold back the tide; you can certainly get involved and try to shape where things end up, but rightly or wrongly I can t think of many cases when a righteously indignant user base managed to arrange for the old version to be maintained in perpetuity so that they never had to deal with the new thing (OK, maybe Perl 5 counts here). I think a more compelling difference between Launchpad and Mercurial, though, may be that very few other people really had a vested interest in what Python version Launchpad happened to be running, because it s all server-side code (aside from some client libraries such as launchpadlib, which were ported years ago). As such, we weren t trying to do this with the internet having Strong Opinions at us. We were doing this because it was obviously the only long-term-maintainable path forward, and in more recent times because some of our library dependencies were starting to drop support for Python 2 and so it was obviously going to become a practical problem for us sooner or later; but if we d just stayed on Python 2 forever then fundamentally hardly anyone else would really have cared directly, only maybe about some indirect consequences of that. I don t follow Mercurial development so I may be entirely off-base, but if other people were yelling at me about how late my project was to finish its port, that in itself would make me feel more negatively about the project even if I thought it was a good idea. Having most of the pressure come from ourselves rather than from outside meant that wasn t an issue for us. I m somewhat inclined to think of the process as an extreme version of paying down technical debt. Moving from Python 2.7 to 3.5, as we just did, means skipping over multiple language versions in one go, and if similar changes had been made more gradually it would probably have felt a lot more like the typical dependency update treadmill. I appreciate why not everyone might want to think of it this way: maybe this is just my own rationalization. Reflections on porting to Python 3 I m not going to defend the Python 3 migration process; it was pretty rough in a lot of ways. Nor am I going to spend much effort relitigating it here, as it s already been done to death elsewhere, and as I understand it the core Python developers have got the message loud and clear by now. At a bare minimum, a lot of valuable time was lost early in Python 3 s lifetime hanging on to flag-day-type porting strategies that were impractical for large projects, when it should have been providing for bilingual strategies (code that runs in both Python 2 and 3 for a transitional period) which is where most libraries and most large migrations ended up in practice. For instance, the early advice to library maintainers to maintain two parallel versions or perhaps translate dynamically with 2to3 was entirely impractical in most non-trivial cases and wasn t what most people ended up doing, and yet the idea that 2to3 is all you need still floats around Stack Overflow and the like as a result. (These days, I would probably point people towards something more like Eevee s porting FAQ as somewhere to start.) There are various fairly straightforward things that people often suggest could have been done to smooth the path, and I largely agree: not removing the u'' string prefix only to put it back in 3.3, fewer gratuitous compatibility breaks in the name of tidiness, and so on. But if I had a time machine, the number one thing I would ask to have been done differently would be introducing type annotations in Python 2 before Python 3 branched off. It s true that it s technically possible to do type annotations in Python 2, but the fact that it s a different syntax that would have to be fixed later is offputting, and in practice it wasn t widely used in Python 2 code. To make a significant difference to the ease of porting, annotations would need to have been introduced early enough that lots of Python 2 library code used them so that porting code didn t have to be quite so much of an exercise of manually figuring out the exact nature of string types from context. Launchpad is a complex piece of software that interacts with multiple domains: for example, it deals with a database, HTTP, web page rendering, Debian-format archive publishing, and multiple revision control systems, and there s often overlap between domains. Each of these tends to imply different kinds of string handling. Web page rendering is normally done mainly in Unicode, converting to bytes as late as possible; revision control systems normally want to spend most of their time working with bytes, although the exact details vary; HTTP is of course bytes on the wire, but Python s WSGI interface has some string type subtleties. In practice I found myself thinking about at least four string-like types (that is, things that in a language with a stricter type system I might well want to define as distinct types and restrict conversion between them): bytes, text, ordinary native strings (str in either language, encoded to UTF-8 in Python 2), and native strings with WSGI s encoding rules. Some of these are emergent properties of writing in the intersection of Python 2 and 3, which is effectively a specialized language of its own without coherent official documentation whose users must intuit its behaviour by comparing multiple sources of information, or by referring to unofficial porting guides: not a very satisfactory situation. Fortunately much of the complexity collapses once it becomes possible to write solely in Python 3. Some of the difficulties we ran into are not ones that are typically thought of as Python 2-to-3 porting issues, because they were changed later in Python 3 s development process. For instance, the email module was substantially improved in around the 3.2/3.3 timeframe to handle Python 3 s bytes/text model more correctly, and since Launchpad sends quite a few different kinds of email messages and has some quite picky tests for exactly what it emits, this entailed a lot of work in our email sending code and in our test suite to account for that. (It took me a while to work out whether we should be treating raw email messages as bytes or as text; bytes turned out to work best.) 3.4 made some tweaks to the implementation of quoted-printable encoding that broke a number of our tests in ways that took some effort to fix, because the tests needed to work on both 2.7 and 3.5. The list goes on. I got quite proficient at digging through Python s git history to figure out when and why some particular bit of behaviour had changed. One of the thorniest problems was parsing HTTP form data. We mainly rely on zope.publisher for this, which in turn relied on cgi.FieldStorage; but cgi.FieldStorage is badly broken in some situations on Python 3. Even if that bug were fixed in a more recent version of Python, we can t easily use anything newer than 3.5 for the first stage of our port due to the version of the base OS we re currently running, so it wouldn t help much. In the end I fixed some minor issues in the multipart module (and was kindly given co-maintenance of it) and converted zope.publisher to use it. Although this took a while to sort out, it seems to have gone very well. A couple of other interesting late-arriving issues were around pickle. For most things we normally prefer safer formats such as JSON, but there are a few cases where we use pickle, particularly for our session databases. One of my colleagues pointed out that I needed to remember to tell pickle to stick to protocol 2, so that we d be able to switch back and forward between Python 2 and 3 for a while; quite right, and we later ran into a similar problem with marshal too. A more surprising problem was that datetime.datetime objects pickled on Python 2 require special care when unpickling on Python 3; rather than the approach that ended up being implemented and documented for Python 3.6, though, I preferred a custom unpickler, both so that things would work on Python 3.5 and so that I wouldn t have to risk affecting the decoding of other pickled strings in the session database. General lessons Writing this over a year after Python 2 s end-of-life date, and certainly nowhere near the leading edge of Python 3 porting work, it s perhaps more useful to look at this in terms of the lessons it has for other large technical debt projects. I mentioned in my previous article that I used the approach of an enormous and frequently-rebased git branch as a working area for the port, committing often and sometimes combining and extracting commits for review once they seemed to be ready. A port of this scale would have been entirely intractable without a tool of similar power to git rebase, so I m very glad that we finished migrating to git in 2019. I relied on this right up to the end of the port, and it also allowed for quick assessments of how much more there was to land. git worktree was also helpful, in that I could easily maintain working trees built for each of Python 2 and 3 for comparison. As is usual for most multi-developer projects, all changes to Launchpad need to go through code review, although we sometimes make exceptions for very simple and obvious changes that can be self-reviewed. Since I knew from the outset that this was going to generate a lot of changes for review, I therefore structured my work from the outset to try to make it as easy as possible for my colleagues to review it. This generally involved keeping most changes to a somewhat manageable size of 800 lines or less (although this wasn t always possible), and arranging commits mainly according to the kind of change they made rather than their location. For example, when I needed to fix issues with / in Python 3 being true division rather than floor division, I did so in one commit across the various places where it mattered and took care not to mix it with other unrelated changes. This is good practice for nearly any kind of development, but it was especially important here since it allowed reviewers to consider a clear explanation of what I was doing in the commit message and then skim-read the rest of it much more quickly. It was vital to keep the codebase in a working state at all times, and deploy to production reasonably often: this way if something went wrong the amount of code we had to debug to figure out what had happened was always tractable. (Although I can t seem to find it now to link to it, I saw an account a while back of a company that had taken a flag-day approach instead with a large codebase. It seemed to work for them, but I m certain we couldn t have made it work for Launchpad.) I can t speak too highly of Launchpad s test suite, much of which originated before my time. Without a great deal of extensive coverage of all sorts of interesting edge cases at both the unit and functional level, and a corresponding culture of maintaining that test suite well when making new changes, it would have been impossible to be anything like as confident of the port as we were. As part of the porting work, we split out a couple of substantial chunks of the Launchpad codebase that could easily be decoupled from the core: its Mailman integration and its code import worker. Both of these had substantial dependencies with complex requirements for porting to Python 3, and arranging to be able to do these separately on their own schedule was absolutely worth it. Like disentangling balls of wool, any opportunity you can take to make things less tightly-coupled is probably going to make it easier to disentangle the rest. (I can see a tractable way forward to porting the code import worker, so we may well get that done soon. Our Mailman integration will need to be rewritten, though, since it currently depends on the Python-2-only Mailman 2, and Mailman 3 has a different architecture.) Python lessons Our database layer was already in pretty good shape for a port, since at least the modern bits of its table modelling interface were already strict about using Unicode for text columns. If you have any kind of pervasive low-level framework like this, then making it be pedantic at you in advance of a Python 3 port will probably incur much less swearing in the long run, as you won t be trying to deal with quite so many bytes/text issues at the same time as everything else. Early in our port, we established a standard set of __future__ imports and started incrementally converting files over to them, mainly because we weren t yet sure what else to do and it seemed likely to be helpful. absolute_import was definitely reasonable (and not often a problem in our code), and print_function was annoying but necessary. In hindsight I m not sure about unicode_literals, though. For files that only deal with bytes and text it was reasonable enough, but as I mentioned above there were also a number of cases where we needed literals of the language s native str type, i.e. bytes in Python 2 and text in Python 3: this was particularly noticeable in WSGI contexts, but also cropped up in some other surprising places. We generally either omitted unicode_literals or used six.ensure_str in such cases, but it was definitely a bit awkward and maybe I should have listened more to people telling me it might be a bad idea. A lot of Launchpad s early tests used doctest, mainly in the style where you have text files that interleave narrative commentary with examples. The development team later reached consensus that this was best avoided in most cases, but by then there were far too many doctests to conveniently rewrite in some other form. Porting doctests to Python 3 is really annoying. You run into all the little changes in how objects are represented as text (particularly u'...' versus '...', but plenty of other cases as well); you have next to no tools to do anything useful like skipping individual bits of a doctest that don t apply; using __future__ imports requires the rather obscure approach of adding the relevant names to the doctest s globals in the relevant DocFileSuite or DocTestSuite; dealing with many exception tracebacks requires something like zope.testing.renormalizing; and whatever code refactoring tools you re using probably don t work properly. Basically, don t have done that. It did all turn out to be tractable for us in the end, and I managed to avoid using much in the way of fragile doctest extensions aside from the aforementioned zope.testing.renormalizing, but it was not an enjoyable experience. Regressions I know of nine regressions that reached Launchpad s production systems as a result of this porting work; of course there were various other regressions caught by CI or in manual testing. (Considering the size of this project, I count it as a resounding success that there were only nine production issues, and that for the most part we were able to fix them quickly.) Equality testing of removed database objects One of the things we had to do while porting to Python 3 was to implement the __eq__, __ne__, and __hash__ special methods for all our database objects. This was quite conceptually fiddly, because doing this requires knowing each object s primary key, and that may not yet be available if we ve created an object in Python but not yet flushed the actual INSERT statement to the database (most of our primary keys are auto-incrementing sequences). We thus had to take care to flush pending SQL statements in such cases in order to ensure that we know the primary keys. However, it s possible to have a problem at the other end of the object lifecycle: that is, a Python object might still be reachable in memory even though the underlying row has been DELETEd from the database. In most cases we don t keep removed objects around for obvious reasons, but it can happen in caching code, and buildd-manager crashed as a result (in fact while it was still running on Python 2). We had to take extra care to avoid this problem. Debian imports crashed on non-UTF-8 filenames Python 2 has some unfortunate behaviour around passing bytes or Unicode strings (depending on the platform) to shutil.rmtree, and the combination of some porting work and a particular source package in Debian that contained a non-UTF-8 file name caused us to run into this. The fix was to ensure that the argument passed to shutil.rmtree is a str regardless of Python version. We d actually run into something similar before: it s a subtle porting gotcha, since it s quite easy to end up passing Unicode strings to shutil.rmtree if you re in the process of porting your code to Python 3, and you might easily not notice if the file names in your tests are all encoded using UTF-8. lazr.restful ETags We eventually got far enough along that we could switch one of our four appserver machines (we have quite a number of other machines too, but the appservers handle web and API requests) to Python 3 and see what happened. By this point our extensive test suite had shaken out the vast majority of the things that could go wrong, but there was always going to be room for some interesting edge cases. One of the Ubuntu kernel team reported that they were seeing an increase in 412 Precondition Failed errors in some of their scripts that use our webservice API. These can happen when you re trying to modify an existing resource: the underlying protocol involves sending an If-Match header with the ETag that the client thinks the resource has, and if this doesn t match the ETag that the server calculates for the resource then the client has to refresh its copy of the resource and try again. We initially thought that this might be legitimate since it can happen in normal operation if you collide with another client making changes to the same resource, but it soon became clear that something stranger was going on: we were getting inconsistent ETags for the same object even when it was unchanged. Since we d recently switched a quarter of our appservers to Python 3, that was a natural suspect. Our lazr.restful package provides the framework for our webservice API, and roughly speaking it generates ETags by serializing objects into some kind of canonical form and hashing the result. Unfortunately the serialization was dependent on the Python version in a few ways, and in particular it serialized lists of strings such as lists of bug tags differently: Python 2 used [u'foo', u'bar', u'baz'] where Python 3 used ['foo', 'bar', 'baz']. In lazr.restful 1.0.3 we switched to using JSON for this, removing the Python version dependency and ensuring consistent behaviour between appservers. Memory leaks This problem took the longest to solve. We noticed fairly quickly from our graphs that the appserver machine we d switched to Python 3 had a serious memory leak. Our appservers had always been a bit leaky, but now it wasn t so much a small hole that we can bail occasionally as the boat is sinking rapidly : A serious memory leak (Yes, this got in the way of working out what was going on with ETags for a while.) I spent ages messing around with various attempts to fix this. Since only a quarter of our appservers were affected, and we could get by on 75% capacity for a while, it wasn t urgent but it was definitely annoying. After spending some quality time with objgraph, for some time I thought traceback reference cycles might be at fault, and I sent a number of fixes to various upstream projects for those (e.g. zope.pagetemplate). Those didn t help the leaks much though, and after a while it became clear to me that this couldn t be the sole problem: Python has a cyclic garbage collector that will eventually collect reference cycles as long as there are no strong references to any objects in them, although it might not happen very quickly. Something else must be going on. Debugging reference leaks in any non-trivial and long-running Python program is extremely arduous, especially with ORMs that naturally tend to end up with lots of cycles and caches. After a while I formed a hypothesis that zope.server might be keeping a strong reference to something, although I never managed to nail it down more firmly than that. This was an attractive theory as we were already in the process of migrating to Gunicorn for other reasons anyway, and Gunicorn also has a convenient max_requests setting that s good at mitigating memory leaks. Getting this all in place took some time, but once we did we found that everything was much more stable: A rather flat memory graph This isn t completely satisfying as we never quite got to the bottom of the leak itself, and it s entirely possible that we ve only papered over it using max_requests: I expect we ll gradually back off on how frequently we restart workers over time to try to track this down. However, pragmatically, it s no longer an operational concern. Mirror prober HTTPS proxy handling After we switched our script servers to Python 3, we had several reports of mirror probing failures. (Launchpad keeps lists of Ubuntu archive and image mirrors, and probes them every so often to check that they re reasonably complete and up to date.) This only affected HTTPS mirrors when probed via a proxy server, support for which is a relatively recent feature in Launchpad and involved some code that we never managed to unit-test properly: of course this is exactly the code that went wrong. Sadly I wasn t able to sort out that gap, but at least the fix was simple. Non-MIME-encoded email headers As I mentioned above, there were substantial changes in the email package between Python 2 and 3, and indeed between minor versions of Python 3. Our test coverage here is pretty good, but it s an area where it s very easy to have gaps. We noticed that a script that processes incoming email was crashing on messages with headers that were non-ASCII but not MIME-encoded (and indeed then crashing again when it tried to send a notification of the crash!). The only examples of these I looked at were spam, but we still didn t want to crash on them. The fix involved being somewhat more careful about both the handling of headers returned by Python s email parser and the building of outgoing email notifications. This seems to be working well so far, although I wouldn t be surprised to find the odd other incorrect detail in this sort of area. Failure to handle non-ISO-8859-1 URL-encoded form input Remember how I said that parsing HTTP form data was thorny? After we finished upgrading all our appservers to Python 3, people started reporting that they couldn t post Unicode comments to bugs, which turned out to be only if the attempt was made using JavaScript, and was because I hadn t quite managed to get URL-encoded form data working properly with zope.publisher and multipart. The current standard describes the URL-encoded format for form data as in many ways an aberrant monstrosity , so this was no great surprise. Part of the problem was some very strange choices in zope.publisher dating back to 2004 or earlier, which I attempted to clean up and simplify. The rest was that Python 2 s urlparse.parse_qs unconditionally decodes percent-encoded sequences as ISO-8859-1 if they re passed in as part of a Unicode string, so multipart needs to work around this on Python 2. I m still not completely confident that this is correct in all situations, but at least now that we re on Python 3 everywhere the matrix of cases we need to care about is smaller. Inconsistent marshalling of Loggerhead s disk cache We use Loggerhead for providing web browsing of Bazaar branches. When we upgraded one of its two servers to Python 3, we immediately noticed that the one still on Python 2 was failing to read back its revision information cache, which it stores in a database on disk. (We noticed this because it caused a deployment to fail: when we tried to roll out new code to the instance still on Python 2, Nagios checks had already caused an incompatible cache to be written for one branch from the Python 3 instance.) This turned out to be a similar problem to the pickle issue mentioned above, except this one was with marshal, which I didn t think to look for because it s a relatively obscure module mostly used for internal purposes by Python itself; I m not sure that Loggerhead should really be using it in the first place. The fix was relatively straightforward, complicated mainly by now needing to cope with throwing away unreadable cache data. Ironically, if we d just gone ahead and taken the nominally riskier path of upgrading both servers at the same time, we might never have had a problem here. Intermittent bzr failures Finally, after we upgraded one of our two Bazaar codehosting servers to Python 3, we had a report of intermittent bzr branch hangs. After some digging I found this in our logs:
Traceback (most recent call last):
  ...
  File "/srv/bazaar.launchpad.net/production/codehosting1-rev-20124175fa98fcb4b43973265a1561174418f4bd/env/lib/python3.5/site-packages/twisted/conch/ssh/channel.py", line 136, in addWindowBytes
    self.startWriting()
  File "/srv/bazaar.launchpad.net/production/codehosting1-rev-20124175fa98fcb4b43973265a1561174418f4bd/env/lib/python3.5/site-packages/lazr/sshserver/session.py", line 88, in startWriting
    resumeProducing()
  File "/srv/bazaar.launchpad.net/production/codehosting1-rev-20124175fa98fcb4b43973265a1561174418f4bd/env/lib/python3.5/site-packages/twisted/internet/process.py", line 894, in resumeProducing
    for p in self.pipes.itervalues():
builtins.AttributeError: 'dict' object has no attribute 'itervalues'
I d seen this before in our git hosting service: it was a bug in Twisted s Python 3 port, fixed after 20.3.0 but unfortunately after the last release that supported Python 2, so we had to backport that patch. Using the same backport dealt with this. Onwards!

18 July 2021

Shirish Agarwal: BBI Kenyan Supreme Court, U.P. Population Bill, South Africa, Suli Deals , IT rules 2021, Sedition Law and Danish Siddiqui s death.

BBI Kenya and live Supreme Court streaming on YT The last few weeks have been unrelenting as all sorts of news have been coming in, mostly about the downturn in the Economy, Islamophobia in India on the rise, Covid, and electioneering. However, in the last few days, Kenya surpassed India in live-streaming proceeds in a Court of Appeals about BBI or Building Bridges Initiative. A background filler article on the topic can be found in BBC. The live-streaming was done via YT and if wants to they can start from

https://www.youtube.com/watch?v=JIQzpmVKvro One can also subscribe to K24TV which took the initiative of sharing the proceedings with people worldwide. If K24TV continues to share SC proceedings of Kenya, that would add to the soft power of Kenya. I will not go into the details of the case as Gautam Bhatia who has been following the goings-on in Kenya is a far better authority on the subject. In fact, just recently he shared about another Kenyan judgment from a trial which can be seen here. He has shared the proceedings and some hot takes on the Twitter thread started by him. Probably after a couple of weeks or more when he has processed what all has happened there, he may also share some nuances although many of his thoughts would probably go to his book on Comparative Constitutional Law which he hopes to publish maybe in 2021/2022 or whenever he can. Such televised proceedings are sure to alleviate the standing of Kenya internationally. There has been a proposal to do similar broadcasts by India but with surveillance built-in, so they know who is watching. The problems with the architecture and the surveillance built-in have been shared by Srinivas Kodali or DigitalDutta quite a few times, but that probably is a story for another day.

Uttar Pradesh Population Control Bill
Hindus comprise 83% of Indian couples with more than two child children
The U.P. Population Bill came and it came with lot of prejudices. One of the prejudices is the idea that Muslims create or procreate to have the most children. Even with data is presented as shared above from NFHS National Family Health Survey which is supposed to carry our surveys every few years did the last one around 4 years back. The analysis from it has been instrumental not only in preparing graphs as above but also sharing about what sort of death toll must have been in rural India. And as somebody who have had the opportunity in the past, can vouch that you need to be extremely lucky if something happens to you when you are in a rural area. Even in places like Bodh Gaya (have been there) where millions of tourists come as it is one of the places not to be missed on the Buddhism tourist circuit, the medical facilities are pretty underwhelming. I am not citing it simply because there are too many such newspaper reports from even before the pandemic, and both the State and the Central Govt. response has been dismal. Just a few months back, they were recalled. There were reports of votes being bought at INR 1000/- (around $14) and a bottle or two of liquor. There used to be a time when election monitoring whether national or state used to be a thing, and you had LTO s (Long-time Observers) and STO s (Short-Term Observers) to make sure that the election has been neutral. This has been on the decline in this regime, but that probably is for another time altogether. Although, have to point out the article which I had shared a few months ago on the private healthcare model is flawed especially for rural areas. Instead of going for cheap, telemedicine centers that run some version of a Linux distro. And can provide a variety of services, I know Kerala and Tamil Nadu from South India have experimented in past but such engagements need to be scaled up. This probably will come to know when the next time I visit those places (sadly due to the virus, not anytime soonish.:( ) . Going back to the original topic, though, I had shared Hans Rosling s famous Ted talk on population growth which shows that even countries which we would not normally associate with family planning for e.g. the middle-east and Africa have also been falling quite rapidly. Of course, when people have deeply held prejudices, then it is difficult. Even when sharing China as to how they had to let go of their old policy in 2016 as they had the thing for leftover men . I also shared the powerful movie So Long my Son. I even shared how in Haryana women were and are trafficked and have been an issue for centuries but as neither suits the RW propaganda, they simply refuse to engage. They are more repulsed by people who publish this news rather than those who are actually practicing it, as that is culture . There is also teenage pregnancy, female infanticide, sex-selective abortion, etc., etc. It is just all too horrible to contemplate. Personal anecdote I know a couple, or they used to be a couple, where the gentleman wanted to have a male child. It was only after they got an autistic child, they got their DNA tested and came to know that the gentleman had a genetic problem. He again forced and had another child, and that too turned out to be autistic. Finally, he left the wife and the children, divorced them and lived with another woman. Almost a decade of the wife s life was ruined. The wife before marriage was a gifted programmer employed at IBM. This was an arranged marriage. After this, if you are thinking of marrying, apart from doing astrology charts, also look up DNA compatibility charts. Far better than ruining yours or the women s life. Both the children whom I loved are now in heaven, god bless them  If one wants to, one can read a bit more about the Uttar Pradesh Population bill here. The sad part is that the systems which need fixing, nobody wants to fix. The reason being simple. If you get good health service by public sector, who will go to the private sector. In Europe, AFAIK they have the best medical bang for the money. Even the U.S. looks at Europe and hopes it had the systems that Europe has but that again is probably for another day.

South Africa and India long-lost brothers. As had shared before, after the 2016 South African Debconf convention, I had been following South Africa. I was happy when FeesMustFall worked and in 2017 the then ANC president Zuma declared it in late 2017. I am sure that people who have been regular visitors to this blog know how my position is on student loans. They also must be knowing that even in U.S. till the 1970s it had free education all the way to be a lawyer and getting a lawyer license. It is only when people like Thurgood Marshall, Martin Luther King Jr., and others from the civil rights movement came out as a major force that the capitalists started imposing fees. They wanted people who could be sold to corporate slavery, and they won. Just last week, Biden took some steps and canceled student loans and is working on steps towards broad debt forgiveness. Interestingly, NASA has an affirmative diversity program for people from diverse backgrounds, where a couple of UC (Upper Caste) women got the job. While they got the job, the RW (Right-Wing) was overjoyed as they got jobs on merit . Later, it was found that both the women were the third or fourth generation of immigrants in U.S.
NASA Federal Equal Opportunity Policy Directive NPD 3713 2H
Going back to the original question and topic, while there has been a concerning spate of violence, some calling it the worst sort of violence not witnessed since 1994. The problem, as ascertained in that article, is the same as here in India or elsewhere. Those, again, who have been on my blog know that merit 90% of the time is a function of privilege and there is a vast amount of academic literature which supports that. If, for a moment, you look at the data that is shared in the graph above which shows that 83% of Hindus and 13% of Muslims have more than 2 children, what does it show, it shows that 83+13 = 96% of the population is living in insecurity. The 5% are the ones who have actually consolidated more power during this regime rule in India. Similarly, from what I understood living in Cape Town for about a month, it is the Dutch Afrikaans as they like to call themselves and the immigrants who come from abroad who have enjoyed the fruits of tourism and money and power while the rest of the country is dying due to poverty. It is the same there, it is the same here. Corruption is also rampant in both countries, and the judiciary is virtually absent from both communities in India and SA. Interestingly, South Africa and India have been at loggerheads, but I suspect that is more due to the money and lobbying power by the Dutch. Usually, those who have money power, do get laws and even press on their side, and it is usually the ruling party in power. I cannot help but share about the Gupta brothers and their corruption as I came to know about it in 2016. And as have shared that I m related to Gupta s on my mother s side, not those specific ones but Gupta as a clan. The history of the Gupta dynasty does go back to the 3rd-4th century. Equally interesting have been Sonali Ranade s series of articles which she wrote in National Herald, the latest on exports which is actually the key to taking India out of poverty rather than anything else. While in other countries Exporters are given all sort of subsidies, here it is being worked as how to give them less. This was in Economic times hardly a week back
Export incentive schemes being reduced
I can t imagine the incredible stupidity done by the Finance Minister. And then in an attempt to prove that, they will attempt to present a rosy picture with numbers that have nothing to do with reality. Interestingly enough, India at one time was a major exporter of apples, especially from Kashmir. Now instead of exporting, we are importing them from Afghanistan as well as Belgium and now even from the UK. Those who might not want to use the Twitter link could use this article. Of course, what India got out of this trade deal is not known. One can see that the UK got the better deal from this. Instead of investing in our own capacity expansion, we are investing in increasing the capacity of others. This is at the time when due to fuel price hike (Central taxes 66%) demand is completely flat. And this is when our own CEA (Chief Economic Adviser) tells us that growth will be at the most 6-7% and that too in 2023-2024 while currently, the inflation rate is around 12%. Is it then any wonder that almost 70% are living on Govt. ration and people in the streets of Kolkata, Assam, and other places have to sell kidneys to make sure they have some money for their kids for tomorrow. Now I have nothing against the UK but trade negotiation is an art. Sadly, this has been going on for the last few years. The politicians in India fool the public by always telling of future trade deals. Sadly, as any businessman knows, once you have compromised, you always have to compromise. And the more you compromise, the more you weaken the hand for any future trade deals.
IIT pupil tries to sell kidney to repay loan, but no takers for Dalit organ.
The above was from yesterday s Times of India. Just goes to show how much people are suffering. There have been reports in vernacular papers of quite a few people from across regions and communities are doing this so they can live without pain a bit. Almost all the time, the politicians are saved as only few understand international trade, the diplomacy and the surrounding geopolitics around it. And this sadly, is as much to do with basic education as much as it is to any other factor

Suli Deals About a month back on the holy day of Ramzan or Ramadan as it is known in the west, which is beloved by Muslims, a couple of Muslim women were targeted and virtually auctioned. Soon, there was a flood and a GitHub repository was created where hundreds of Muslim women, especially those who have a voice and fearlessly talk about their understanding about issues and things, were being virtually auctioned. One week after the FIR was put up, to date none of the people mentioned in the FIR have been arrested. In fact, just yesterday, there was an open letter which was published by livelaw. I have saved a copy on WordPress just in case something does go wrong. Other than the disgust we feel, can t say much as no action being taken by GOI and police.

IT Rules 2021 and Big Media After almost a year of sleeping when most activists were screaming hoarsely about how the new IT rules are dangerous for one and all, big media finally woke up a few weeks back and listed a writ petition in Madras High Court of the same. Although to be frank, the real writ petition was filed In February 2021, classical singer, performer T.M. Krishna in Madras High Court. Again, a copy of the writ petition, I have hosted on WordPress. On 23rd June 2021, a group of 13 media outlets and a journalist have challenged the IT Rules, 2021. The Contention came from Digital News Publishers Association which is made up of the following news companies: ABP Network Private Limited, Amar Ujala Limited, DB Corp Limited, Express Network Pvt Ltd, HT Digital Streams Limited, IE Online Media Services Pvt Ltd, Jagran Prakashan Limited, Lokmat Media Private Limited, NDTV Convergence Limited, TV Today Network Limited, The Malayala Manorama Co (P) Ltd, Times Internet Limited, and Ushodaya Enterprises Private Limited. All the above are heavyweights in the markets where they operate. The reason being simple, when these media organizations came into being, the idea was to have self-regulation, which by and large has worked. Now, the present Govt. wants each news item to be okayed by them before publication. This is nothing but blatant misuse of power and an attempt at censorship. In fact, the Tamil Nadu BJP president himself made a promise of the same. And of course, what is true and what is a lie, only GOI knows and will decide for the rest of the country. If somebody remembers Joseph Goebbels at this stage, it is merely a coincidence. Anyways, 3 days ago Supreme Court on 14th July the Honorable Supreme Court asked the Madras High Court to transfer all the petitions to SC. This, the Madras High Court denied as cited/shared by Meera Emmanuel, a reporter who works with barandbench. The Court says nothing doing, let this happen and then the SC can entertain the motion of doing it that level. At the same time, they would have the benefit of Madras High Court opinion as well. It gave the center two weeks to file a reply. So, either of end-week of July or latest by August first week, we might be able to read the Center s reply on the same. The SC could do a forceful intervention, but it would lead to similar outrage as has been witnessed in the past when a judge commented that if the SC has to do it all, then why do we need the High Courts, district courts etc. let all the solutions come from SC itself. This was, admittedly, frustration on the part of the judge, but due in part to the needless intervention of SC time and time again. But the concerns had been felt around all the different courts in the country.

Sedition Law A couple of days ago, the Supreme Court under the guidance of Honorable CJI NV Ramanna, entertained the PIL filed by Maj Gen S G Vombatkere (Retd.) which asked simply that the sedition law which was used in the colonial times by the British to quell dissent by Mahatma Gandhi and Bal Gangadhar Tilak during the Indian freedom struggle. A good background filler article can be found on MSN which tells about some recent cases but more importantly how historically the sedition law was used to quell dissent during India s Independence. Another article on MSN actually elaborates on the PIL filed by Maj Gen S. G. Vombatkere. Another article on MSN tells how sedition law has been challenged and changed in 10 odd countries. I find it equally sad and equally hilarious that the Indian media whose job is to share news and opinion on this topic is being instead of being shared more by MSN. Although, I would be bereft of my duty if I did not share the editorial on the same topic by the Hindu and Deccan Chronicle. Also, an interesting question to ask is, are there only 10 countries in the world that have sedition laws? AFAIK, there are roughly 200 odd countries as recognized by WTO. If 190 odd countries do not have sedition laws, it also tells a lot about them and a lot about the remaining 10. Also, it came to light that police are still filing laws under sec66A which was declared null and void a few years ago. It was replaced with section 124A if memory serves right and it has more checks and balances.

Danish Siddiqui, Pulitzer award-winning and death in Afghanistan Before I start with Danish Siddiqui, let me share an anecdote that I think I have shared on the blog years ago about how photojournalists are. Again, those who know me and those who follow me know how much I am mad both about trains and planes (civil aviation). A few months back, I had shared a blog post about some of the biggest railway systems in the world which shows that privatization of Railways doesn t necessarily lead to up-gradation of services but definitely leads to an increase in tariff/fares. Just had a conversation couple of days ago on Twitter and realized that need to also put a blog post about civil aviation in India and the problems it faces, but I digress. This was about a gentleman who wanted to take a photo of a particular train coming out of a valley at a certain tunnel at two different heights, one from below and one from above the train. This was several years ago, and while I did share that award-winning photograph then, it probably would take me quite a bit of time and effort to again look it up on my blog and share. The logistics though were far more interesting and intricate than I had first even thought of. We came around a couple of days before the train was supposed to pass that tunnel and the valley. More than half a dozen or maybe more shots were taken throughout the day by the cameras. The idea was to see how much light was being captured by the cameras and how much exposure was to be given so that the picture isn t whitened out or is too black. Weather is the strangest of foes for a photojournalist or even photographers, and the more you are in nature, the more unpredictable it is and can be. We were also at a certain height, so care had to be taken in case light rainfall happens or dew falls, both not good for digital cameras. And dew is something which will happen regardless of what you want. So while the two days our gentleman cameraman fiddled with the settings to figure out correct exposure settings, we had one other gentleman who was supposed to take the train from an earlier station and apprise us if the train was late or not. The most ideal time would be at 0600 hrs. When the train would enter the tunnel and come out and the mixture of early morning sun rays, dew, the flowers in the valley, and the train would give a beautiful effect. We could stretch it to maybe 0700 hrs. Anything after that would just be useless, as it wouldn t have the same effect. And of all this depended on nature. If the skies were to remain too dark, nothing we could do about it, if the dewdrops didn t fall it would all be over. On the day of the shoot, we were told by our compatriot that the train was late by half an hour. We sank a little on hearing that news. Although Photoshop and others can do touch-ups, most professionals like to take as authentic a snap as possible. Everything had been set up to perfection. The wide-angle lenses on both the cameras with protections were set up. The tension you could cut with a knife. While we had a light breakfast, I took a bit more and went in the woods to shit and basically not be there. This was too tensed up for me. Returned an hour to find everybody in a good mood. Apparently, the shoot went well. One of the two captured it for good enough. Now, this is and was in a benign environment where the only foe was the environment. A bad shot would have meant another week in the valley, something which I was not looking forward to. Those who have lived with photographers and photojournalists know how self-involved they can be in their craft, while how grumpy they can be if they had a bad shoot. For those, who don t know, it is challenging to be friends with such people for a long time. I wish they would scream more at nature and let out the frustrations they have after a bad shoot. But again, this is in a very safe environment. Now let s cut to Danish Siddiqui and the kind of photojournalism he followed. He followed a much more riskier sort of photojournalism than the one described above. Krittivas Mukherjee in his Twitter thread shared how reporters in most advanced countries are trained in multiple areas, from risk assessment to how to behave in case you are kidnapped, are in riots, hostage situations, etc. They are also trained in all sorts of medical training from treating gunshot wounds, CPR, and other survival methods. They are supposed to carry medical equipment along with their photography equipment. Sadly, these concepts are unknown in India. And even then they get killed. Sadly, he attributes his death to the thrill of taking an exclusive photograph. And the gentleman s bio reads that he is a diplomat. Talk about tone-deafness  On another completely different level was Karen Hao who was full of empathy as she shared the humility, grace, warmth and kinship she describes in her interaction with the photojournalist. His body of work can be seen via his ted talk in 2020 where he shared a brief collage of his works. Latest, though in a turnaround, the Taliban have claimed no involvement in the death of photojournalist Danish Siddiqui. This could be in part to show the Taliban in a more favorable light as they do and would want to be showcased as progressive, even though they are forcing that all women within a certain age become concubines or marry the fighters and killing the minority Hazaras or doing vile deeds with them. Meanwhile, statements made by Hillary Clinton almost a decade, 12 years ago have come back into circulation which stated how the U.S. itself created the Taliban to thwart the Soviet Union and once that job was finished, forgot all about it. And then in 2001, it landed back in Afghanistan while the real terrorists were Saudi. To date, not all documents of 9/11 are in the public domain. One can find more information of the same here. This is gonna take probably another few years before Saudi Arabia s whole role in the September 11 attacks will be known. Last but not the least, came to know about the Pegasus spyware and how many prominent people in some nations were targeted, including in mine India. Will not talk more as it s already a big blog post and Pegasus revelations need an article on its own.

28 June 2021

Shirish Agarwal: Indian Capital Markets, BSE, NSE

I had been meaning to write on the above topic for almost a couple of months now but just kept procrastinating about it. That push came to a shove when Sucheta Dalal and Debasis Basu shared their understanding, wisdom, and all in the new book called Absolute Power Inside story of the National Stock Exchange s amazing success, leading to hubris, regulatory capture and algo scam . Now while I will go into the details of the new book as currently, I have not bought it but even if I had bought it and shared some of the revelations from it, it wouldn t have done justice to either the book or what is sharing before knowing some of the background before it.

Before I jump ahead, I would suggest people to read my sort of introductory blog post on banking history so they know where I m coming from. I m going to deviate a bit from Banking as this is about trade and capital markets, although Banking would come in later on. And I will also be sharing some cultural insights along with history so people are aware of why things happened the way they did. Calicut, Calcutta, Kolkata, one-time major depot around the world Now, one cannot start any topic about trade without talking about Kolkata. While today, it seems like a bastion of communism, at one time it was one of the major trade depots around the world. Both William Dalrymple and the Chinese have many times mentioned Kolkata as being one of the major centers of trade. This was between the 13th and the late 19th century. A cursory look throws up this article which talks about Kolkata or Calicut as it was known as a major trade depot. There are of course many, many articles and even books which do tell about how Kolkata was a major trade depot. Now between the 13th and 19th century, a lot of changes happened which made Kolkata poorer and shifted trade to Mumbai/Bombay which in those times was nothing but just a port city like many others.

The Rise of the Zamindar Around the 15th century when Babur Invaded Hindustan, he realized that Hindustan is too big a country to be governed alone. And Hindustan was much broader than independent India today. So he created the title of Zamindars. Interestingly, if you look at the Mughal period, they were much more in tune with Hindustani practices than the British who came later. They used the caste divisions and hierarchy wisely making sure that the status quo was maintained as far as castes/creed were concerned. While in-fighting with various rulers continued, it was more or less about land and power other than anything else. When the Britishers came they co-opted the same arrangement with a minor adjustment. While in the before system, the zamindars didn t have powers to be landowners. The Britishers gave them land ownerships. A huge percentage of thess zamindars especially in Bengal were from my own caste Banias or Baniyas. The problem and the solution for the Britishers had been this was a large land to control and exploit and the number of British officers and nobles were very less. So they gave virtually a lot of powers to the Banias. The only thing the British insisted on were very high rents from the newly minted Zamindars. The Zamindar in turn used the powers of personal fiefdom to give loans at very high interest rates when the poor were unable to pay the interest rate, they would take the land while at the same time slavery was forced on both men and women, many a time rapes and affairs. While there have been many records shedding light on it, don t think it could be any more powerful as enacted and shared by Shabana Azmi in Ankur:the Seedling. Another prominent grouping was formed around the same time was the Bhadralok. Now as shared Bhadralok while having all the amenities of belonging to the community, turned a blind eye to the excesses being done by the Zamindars. How much they played a hand in the decimation of Bengal has been a matter of debate, but they did have a hand, that much is not contested.

The Rise of Stock Exchanges Sadly and interestingly, many people believe and continue to believe that stock exchanges is recent phenomena. The first stock exchange though was the Calcutta Stock Exchange rather than the Bombay Stock Exchange. How valuable was Calcutta to the Britishers in its early years can be gauged from the fact that at one time it was made the capital of India in 1772 . In fact, after the Grand Trunk Road (on which there had been even Train names in both countries) x number of books have been written of the trade between Calcutta and Peshawar (Now in Pakistan). And it was not just limited to trade but also cultural give-and-take between the two centers. Even today, if you look at YT (Youtube) and look up some interviews of old people, you find many interesting anecdotes of people sharing both culture and trade.

The problem of the 60 s and rise of BSE
After India became independent and the Constitutional debates happened, the new elites understood that there cannot be two power centers that could govern India. On one hand, were the politicians who had come to power on the back of the popular vote, the other was the Zamindars, who more often than not had abused their powers which resulted in widespread poverty. The Britishers are to blame, but so do the middlemen as they became willing enablers to the same system of oppression. Hence, you had the 1951 amendment to the Constitution and the 1956 Zamindari Abolition Act. In fact, you can find much more of an in-depth article both about Zamindars and their final abolition here. Now once Zamindari was gone, there was nothing to replace it with. The Zamindars ousted of their old roles turned and tried to become Industrialists. The problem was that the poor and the downtrodden had already had experiences with the Zamindars. Also, some Industrialists from North and West also came to Bengal but they had no understanding of either the language or the cultural understanding of what had happened in Bengal. And notice that I have not talked about both the famines and the floods that wrecked Bengal since time immemorial and some of the ones which got etched on soul of Bengal and has marks even today  The psyche of the Bengali and the Bhadralok has gone through enormous shifts. I have met quite a few and do see the guilt they feel. If one wonders as to how socialist parties are able to hold power in Bengal, look no further than Tarikh which tells and shares with you that even today how many Bengalis still feel somewhat lost.

The Rise of BSE Now, while Kolkata Stock Exchange had been going down, for multiple reasons other than listed above. From the 1950s onwards Jawaharlal Nehru had this idea of 5-year plans, borrowed from socialist countries such as Russia, China etc. His vision and ambition for the newly minted Indian state were huge, while at the same time he understood we were poor. The loot by East India Company and the Britishers and on top of that the division of wealth with Pakistan even though the majority of Muslims chose and remained with India. Travel on Indian Railways was a risky affair. My grandfather had shared numerous tales where he used to fill money in socks and put the socks on in boots when going between either Delhi Kolkata or Pune Kolkata. Also, as the Capital became Delhi, it unofficially was for many years, the transparency from Kolkata-based firms became less. So many Kolkata firms either mismanaged and shut down while Maharashtra, my own state, saw a huge boon in Industrialization as well as farming. From the 1960s to the 1990s there were many booms and busts in the stock exchanges but most were manageable.

While the 60s began on a good note as Goa was finally freed from the Portuguese army and influence, the 1962 war with the Chinese made many a soul question where we went wrong. Jawaharlal Nehru went all over the world to ask for help but had to return home empty-handed. Bollywood showed a world of bell-bottoms and cars and whatnot, while the majority were still trying to figure out how to put two square meals on the table. India suffered one of the worst famines in those times. People had to ration food. Families made do with either one meal or just roti (flatbread) rather than rice. In Bengal, things were much more severe. There were huge milk shortages, so Bengalis were told to cut down on sweets. This enraged the Bangalis as nothing else could. Note If one wants to read how bad Indians felt at that time, all one has to read is V.S. Naipaul s An Area of darkness . This was also the time when quite a few Indians took their first step out of India. While Air India had just started, the fares were prohibitive. Those who were not well off, either worked on ships or went via passenger or cargo ships to Dubai/Qatar middle-east. Some went to Russia and some even to States. While today s migr s want to settle in the west forever and have their children and grandchildren grow up in the West, in the 1960s and 70s the idea was far different. The main purpose for a vast majority was to get jobs and whatnot, save maximum money and send it back to India as a remittance. The idea was to make enough money in 3-5-10 years, come back to India, and then lead a comfortable life. Sadly, there has hardly been any academic work done in India, at least to my knowledge to document the sacrifices done by Indians in search of jobs, life, purpose, etc. in the 1960s and 1970s. The 1970s was also when alternative cinema started its journey with people like Smita Patil, Naseeruddin Shah who portrayed people s struggles on-screen. Most of them didn t have commercial success because the movies and the stories were bleak. While the acting was superb, most Indians loved to be captured by fights, car-chases, and whatnot rather than the deary existence which they had. And the alt cinema forced them to look into the mirror, which was frowned upon both by the masses and the classes. So cinema which could have been a wake-up call for a lot of Indians failed. One of the most notable works of that decade, at least to me, was Manthan. 1961 was also marked by the launch of Economic Times and Financial Express which tells that there was some appetite for financial news and understanding. The 1970s was also a very turbulent time in the corporate sector and stock exchanges. Again, the companies which were listed were run by the very well-off and many of them had been abroad. At the same time, you had fly-by-night operators. One of the happenings which started in this decade is you had corporate wars and hostile takeovers, quite a few of them of which could well have a Web series or two of their own. This was also a decade marked by huge labor unrest, which again changed the face of Bombay/Mumbai. From the 1950s till the 1970s, Bombay was known for its mills. So large migrant communities from all over India came to Bombay to become the next Bollywood star and if that didn t happen, they would get jobs in the mills. Bombay/Mumbai has/had this unique feature that somehow you will make money to make ends meet. Of course, with the pandemic, even that has gone for a toss. Labor unrest was a defining character of that decade. Three movies, Kaala Patthar, Kalyug, and Ankush give a broad outlook of what happened in that decade. One thing which is present and omnipresent then and now is how time and time again we lost our demographic dividend. Again there was an exodus of young people who ventured out to seek fortunes elsewhere. The 1970s and 80s were also famous for the license Raj which they bought in. Just like the Soviets, there were waiting periods for everything. A telephone line meant waiting for things anywhere from 4 to 8 years. In 1987, when we applied and got a phone within 2-3 months, most of my relatives both from my mother and father s side could not believe we paid 0 to get a telephone line. We did pay the telephone guy INR 10/- which was a somewhat princely sum when he was installing it, even then they could not believe it as in Northern India, you couldn t get a phone line even if your number had come. You had to pay anywhere from INR 500/1000 or more to get a line. This was BSNL and to reiterate there were no alternatives at that time.

The 1990s and the Harshad Mehta Scam The 90s was when I was a teenager. You do all the stupid things for love, lust, whatever. That is also the time you are introduced really to the world of money. During my time, there were only three choices, Sciences, Commerce, and Arts. If History were your favorite subject then you would take Arts and if it was not, and you were not studious, then you would up commerce. This is how careers were chosen. So I enrolled in Commerce. Due to my grandfather and family on my mother s side interested in stocks both as a saving and compounding tool, I was able to see Pune Stock Exchange in action one day. The only thing I remember that day is people shouting loudly with various chits. I had no idea that deals of maybe thousands or even lakhs. The Pune Stock Exchange had been newly minted. I also participated in a couple of mock stock exchanges and came to understand that one has to be aggressive in order to win. You had to be really loud to be heard over others, you could not afford to be shy. Also, spread your risks. Sadly, nothing about the stock markets was there in the syllabus. 1991 was also when we saw the Iraq war, the balance of payments crisis in India, and didn t know that the Harshad Mehta scam was around the corner. Most of the scams in India have been caught because the person who was doing it was flashy. And this was the reason that even he was caught as Ms. Sucheta Dalal, a young beat reporter from Indian Express who had been covering Indian stock market. Many of her articles were thought-provoking. Now, a brief understanding is required to know before we actually get to the scam. Because of the 1991 balance of payments crisis, IMF rescued India on the condition that India throws its market open. In the 1980s itself, Rajeev Gandhi had wanted to partially make India open but both politicians and Industrialists advised him not to do the same, we are/were not ready. On 21st May 1991, Rajeev Gandhi was assassinated by the LTTE. A month later, due to the sympathy vote, the Narsimha Rao Govt. took power. While for most new Governments there is usually a honeymoon period lasting 6 months or so till they get settled in their roles before people start asking tough questions. It was not to be for this Govt. Immediately, The problem had been building for a few years. Although, in many ways, our economy was better than it is today. The only thing India didn t do well at that time was managing foreign exchange. As only a few Indians had both the money and the opportunity to go abroad and need for electronics was limited. One of the biggest imports of the time then and still today is Energy, Oil. While today it is Oil/Gas and electronics, at that time it was only OIl. The Oil import bill was ballooning while exports were more or less stagnant and mostly comprised of raw materials rather than finished products. Even today, it is largely this, one of the biggest Industrialists in India Ambani exports gas/oil while Adani exports coal. Anyways, the deficit was large enough to trigger a payment crisis. And Narsimha Rao had to throw open the Indian market almost overnight. Some changes became quickly apparent, while others took a long time to come.

Satellite Television and Entry of Foreign Banks Almost overnight, from 1 channel we became multi-channel. Star TV (Rupert Murdoch) bought us Bold and Beautiful, while CNN broadcasted the Iraq War. It was unbelievable for us that we were getting reports of what had happened 24-48 hours earlier. Fortunately or unfortunately, I was still very much a teenager to understand the import of what was happening. Even in my college, except for one or two-person, it wasn t a topic for debate or talk or even the economy. We were basically somehow cocooned in our own little world. But this was not the case for the rest of India and especially banks. The entry of foreign banks was a rude shock to Indian banks. The foreign banks were bringing both technology and sophistication in their offerings, and Indian Banks needed and wanted fast money to show hefty profits. Demand for credit wasn t much, at least nowhere the level it today is. At the same time, default on credit was nowhere high as today is. But that will require its own space and article. To quench the thirst for hefty profits by banks, Enter Harshad Mehta. At that point in time, banks were not permitted at all to invest in the securities/share market. They could only buy Government securities or bonds which had a coupon rate of say 8-10% which was nowhere enough to satisfy the need for hefty profits as desired by Indian banks. On top of it, that cash was blocked for a long time. Most of these Government bonds had anywhere between 10-20 year maturity date and some even longer. Now, one loophole in that was that the banks themselves could not buy these securities. They had to approach a registered broker of the share market who will do these transactions on their behalf. Here is where Mr. Mehta played his game. He shared both legal and illegal ways in which both the bank and he would prosper. While banking at one time was thought to be conservative and somewhat cautious, either because they were too afraid that Western private banks will take that pie or whatever their reasons might be, they agreed to his antics. To play the game, Harshad Mehta needed lots of cash, which the banks provided him in the guise of buying securities that were never bought, but the amounts were transferred to his account. He actively traded stocks, at the same time made a group, and also made the rumor mill work to his benefit. The share market is largely a reactionary market. It operates on patience, news, and rumor-mill. The effect of his shenanigans was that the price of a stock that was trending at say INR 200 reached the stratospheric height of INR 9000/- without any change in the fundamentals or outlook of the stock. His thirst didn t remain restricted to stocks but also ventured into the unglamorous world of Govt. securities where he started trading even in them in large quantities. In order to attract new clients, he coveted a fancy lifestyle. The fancy lifestyle was what caught the eye of Sucheta Dalal, and she started investigating the deals he was doing. Being a reporter, she had the advantage of getting many doors to open and get information that otherwise would be under lock and key. On 23rd April 1992, Sucheta Dalal broke the scam.

The Impact The impact was almost like a shock to the markets. Even today, it can be counted as one of the biggest scams in the Indian market if you adjust it for inflation. I haven t revealed much of the scam and what happened, simply because Sucheta Dalal and Debasis Basu wrote The Scam for that purpose. How do I shorten a story and experience which has been roughly written in 300 odd pages in one or two paragraphs, it is simply impossible. The impact though was severe. The Indian stock market became a bear market for two years. Sucheta Dalal was kicked out/made to resign out of Indian Express. The thing is simple, all newspapers survive on readership and advertisements with advertisements. Companies who were having a golden run, whether justified or not, on the bourses/Stock Exchange. For many companies, having a good number on the stock exchange was better than the company fundamentals. There was supposed to be a speedy fast-track court setup for Financial crimes, but it worked only for the Harshad Mehta case and still took over 5 years. It led to the creation of NSE (National Stock Exchange). It also led to the creation of SEBI, perhaps one of the most powerful regulators, giving it a wide range of powers and remit but on the ground more often that proved to be no more than a glorified postman. And the few times it used, it used on the wrong people and people had to go to courts to get justice. But then this is not about SEBI nor is this blog post about NSE. I have anyways shared about Absolute power above, so will not repeat the link here. The Anecdotal impact was widespread. Our own family broker took the extreme step. For my grandfather on the mother s side, he was like the second son. The news of his suicide devastated my grandfather quite a bit, which we realized much later when he was diagnosed with Alzheimer s. Our family stockbroker had been punting, taking lots of cash from the market at very high rates, betting on stocks wildly as the stock market was reaching for the stars when the market crashed, he was insolvent. How the family survived is a tale in itself. They had just got married just a few years ago and had a cute boy and girl soon after. While today, both are grown-up, at that time what the wife faced only she knows. There were also quite a few shareholders who also took the extreme step. The stock markets in those days were largely based on trust and even today is unless you are into day-trading. So there was always some money left on the table for the share/stockbroker which would be squared off in the next deal/transaction where again you will leave something. My grandfather once thought of going over and meeting them, and we went to the lane where their house is, seeing the line of people who had come for recovery of loans, we turned back with a heavy heart. There was another taboo that kinda got broken that day. The taboo was that the stock market is open to scams. From 1992 to 2021 has been a cycle of scams. Even now, today, the stock market is at unnatural highs. We know for sure that a lot of hot money is rolling around, a lot of American pension funds etc. Till it will work, it will work, some news something and that money will be moved out. Who will be left handing the can, the Indian investors? A Few days back, Ambani writes about Adani. Now while the facts shared are correct, is Adani the only one, the only company to have a small free float in the market. There probably are more than 1/4th or 1/3rd of well-respected companies who may have a similar configuration, the only problem is it is difficult to know who the proxies are. Now if I were to reflect and compare this either with the 1960s or even the 1990s I don t find much difference apart from the fact that the proxy is sitting in Mauritius. At the same time, today you can speculate on almost anything. Whether it is stocks, commodities, derivatives, foreign exchange, cricket matches etc. the list is endless. Since 2014, the rise in speculation rather than investment has been dramatic, almost stratospheric. Sadly, there are no studies or even attempts made to document this. How much official and unofficial speculation is there in the market nobody knows. Money markets have become both fluid and non-transparent. In theory, you have all sorts of regulators, but it is still very much like the Wild West. One thing to note that even Income tax had to change and bring it provisions to account for speculative income.So, starting from being totally illegitimate, it has become kind of legal and is part of Income Tax. And if speculation is not wrong, why not make Indian cricket officially a speculative event, that will be honest and GOI will get part of the proceeds.

Conclusion I wish there was some positive conclusion I could drive, but sadly there is not. Just today read two articles about the ongoing environmental issues in Himachal Pradesh. As I had shared even earlier, the last time I visited those places in 2011, and even at that time I was devastated to see the kind of construction going on. Jogiwara Road which they showed used to be flat single ground/first floor dwellings, most of which were restaurants and whatnot. I had seen the water issues both in Himachal and UT (Uttarakhand) back then and this is when they made huge dams. In U.S. they are removing dams and here we want more dams

24 June 2021

Louis-Philippe V ronneau: Hardening Weechat Relays Against RCE on Bullseye

I've been using weechat to connect to IRC since late 2016 and one of its killer feature is relays. They let use other frontends like the Weechat Android app or the amazing Glowing Bear (packaged in Debian Bullseye by yours truly). Sadly, relays also used to be somewhat of a security risk: anyone with access to a relay1 could run scripts on the machine running weechat by using commands such as /exec or /script. Not great. Since version 2.5 (Buster had version 2.3), you can mitigate this risk by setting a command allowlist for relays. Later versions implemented a sane default by blocking the following commands: Sadly, this default didn't make in into Bullseye. If you are running weechat and are using the relays feature, after upgrading to Bullseye, I would recommend you run the following commands in the weechat TUI:
/set relay.weechat.commands *,!exec,!fset,!set,!unset,!plugin,!script,!python,!perl,!ruby,!lua,!tcl,!guile,!javascript,!php,!secure,!upgrade,!quit
/save

  1. For example, someone steals your phone and connects to IRC via the Weechat app...

21 June 2021

Russ Allbery: Review: Demon Lord of Karanda

Review: Demon Lord of Karanda, by David Eddings
Series: The Malloreon #3
Publisher: Del Rey
Copyright: September 1988
Printing: February 1991
ISBN: 0-345-36331-0
Format: Mass market
Pages: 404
This is the third book of the Malloreon, which in turn is a sequel trilogy to The Belgariad. Eddings, unlike most series authors, does a great job of reminding you what's happening with prologues in each book, but you definitely do not want to start reading here. When we last left our heroes, they had been captured. (This is arguably a spoiler for King of the Murgos, but it's not much of one, nor one that really matters.) This turns out to be an opportunity to meet the Emperor of Mallorea, the empire from which their adversary Zandramas (and, from the earlier trilogy, the god Torak) comes. This goes much better than one might expect, continuing the trend in this series of showing the leaders of the enemy countries as substantially similar to the leaders of the supposedly good countries. This sounds like open-mindedness on Eddings's part, and I suppose it partly is. It's at least a change from the first series, in which the bad guys were treated more like orcs. But the deeper I read into this series, the more obvious how invested Eddings is in a weird sort of classism. Garion and the others get along with Zakath in part because they're all royalty, or at least run in those circles. They just disagree about how to be a good ruler (and not as much as one might think, or hope). The general population of any of the countries is rarely of much significance. Zakath is a bit more cynical than Garion and company and has his own agenda, but he's not able to overcome the strong conviction of this series that the Prophecy and the fight between the Child of Light and the Child of Dark is the only thing of importance that's going on, and other people matter only to the extent that they're involved in that story. When I first read these books as a teenager, I was one of the few who liked the second series and didn't mind that its plot was partly a rehash of the first. I found, and still find, the blatantness with which Eddings manipulates the plot by making prophecy a character in the novel amusing. What I had forgotten, however, was how much of a slog the middle of this series is. It takes about three quarters of this book before there are any significant plot developments, and that time isn't packed with interesting diversions. It's mostly the heroes having conversations with each other or with Zakath, being weirdly sexist, rehashing their personality quirks, or shrugging about horrific events that don't matter to them personally. The last is a reference to the plague that appears in this book, and which I had completely forgotten. To be fair to my memory, that's partly because none of the characters seem to care much about it either. They're cooling their heels in a huge city, a plague starts killing people, they give Zakath amazingly brutal and bloodthirsty advice to essentially set fire to all the parts of the city with infected people, and then they blatantly ignore all the restrictions on movement because, well, they're important unlike all those other people and have places to go. It's rather stunningly unempathetic under the best of circumstances and seems even more vile in a 2021 re-read. Eddings also manages to make Ce'Nedra even more obnoxious than she has been by turning her into a walking zombie with weird fits where she's obsessed with her child, and adding further problems (which would be a spoiler) on top of that. I have never been a Ce'Nedra fan (that Garion's marriage ever works at all appears to be by authorial decree), but in this book she's both useless and irritating while supposedly being a tragic figure. You might be able to tell that I'm running sufficiently low on patience for Eddings's character quirks that I'm losing my enthusiasm for re-reading this bit of teenage nostalgia. This is the third book of a five-book series, so while there's a climax of sorts just like there was in the third book of the Belgariad, it's a false climax. One of the secondary characters is removed, but nothing is truly resolved; the state of the plot isn't much different at the end of this book than it was at the start. And to get there, one has to put up with Garion being an idiot, Ce'Nedra being a basket case, the supposed heroes being incredibly vicious about a plague, and one character pretending to have an absolutely dreadful Irish accent for pages upon pages upon pages. (His identity is supposedly a mystery, but was completely obvious a hundred pages before Garion figured it out. Garion isn't the sharpest knife in the drawer.) The one redeeming merit to this series is the dry voice in Garion's head and the absurd sight of the prophecy telling all the characters what to do, and we barely get any of that in this book. When it wasn't irritating or offensive, it was just a waste of time. The worst book of the series so far. Followed by Sorceress of Darshiva. Rating: 3 out of 10

18 June 2021

Gunnar Wolf: Fighting spam on roundcube with modsecurity

Every couple of months, one of my users falls prey to phishing attacks, and send their login/password data to an unknown somebody who poses as Well, as me, their always-friendly and always-helpful systems administrator. What follows is, of course, me spending a week trying to get our systems out of all of the RBLs/DNSBLs. But, no matter how fast I act, there s always distruption and lost mails (bounced or classified as spam) for my users. Most of my users use the Webmail I have configured on our institute s servers, Roundcube, for which I have the highest appreciation. Only that Of course, when a user yields their username and password to an attacker, it is very successful at Sending huge amounts of unrequested mail, leading to my server losing its reputation This week, I set two bits of mitigation strategies. The first one, most straightforward, was to ask Roundcube to disallow sending mails with over ten recipients. In a Debian install, this is as easy as setting up the following variable in /etc/roundcube/config.inc.php:
$config['max_recipients'] = 10
However, a dilligent spammer can still clog the server by sending many, many, many, many requests maybe each of them with ten recipients only; last weekend, I got a new mail every three seconds or so. Adding rate limit to a specific Roundcube action is not easy, however, or at least it took me quite a bit of headbanging to get it right . Roundcube is a very AJAX-y system where all (most, at least) actions are received by /index.php and there is quite a bit of parsing to do to understand the actions done. When sending a mail, of course, it is done using the POST HTTP verb, and the URI-specified variables include _task=mail&_unlock=loading<message_id> (of course, with changing message IDs). After some poking here and there, I faced to SpiderLabs ModSecurity Only that I am not yet well versed in writing rules for it. But after quite a bit of reading, poking, breaking I was able to come up with the following rules:
# How often does the limit counter expire   ratelimit_client=60,
# every 60 seconds
SecRule REQUEST_LINE "@rx POST.*_task=mail&_unlock" id:10,phase:2,nolog,pass,setuid:% tx.ua_hash ,setvar:user.ratelimit_client=+1,expirevar:user.ratelimit_client=60
# How many requests do we allow in the specified time period?  
# @gt 3, 3 requests
SecRule user:ratelimit_client "@gt 2" chain,id:100009,phase:2,deny,status:429,setenv:RATELIMITED,log,msg:RATE-LIMITED
SecRule REQUEST_LINE "@rx POST.*_task=mail&_unlock"
The first line specifies the rule will match request lines specifying the POST verb aind including the _task=mail&_unlock fragment in the URL. It increments tht ratelimit_client user variable, but expires it after 60 seconds. The first line verifies whether the above specified variable (do note that it s user: instead of user.) is greater than 2. If so, it sets the deny action, HTTP return status of 429 (Too Many Requests), and logs the reason why this request was denied (rate-limited). And Given the way Roundcube works, this even works transparently! If a user hits the limit, the mail sending component will just wait and, after a while, time out. Then, the user can click Send again. If legitimate users are too productive and try to send over three mails in a minute, they won t lose any of it; spammers will (hopefully!) find it unbearably slow and give up. Logging is quite informative; I will probably later restrict it to show fewer parts (even if just for privacy sake, as it logs the full request!) For a complex permissions framework such as mod_security, having information such as the following is most welcome in order to find a possibly misbehaving rule:
--76659f4b-H--
Message: Access denied with code 429 (phase 2). Pattern match "POST.*_task=mail&_unlock" at REQUEST_LINE. [file "/etc/modsecurity/rate_limit_sender.conf"] [line "20"] [id "100009"] [msg "RATELIMITED BOT"]
Apache-Error: [file "apache2_util.c"] [line 273] [level 3] [client 192.168.1.48] ModSecurity: Access denied with code 429 (phase 2). Pattern match "POST.*_task=mail&_unlock" at REQUEST_LINE. [file "/etc/modsecurity/rate_limit_sender.conf"] [line "20"] [id "100009"] [msg "RATELIMITED BOT"] [hostname "my.server.mx"] [uri "/roundcube/"] [unique_id "YMzJLR9jVDMGsG@18kB1qAAAAAY"]
Action: Intercepted (phase 2)
Stopwatch: 1624033581838813 1204 (- - -)
Stopwatch2: 1624033581838813 1204; combined=352, p1=29, p2=140, p3=0, p4=0, p5=94, sr=81, sw=89, l=0, gc=0
Response-Body-Transformed: Dechunked
Producer: ModSecurity for Apache/2.9.3 (http://www.modsecurity.org/).
Server: Apache
WebApp-Info: "default" "-" ""
Engine-Mode: "ENABLED"
I truly, truly hope this is the last time my server falls in the black pits of DNSBL/RBL lists

14 June 2021

Enrico Zini: Pipelining

This is part of a series of posts on ideas for an ansible-like provisioning system, implemented in Transilience. Running actions on a server is nice, but a network round trip for each action is not very efficient. If I need to run a linear sequence of actions, I can stream them all to the server, and then read replies streamed from the server as they get executed. This technique is called pipelining and one can see it used, for example, in Redis, or Mitogen. Roles Ansible has the concept of "Roles" as a series of related tasks: I'll play with that. Here's an example role to install and setup fail2ban:
class Role(role.Role):
    def main(self):
        self.add(builtin.apt(
            name=["fail2ban"],
            state="present",
        ))
        self.add(builtin.copy(
            content=inline("""
                [postfix]
                enabled = true
                [dovecot]
                enabled = true
            """),
            dest="/etc/fail2ban/jail.local",
            owner="root",
            group="root",
            mode=0o644,
        ), name="configure fail2ban")
I prototyped roles as classes, with methods that push actions down the pipeline. If an action fails, all further actions for the same role won't executed, and will be marked as skipped. Since skipping is applied per-role, it means that I can blissfully stream actions for multiple roles to the server down the same pipe, and errors in one role will stop executing that role and not others. Potentially I can get multiple roles going with a single network round-trip:
#!/usr/bin/python3
import sys
from transilience.system import Mitogen
from transilience.runner import Runner
@Runner.cli
def main():
    system = Mitogen("my server", "ssh", hostname="server.example.org", username="root")
    runner = Runner(system)
    # Send roles to the server
    runner.add_role("general")
    runner.add_role("fail2ban")
    runner.add_role("prosody")
    # Run until all roles are done
    runner.main()
if __name__ == "__main__":
    sys.exit(main())
That looks like a playbook, using Python as glue rather than YAML. Decision making in roles Besides filing a series of actions, a role may need to take decisions based on the results of previous actions, or on facts discovered from the server. In that case, we need to wait until the results we need come back from the server, and then decide if we're done or if we want to send more actions down the pipe. Here's an example role that installs and configures Prosody:
from transilience import actions, role
from transilience.actions import builtin
from .handlers import RestartProsody
class Role(role.Role):
    """
    Set up prosody XMPP server
    """
    def main(self):
        self.add(actions.facts.Platform(), then=self.have_facts)
        self.add(builtin.apt(
            name=["certbot", "python-certbot-apache"],
            state="present",
        ), name="install support packages")
        self.add(builtin.apt(
            name=["prosody", "prosody-modules", "lua-sec", "lua-event", "lua-dbi-sqlite3"],
            state="present",
        ), name="install prosody packages")
    def have_facts(self, facts):
        facts = facts.facts  # Malkovich Malkovich Malkovich!
        domain = facts["domain"]
        ctx =  
            "ansible_domain": domain
         
        self.add(builtin.command(
            argv=["certbot", "certonly", "-d", f"chat. domain ", "-n", "--apache"],
            creates=f"/etc/letsencrypt/live/chat. domain /fullchain.pem"
        ), name="obtain chat certificate")
        with self.notify(RestartProsody):
            self.add(builtin.copy(
                content=self.template_engine.render_file("roles/prosody/templates/prosody.cfg.lua", ctx),
                dest="/etc/prosody/prosody.cfg.lua",
            ), name="write prosody configuration")
            self.add(builtin.copy(
                src="roles/prosody/templates/firewall-ruleset.pfw",
                dest="/etc/prosody/firewall-ruleset.pfw",
            ), name="write prosody firewall")
    # ...
This files some general actions down the pipe, with a hook that says: when the results of this action come back, run self.have_facts(). At that point, the role can use the results to build certbot command lines, render prosody's configuration from Jinja2 templates, and use the results to file further action down the pipe. Note that this way, while the server is potentially still busy installing prosody, we're already streaming prosody's configuration to it. If anything goes wrong with the installation of prosody's package, the role will be marked as failed and all further actions of the same role, even those filed by have_facts() will be skipped. Notify and handlers In the previous example self.notify() also appears: that's my attempt to model the equivalent of Ansible's handlers. If any of the actions inside the with produce changes, then the RestartProsody role will be executed, potentially filing more actions ad the end of the playbook. The runner will take care of collecting all the triggered role classes in a set, which discards duplicates, and then running the main() method of all resulting roles, which will cause more actions to be filed down the pipe. Action conditions Sometimes some actions are only meaningful as consequences of other actions. Let's take, for example, enabling buster-backports as an extra apt source:
        a = self.add(builtin.copy(
            owner="root",
            group="root",
            mode=0o644,
            dest="/etc/apt/sources.list.d/debian-buster-backports.list",
            content="deb [arch=amd64] https://mirrors.gandi.net/debian/ buster-backports main contrib",
        ), name="enable backports")
        self.add(builtin.apt(
            update_cache=True
        ), name="update after enabling backports",
           # Run only if the previous copy changed anything
           when= a: ResultState.CHANGED ,
        )
Here we want to update Apt's cache, which is a slow operation, only after we actually write /etc/apt/sources.list.d/debian-buster-backports.list. If the file was already there from a previous run, we can skip downloading the new package lists. The when= attributes adds an annotation to the action that is sent town the pipeline, that says that it should only be run if the state of a previous action matches the given one. In this case, when on the remote it's the turn of "update after enabling backports", it gets skipped unless the state of the previous "enable backports" action is CHANGED. Effects of pipelining I ported enough of Ansible's modules to be able to run the provisioning scripts of my VPS entirely via ansible. This is the playbook run as plain Ansible:
$ time ansible-playbook vps.yaml
[...]
servername       : ok=55   changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
real    2m10.072s
user    0m33.149s
sys 0m10.379s
This is the same playbook run with Ansible speeded up via the Mitogen backend, which makes Ansible more bearable:
$ export ANSIBLE_STRATEGY=mitogen_linear
$ time ansible-playbook vps.yaml
[...]
servername       : ok=55   changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
real    0m24.428s
user    0m8.479s
sys 0m1.894s
This is the same playbook ported to Transilience:
$ time ./provision
[...]
real    0m2.585s
user    0m0.659s
sys 0m0.034s
Doing nothing went from 2 minutes down to 3 seconds! That's the kind of running time that finally makes me comfortable with maintaining my VPS by editing the playbook only, and never logging in to mess with the system configuration by hand! Next steps I'm quite happy with what I have: I can now maintain my VPS with a simple script with quick iterative cycles. I might use it to develop new playbooks, and port them to ansible only when they're tested and need to be shared with infrastructure that needs to rely on something more solid and battle tested than a prototype provisioning system. I might also keep working on it as I have more interesting ideas that I'd like to try. I feel like Ansible reached some architectural limits that are hard to overcome without a major redesign, and are in many way hardcoded in its playbook configuration. It's nice to be able to try out new designs without that baggage. I'd love it if even just the library of Transilience actions could grow, and gain widespread use. Ansible modules standardized a set of management operations, that I think became the way people think about system management, and should really be broadly available outside of Ansible. If you are interesting in playing with Transilience, such as: do get in touch or send a pull request! :) Next step: Reimagining Ansible variables.

Enrico Zini: Use ansible actions in a script

This is part of a series of posts on ideas for an ansible-like provisioning system, implemented in Transilience. I like many of the modules provided with Ansible: they are convenient, platform-independent implementations of common provisioning steps. They'd be fantastic to have in a library that I could use in normal programs. This doesn't look easy to do with Ansible code as it is. Also, the code quality of various Ansible modules doesn't fit something I'd want in a standard library of cross-platform provisioning functions. Modeling Actions I want to keep the declarative, idempotent aspect of describing actions on a system. A good place to start could be a hierarchy of dataclasses that hold the same parameters as ansible modules, plus a run() method that performs the action:
@dataclass
class Action:
    """
    Base class for all action implementations.
    An Action is the equivalent of an ansible module: a declarative
    representation of an idempotent operation on a system.
    An Action can be run immediately, or serialized, sent to a remote system,
    run, and sent back with its results.
    """
    uuid: str = field(default_factory=lambda: str(uuid.uuid4()))
    result: Result = field(default_factory=Result)
    def summary(self):
        """
        Return a short text description of this action
        """
        return self.__class__.__name__
    def run(self, system: transilience.system.System):
        """
        Perform the action
        """
        self.result.state = ResultState.NOOP
I like that Ansible tasks have names, and I hate having to give names to trivial tasks like "Create directory /foo/bar", so I added a summary() method so that trivial tasks like that can take care of naming themselves. Dataclasses allow to introspect fields and annotate them with extra metadata, and together with docstrings, I can make actions reasonably self-documeting. I ported some of Ansible's modules over: see complete list in the git repository. Running Actions in a script With a bit of glue code I can now run Ansible-style functions from a plain Python script:
#!/usr/bin/python3
from transilience.runner import Script
script = Script()
for i in range(10):
    script.builtin.file(state="touch", path=f"/tmp/test i ")
Running Actions remotely Dataclasses have an asdict function that makes them trivially serializable. If their members stick to data types that can be serialized with Mitogen and the run implementation doesn't use non-pure, non-stdlib Python modules, then I can trivially run actions on all sorts of remote systems using Mitogen:
#!/usr/bin/python3
from transilience.runner import Script
from transilience.system import Mitogen
script = Script(system=Mitogen("my server", "ssh", hostname="machine.example.org", username="user"))
for i in range(10):
    script.builtin.file(state="touch", path=f"/tmp/test i ")
How fast would that be, compared to Ansible?
$ time ansible-playbook test.yaml
[...]
real    0m15.232s
user    0m4.033s
sys 0m1.336s
$ time ./test_script
real    0m4.934s
user    0m0.547s
sys 0m0.049s
With a network round-trip for each single operation I'm already 3x faster than Ansible, and it can run on nspawn containers, too! I always wanted to have a library of ansible modules useable in normal scripts, and I've always been angry with Ansible for not bundling their backend code in a generic library. Well, now there's the beginning of one! Sweet! Next step, pipelining.

1 June 2021

Russell Coker: Internode NBN with Arris CM8200 on Debian

I ve recently signed up for Internode NBN while using the Arris CM8200 device supplied by Optus (previously used for a regular phone service). I took the configuration mostly from Dean s great blog post on the topic [1]. One thing I changed was the /etc/networ/interfaces configuration, I used the following:
# VLAN ID 2 for Internode's NBN HFC.
auto eth1.2
iface eth1.2 inet manual
  vlan-raw-device eth1
auto nbn
iface nbn inet ppp
    pre-up /bin/ip link set eth1.2 up
    provider nbn
There is no need to have a section for eth1 when you have a section for eth1.2. IPv6 IPv6 for only one system With a line in /etc/ppp/options containing only ipv6 , you get an IPv6 address automatically for the ppp0 interface after starting pppd. IPv6 for your lan Internode has documented how to configure the WIDE DHCPv6 client to get an IPv6 prefix (subnet) [2]. Just install the wide-dhcpv6-client package and put your interface names in a copy of the Internode example config and that works. That gets you a /64 assigned to your local Ethernet. Here s an example of /etc/wide-dhcpv6/dhcp6c.conf:
interface ppp0  
    send ia-pd 0;
    script "/etc/wide-dhcpv6/dhcp6c-script";
 ;
id-assoc pd  
    prefix-interface br0  
        sla-id 0;
        sla-len 8;
     ;
 ;
For providing addresses to other systems on your LAN they recommend radvd version 1.1 or greater, Debian/Bullseye will ship with version 2.18. Here is an example /etc/radvd.conf that will work with it. It seems that you have to manually (or with a script) set the value to use in place of xxxx:xxxx:xxxx:xxxx from the value that is assigned to eth0 (or whichever interface you are using) by the wide-dhcpv6-client.
interface eth0   
        AdvSendAdvert on;
        MinRtrAdvInterval 3; 
        MaxRtrAdvInterval 10;
        prefix xxxx:xxxx:xxxx:xxxx::/64   
                AdvOnLink on; 
                AdvAutonomous on; 
                AdvRouterAddr on; 
         ;
 ;
Either the configuration of the wide dhcp client or radvd removes the default route from ppp0, so you need to run a command like
ip -6 route add default dev ppp0 to put it back. Probably having ipv6 , is the wrong thing to do when using wide-dhcp-client and radvd. On a client machine with bridging I needed to have net.ipv6.conf.br0.accept_ra=2 in /etc/sysctl.conf to allow it to accept route advisory messages on the interface (in this case eth0), for machines without bridging I didn t need that. Firewalling The default model for firewalling nowadays seems to be using NAT and only configuring specific ports to be forwarded to machines on the LAN. With IPv6 on the LAN every system can directly communicate with the rest of the world which may be a bad thing. The following lines in a firewall script will drop all inbound packets that aren t in response to packets that are sent out. This will give an equivalent result to the NAT firewall people are used to and you can always add more rules to allow specific ports in.
ip6tables -A FORWARD -i ppp+ -m state --state ESTABLISHED,RELATED -j ACCEPT
ip6tables -A FORWARD -i ppp+ -i DROP

Russ Allbery: Review: The Horse and His Boy

Review: The Horse and His Boy, by C.S. Lewis
Illustrator: Pauline Baynes
Series: Chronicles of Narnia #5
Publisher: Collier Books
Copyright: 1954
Printing: 1978
ISBN: 0-02-044200-9
Format: Mass market
Pages: 217
The Horse and His Boy was the fifth published book in the Chronicles of Narnia, but it takes place during the last chapter of The Lion, the Witch and the Wardrobe, in the midst of the golden age of Narnia. It's the only true side story of the series and it doesn't matter much where in sequence you read it, as long as it's after The Lion, the Witch and the Wardrobe and before The Last Battle (which would spoil its ending somewhat). MAJOR SPOILERS BELOW. The Horse and His Boy is also the only book of the series that is not a portal fantasy. The Pevensie kids make an appearance, but as the ruling kings and queens of Narnia, and only as side characters. The protagonists are a boy named Shasta, a girl named Aravis, and horses named Bree and Hwin. Aravis is a Calormene, a native of the desert (and extremely Orientalist, but more on that later) kingdom to the south of Narnia and Archenland. Shasta starts the book as the theoretically adopted son but mostly slave of a Calormene fisherman. The Horse and His Boy is the story of their journey from Calormen north to Archenland and Narnia, just in time to defend Narnia and Archenland from an invasion. This story starts with a great hook. Shasta's owner is hosting a passing Tarkaan, a Calormene lord, and overhears a negotiation to sell Shasta to the Tarkaan as his slave (and, in the process, revealing that he rescued Shasta as an infant from a rowboat next to a dead man). Shasta starts talking to the Tarkaan's horse and is caught by surprise when the horse talks back. He is a Talking Horse from Narnia, kidnapped as a colt, and eager to return to Narnia and the North. He convinces Shasta to attempt to escape with him. This has so much promise. For once, we're offered a story where one of the talking animals of Narnia is at least a co-protagonist and has some agency in the story. Bree takes charge of Shasta, teaches him to ride (or, mostly, how to fall off a horse), and makes most of the early plans. Finally, a story that recognizes that Narnia stories don't have to revolve around the humans! Unfortunately, Bree is an obnoxious, arrogant character. I wanted to like him, but he makes it very hard. This gets even worse when Shasta is thrown together with Aravis, a noble Calormene girl who is escaping an arranged marriage on her own talking mare, Hwin. Bree is a warhorse, Hwin is a lady's riding mare, and Lewis apparently knows absolutely nothing about horses, because every part of Bree's sexist posturing and Hwin's passive meekness is awful and cringe-worthy. I am not a horse person, so will link to Judith Tarr's much more knowledgeable critique at Tor.com, but suffice it to say that mares are not meekly deferential or awed by stallions. If Bree had behaved that way with a real mare, he would have gotten the crap beaten out of him (which might have improved his attitude considerably). As is, we have to put up with rather a lot of Bree's posturing and Hwin (who I liked much better) barely gets a line and acts disturbingly like she was horribly abused. This makes me sad, because I like Bree's character arc. He's spent his whole life being special and different from those around him, and while he wants to escape this country and return home, he's also gotten used to being special. In Narnia, he will just be a normal talking horse. To get everything else he wants, he also has to let go of the idea that he's someone special. If Lewis had done more with this and made Bree a more sympathetic character, this could have been very effective. As written, it only gets a few passing mentions (mostly via Bree being weirdly obsessed with whether talking horses roll) and is therefore overshadowed by Shasta's chosen one story and Bree's own arrogant behavior. The horses aside, this is a passable adventure story with some well-done moments. The two kids and their horses end up in Tashbaan, the huge Calormene capital, where they stumble across the Narnians and Shasta is mistaken for one of their party. Radagast, the prince of Calormen, is proposing marriage to Susan, and the Narnians are in the process of realizing he doesn't plan to take no for an answer. Aravis, meanwhile, has to sneak out of the city via the Tisroc's gardens, which results in her hiding behind a couch as she hears Radagast's plans to invade Archenland and Narnia to take Susan as his bride by force. Once reunited, Shasta, Aravis, and the horses flee across the desert to bring warning to Archenland and then Narnia. Of all the Narnia books, The Horse and His Boy leans the hardest into the personal savior angle of Christianity. Parts of it, such as Shasta's ride over the pass into Narnia, have a strong "Footprints" feel to them. Most of the events of the book are arranged by Aslan, starting with Shasta's early life. Readers of the series will know this when a lion shows up early to herd the horses where they need to go, or when a cat keeps Shasta company in the desert and frightens away jackals. Shasta only understands near the end. I remember this being compelling stuff as a young Christian reader. This personal attention and life shaping from God is pure Christian wish fulfillment of the "God has a plan for your life" variety, even more so than Shasta turning out to be a lost prince. As an adult re-reader, I can see that Lewis is palming the theodicy card rather egregiously. It's great that Aslan was making everything turn out well in the end, but why did he have to scare the kids and horses half to death in the process? They were already eager to do what he wanted, but it's somehow inconceivable that Aslan would simply tell them what to do rather than manipulate them. There's no obvious in-story justification why he couldn't have made the experience much less terrifying. Or, for that matter, prevented Shasta from being kidnapped as an infant in the first place and solved the problem of Radagast in a more direct way. This sort of theology takes as an unexamined assumption that a deity must refuse to use his words and instead do everything in weirdly roundabout and mysterious ways, which makes even less sense in Narnia than in our world given how directly and straightforwardly Aslan has acted in previous books. It was also obvious to me on re-read how unfair Lewis's strict gender roles are to Aravis. She's an excellent rider from the start of the book and has practiced many of the things Shasta struggles to do, but Shasta is the boy and Aravis is the girl, so Aravis has to have girl adventures involving tittering princesses, luxurious baths, and eavesdropping behind couches, whereas Shasta has boy adventures like riding to warn the king or bringing word to Narnia. There's nothing very objectionable about Shasta as a character (unlike Bree), but he has such a generic character arc. The Horse and Her Girl with Aravis and Hwin as protagonists would have been a more interesting story, and would have helpfully complicated the whole Narnia and the North story motive. As for that storyline, wow the racism is strong in this one, starting with the degree that The Horse and His Boy is deeply concerned with people's skin color. Shasta is white, you see, clearly marking him as from the North because all the Calormenes are dark-skinned. (This makes even less sense in this fantasy world than in our world because it's strongly implied in The Magician's Nephew that all the humans in Calormen came from Narnia originally.) The Calormenes all talk like characters from bad translations of the Arabian Nights and are shown as cruel, corrupt slavers with a culture that's a Orientalist mishmash of Arab, Persian, and Chinese stereotypes. Everyone is required to say "may he live forever" after referencing the Tisroc, which is an obvious and crude parody of Islam. This stereotype fest culminates in the incredibly bizarre scene that Aravis overhears, in which the grand vizier literally grovels on the floor while Radagast kicks him and the Tisroc, Radagast's father, talks about how Narnia's freedom offends him and the barbarian kingdom would be more profitable and orderly when conquered. The one point to Lewis's credit is that Aravis is also Calormene, tells stories in the same style, and is still a protagonist and just as acceptable to Aslan as Shasta is. It's not enough to overcome the numerous problems with Lewis's lazy world-building, but it makes me wish even more that Aravis had gotten her own book and more meaningful scenes with Aslan. I had forgotten that Susan appears in this book, although that appearance doesn't add much to the general problem of Susan in Narnia except perhaps to hint at Lewis's later awful choices. She is shown considering marriage to the clearly villainous Radagast, and then only mentioned later with a weird note that she doesn't ride to war despite being the best archer of the four. I will say again that it's truly weird to see the Pevensie kids as (young) adults discussing marriage proposals, international politics, and border wars while remembering they all get dumped back into their previous lives as British schoolkids. This had to have had dramatic effects on their lives that Lewis never showed. (I know, the real answer is that Lewis is writing these books according to childhood imaginary adventure logic, where adventures don't have long-term consequences of that type.) I will also grumble once more at how weirdly ineffectual Narnians are until some human comes to tell them what to do. Calormen is obviously a threat; Susan just escaped from an attempted forced marriage. Archenland is both their southern line of defense and is an ally separated by a mountain pass in a country full of talking eagles, among other obvious messengers. And yet, it falls to Shasta to ride to give warning because he's the human protagonist of the story. Everyone else seems to be too busy with quirky domesticity or endless faux-medieval chivalric parties. The Horse and His Boy was one of my favorites when I was a kid, but reading as an adult I found it much harder to tolerate Bree or read past the blatant racial and cultural stereotyping. The bits with Aslan also felt less magical to me than they did as a kid because I was asking more questions about why Aslan had to do everything in such an opaque and perilous way. It's still not a bad adventure; Aravis is a great character, the bits in Tashbaan are at least memorable, and I still love the Hermit of the Southern March and want to know more about him. But I would rank it below the top tier of Narnia books, alongside Prince Caspian as a book with some great moments and some serious flaws. Followed in original publication order by The Magician's Nephew. Rating: 7 out of 10

31 May 2021

Russ Allbery: Review: The Relentless Moon

Review: The Relentless Moon, by Mary Robinette Kowal
Series: Lady Astronaut #3
Publisher: Tor
Copyright: 2020
ISBN: 1-250-23648-7
Format: Kindle
Pages: 542
Content note: Discussion of eating disorders in this review and portrayal of an eating disorder in the novel. The Relentless Moon is the third book of the Lady Astronaut series and the first one that doesn't feature Elma. It takes place simultaneously with The Fated Sky and tells the story of what happened on Earth, and the Moon, while Elma was in transit to Mars. It's meant to be read after The Fated Sky and would be a significant spoiler for that novel. The protagonist of this novel is Nicole Wargin: wife of the governor of Kansas (a more prestigious state in this universe since the seat of government for the United States was relocated to Kansas City after the Meteor), expert politician's wife, and another of the original group of female astronauts. Kenneth, her husband, is considering a run for president. Nicole is working as an astronaut, helping build out the permanent Moon base. But there are a lot of people on Earth who are not happy with the amount of money and political attention that the space program is getting. They have decided to move beyond protests and political opposition to active sabotage. Nicole was hoping to land an assignment piloting one of the larger rockets. What she gets instead is an assignment as secretary to the Lunar Colony Administrator, as cover. Her actual job is to watch for saboteurs that may or may not be operating on the Moon. Before she even leaves the planet, one of the chief engineers of the space program is poisoned. The pilot of the translunar shuttle falls ill during the flight to the Moon. And then the shuttle's controls fail during landing and disaster is only narrowly averted. The story from there is a cloak and dagger sabotage investigation mixed with Kowal's meticulously-researched speculation about a space program still running on 1950s technology but drastically accelerated by the upcoming climate collapse of Earth. Nicole has more skills for this type of mission than most around her realize due to very quiet work she did during the war, not to mention the feel for personalities and politics that she's honed as a governor's wife. But, like Elma, she's also fighting some personal battles. Elma's are against anxiety; Nicole's are against an eating disorder. I think my negative reaction to this aspect of the book is not the book's fault, but it was sufficiently strong that it substantially interfered with my enjoyment. The specific manifestation of Nicole's eating disorder is that she skips meals until she makes herself ill. My own anxious tendencies hyperfocus on prevention and on rule-following. The result is that once Kowal introduces the eating disorder subplot, my brain started anxiously monitoring everything that Nicole ate and keeping track of her last meal. This, in turn, felt horribly intrusive and uncomfortable. I did not want to monitor and police Nicole's eating, particularly when Nicole clearly was upset by anyone else doing exactly that, and yet I couldn't stop the background thread of my brain that did so. The result was a highly unsettling feeling that I was violating the privacy of the protagonist of the book that I was reading, mixed with anxiety and creeping dread about her calorie intake. Part of this may have been intentional to give the reader some sense of how this felt to Nicole. (The negative interaction with my own anxiety was likely not intentional.) Kowal did an exceptionally good job at invoking reader empathy (at least in me) for Elma's anxiety in The Calculating Stars. I didn't like the experience much this time, but that doesn't make it an invalid focus for a book. It may, however, make me a poor reviewer for this part of the reading experience. This was a major subplot, so it was hard to escape completely, but I quite enjoyed the rest of the book. It's not obvious who the saboteurs are or even how the sabotage is happening, and the acts of clear sabotage are complicated by other problems that may be more subtle sabotage, may be bad luck, or may be the inherent perils of trying to survive in space. Many of Nicole's suspicions do not pan out, which was a touch that I appreciated. She has to look for ulterior motives in everything, and in reality that means she'll be wrong most of the time, but fiction often unrealistically short-cuts that process. I also liked how Kowal handles the resolution, which avoids villain monologues and gives Nicole's opposition their own contingency plans, willingness to try to adapt to setbacks, and the intelligence to keep trying to manipulate the situation even when their plans fail. As with the rest of this series, there's a ton of sexism and racism, which the characters acknowledge and which Nicole tries to resist as much as she can, but which is so thoroughly baked into the society that it's mostly an obstacle that has to be endured. This is not the book, or series, to read if you're looking for triumph over discrimination or for women being allowed to be awesome without having to handle and soothe men's sexist feelings about their abilities. Nicole gets a clear victory arc, but it's a victory despite sexism rather than an end to it. The Relentless Moon did feel a bit long. There are a lot of scene-setting preliminaries before Nicole leaves for the Moon, and I'm not sure all of them were necessary at that length. Nicole also spends a lot of time being suspicious of everyone and second-guessing her theories, and at a few points I thought that dragged. But once things start properly happening, I thoroughly enjoyed the technological details and the thought that Kowal put into the mix of sabotage, accidents, and ill-advised human behavior that Nicole has to sort through. The last half of the book is the best, which is always a good property for a book to have. The eating disorder subplot made me extremely uncomfortable for reasons that are partly peculiar to me, but outside of that, this is a solid entry in the series and fills in some compelling details of what was happening on the other end of the intermittent radio messages Elma received. If you've enjoyed the series to date, you will probably enjoy this installment as well. But if you didn't like the handling of sexism and racism as deeply ingrained social forces that can at best be temporarily bypassed, be warned that The Relentless Moon continues the same theme. Also, if you're squeamish about medical conditions in your fiction, be aware that the specific details of polio feature significantly in the book. Rating: 7 out of 10

30 May 2021

Russ Allbery: Review: The Silver Chair

Review: The Silver Chair, by C.S. Lewis
Illustrator: Pauline Baynes
Series: Chronicles of Narnia #4
Publisher: Collier
Copyright: 1953
Printing: 1978
ISBN: 0-02-044250-5
Format: Mass market
Pages: 217
The Silver Chair is a sequel to The Voyage of the Dawn Treader and the fourth book of the Chronicles of Narnia in original publication order. (For more about publication order, see the introduction to my review of The Lion, the Witch and the Wardrobe.) Apart from a few references to The Voyage of the Dawn Treader at the start, it stands sufficiently on its own that you could read it without reading the other books, although I have no idea why you'd want to. We have finally arrived at my least favorite of the Narnia books and the one that I sometimes skipped during re-reads. (One of my objections to the new publication order is that it puts The Silver Chair and The Last Battle back-to-back, and I don't think you should do that to yourself as a reader.) I was hoping that there would be previously unnoticed depth to this book that would redeem it as an adult reader. Sadly, no; with one very notable exception, it's just not very good. MAJOR SPOILERS BELOW. The Silver Chair opens on the grounds of the awful school to which Eustace's parents sent him: Experiment House. That means it opens (and closes) with a more extended version of Lewis's rant about schools. I won't get into this in detail since it's mostly a framing device, but Lewis is remarkably vicious and petty. His snide contempt for putting girls and boys in the same school did not age well, nor did his emphasis at the end of the book that the incompetent head of the school is a woman. I also raised an eyebrow at holding up ordinary British schools as a model of preventing bullying. Thankfully, as Lewis says at the start, this is not a school story. This is prelude to Jill meeting Eustace and the two of them escaping the bullies via a magical door into Narnia. Unfortunately, that's the second place The Silver Chair gets off on the wrong foot. Jill and Eustace end up in what the reader of the series will recognize as Aslan's country and almost walk off the vast cliff at the end of the world, last seen from the bottom in The Voyage of the Dawn Treader. Eustace freaks out, Jill (who has a much better head for heights) goes intentionally close to the cliff in a momentary impulse of arrogance before realizing how high it is, Eustace tries to pull her back, and somehow Eustace falls over the edge. I do not have a good head for heights, and I wonder how much of it is due to this memorable scene. I certainly blame Lewis for my belief that pulling someone else back from the edge of a cliff can result in you being pushed off, something that on adult reflection makes very little sense but which is seared into my lizard brain. But worse, this sets the tone for the rest of the story: everything is constantly going wrong because Eustace and Jill either have normal human failings that are disproportionately punished or don't successfully follow esoteric and unreasonably opaque instructions from Aslan. Eustace is safe, of course; Aslan blows him to Narnia and then gives Jill instructions before sending her afterwards. (I suspect the whole business with the cliff was an authorial cheat to set up Jill's interaction with Aslan without Eustace there to explain anything.) She and Eustace have been summoned to Narnia to find the lost Prince, and she has to memorize four Signs that will lead her on the right path. Gah, the Signs. If you were the sort of kid that I was, you immediately went back and re-read the Signs several times to memorize them like Jill was told to. The rest of this book was then an exercise in anxious frustration. First, Eustace is an ass to Jill and refuses to even listen to the first Sign. They kind of follow the second but only with heavy foreshadowing that Jill isn't memorizing the Signs every day like she's supposed to. They mostly botch the third and have to backtrack to follow it. Meanwhile, the narrator is constantly reminding you that the kids (and Jill in particular) are screwing up their instructions. On re-reading, it's clear they're not doing that poorly given how obscure the Signs are, but the ominous foreshadowing is enough to leave a reader a nervous wreck. Worse, Eustace and Jill are just miserable to each other through the whole book. They constantly bicker and snipe, Eustace doesn't want to listen to her and blames her for everything, and the hard traveling makes it all worse. Lewis does know how to tell a satisfying redemption arc; one of the things I have always liked about Edmund's story is that he learns his lesson and becomes my favorite character in the subsequent stories. But, sadly, Eustace's redemption arc is another matter. He's totally different here than he was at the start of The Voyage of the Dawn Treader (to the degree that if he didn't have the same name in both books, I wouldn't recognize him as the same person), but rather than a better person he seems to have become a different sort of ass. There's no sign here of the humility and appreciation for friendship that he supposedly learned from his time as a dragon. On top of that, the story isn't very interesting. Rilian, the lost Prince, is a damp squib who talks in the irritating archaic accent that Lewis insists on using for all Narnian royalty. His story feels like Lewis lifted it from medieval Arthurian literature; most of it could be dropped into a collection of stories of knights of the Round Table without seeming out of place. When you have a country full of talking animals and weirdly fascinating bits of theology, it's disappointing to get a garden-variety story about an evil enchantress in which everyone is noble and tragic and extremely stupid. Thankfully, The Silver Chair has one important redeeming quality: Puddleglum. Puddleglum is a Marsh-wiggle, a bipedal amphibious sort who lives alone in the northern marshes. He's recruited by the owls to help the kids with their mission when they fail to get King Caspian's help after blowing the first Sign. Puddleglum is an absolute delight: endlessly pessimistic, certain the worst possible thing will happen at any moment, but also weirdly cheerful about it. I love Eeyore characters in general, but Puddleglum is even better because he gives the kids' endless bickering exactly the respect that it deserves.
"But we all need to be very careful about our tempers, seeing all the hard times we shall have to go through together. Won't do to quarrel, you know. At any rate, don't begin it too soon. I know these expeditions usually end that way; knifing one another, I shouldn't wonder, before all's done. But the longer we can keep off it "
It's even more obvious on re-reading that Puddleglum is the only effective member of the party. Jill has only a couple of moments where she gets the three of them past some obstacle. Eustace is completely useless; I can't remember a single helpful thing he does in the entire book. Puddleglum and his pessimistic determination, on the other hand, is right about nearly everything at each step. And he's the one who takes decisive action to break the Lady of the Green Kirtle's spell near the end. I was expecting a bit of sexism and (mostly in upcoming books) racism when re-reading these books as an adult given when they were written and who Lewis was, but what has caught me by surprise is the colonialism. Lewis is weirdly insistent on importing humans from England to fill all the important roles in stories, even stories that are entirely about Narnians. I know this is the inherent weakness of portal fantasy, but it bothers me how little Lewis believes in Narnians solving their own problems. The Silver Chair makes this blatantly obvious: if Aslan had just told Puddleglum the same information he told Jill and sent a Badger or a Beaver or a Mouse along with him, all the evidence in the book says the whole affair would have been sorted out with much less fuss and anxiety. Jill and Eustace are far more of a hindrance than a help, which makes for frustrating reading when they're supposedly the protagonists. The best part of this book is the underground bits, once they finally get through the first three Signs and stumble into the Lady's kingdom far below the surface. Rilian is a great disappointment, but the fight against the Lady's mind-altering magic leads to one of the great quotes of the series, on par with Reepicheep's speech in The Voyage of the Dawn Treader.
"Suppose we have only dreamed, or made up, all those things trees and grass and sun and moon and stars and Aslan himself. Suppose we have. Then all I can say is that, in that case, the made-up things seem a good deal more important than the real ones. Suppose this black pit of a kingdom of yours is the only world. Well, it strikes me as a pretty poor one. And that's a funny thing, when you come to think of it. We're just babies making up a game, if you're right. But four babies playing a game can make a play-world which licks your real world hollow. That's why I'm going to stand by the play world. I'm on Aslan's side even if there isn't any Aslan to lead it. I'm going to live as like a Narnian as I can even if there isn't any Narnia. So, thanking you kindly for our supper, if these two gentlemen and the young lady are ready, we're leaving your court at once and setting out in the dark to spend our lives looking for Overland. Not that our lives will be very long, I should think; but that's small loss if the world's as dull a place as you say."
This is Puddleglum, of course. And yes, I know that this is apologetics and Lewis is talking about Christianity and making the case for faith without proof, but put that aside for the moment, because this is still powerful life philosophy. It's a cynic's litany against cynicism. It's a pessimist's defense of hope. Suppose we have only dreamed all those things like justice and fairness and equality, community and consensus and collaboration, universal basic income and effective environmentalism. The dreary magic of the realists and the pragmatists say that such things are baby's games, silly fantasies. But you can still choose to live like you believe in them. In Alasdair Gray's reworking of a line from Dennis Lee, "work as if you live in the early days of a better nation." That's one moment that I'll always remember from this book. The other is after they kill the Lady of the Green Kirtle and her magic starts to fade, they have to escape from the underground caverns while surrounded by the Earthmen who served her and who they believe are hostile. It's a tense moment that turns into a delightful celebration when they realize that the Earthmen were just as much prisoners as the Prince was. They were forced from a far deeper land below, full of living metals and salamanders who speak from rivers of fire. It's the one moment in this book that I thought captured the magical strangeness of Narnia, that sense that there are wonderful things just out of sight that don't follow the normal patterns of medieval-ish fantasy. Other than a few great lines from Puddleglum and some moments in Aslan's country, the first 60% of this book is a loss and remarkably frustrating to read. The last 40% isn't bad, although I wish Rilian had any discernible character other than generic Arthurian knight. I don't know what Eustace is doing in this book at all other than providing a way for Jill to get into Narnia, and I wish Lewis had realized Puddleglum could be the protagonist. But as frustrating as The Silver Chair can be, I am still glad I re-read it. Puddleglum is one of the truly memorable characters of children's literature, and it's a shame he's buried in a weak mid-series book. Followed, in the original publication order, by The Horse and His Boy. Rating: 6 out of 10

29 May 2021

Shirish Agarwal: Planes, Pandemic and Medical Devices I

The Great Electric Airplane Race It took me quite sometime to write as have been depressed about things. Then a few days back saw Nova s The Great Electric Airplane Race. While it was fabulous and a pleasure to see and know that there are more than 200 odd startups who are in the race of making an electric airplane which works and has FAA certification. I was disappointed though that there no coverage of any University projects. From what little I know, almost all advanced materials which U.S. had made has been first researched in mostly Universities and when it is close to fruition then either spin-off as a startup or give to some commercial organization/venture to make it scalable and profitable. If they had, I am sure more people could be convinced to join sciences and engineering in college. I actually do want to come to this as part of both general medicine and vaccine development in U.S. but will come later. The idea that industry works alone should be discouraged, but that perhaps may require another article to articulate why I believe so.

Medical Device Ventilators in India Before the pandemic, probably most didn t know what a ventilator is and was, at least I didn t, although I probably used it during my somewhat brief hospital stay a couple of years ago. It entered into the Indian twitter lexicon more so in the second wave as the number of people who got infected became more and more and the ventilators which were serving them became less and less just due to sheer mismatch of numbers and requirements. Rich countries donated/gifted ventilators to India on which GOI put GST of 28%. Apparently, they are a luxury item, just like my hearing aid. Last week Delhi High Court passed a judgement that imposition of GST should not be on a gift like ventilators or oxygenators. The order can be found here. Even without reading the judgement the shout from the right was judicial activism while after reading it is a good judgement which touches on several points. The first, in itself, stating the dichotomy that if a commercial organization wanted to import a ventilator or an oxygenator the IGST payable is nil while for an individual it is 12%. The State (here State refers to State Government in this case Gujarat Govt.) did reduce the IGST for state from 12% to NIL IGST for federal states but that to till only 30.06.2021. No relief to individuals on that account. The Court also made use of Mr. Arvind Datar, as Amicus Curiae or friend of court. The petitioner, an 85-year-old gentleman who has put it up has put broad assertions under Article 21 (right to live) and the court in its wisdom also added Article 14 which enshrines equality of everyone before law. The Amicus Curiae, as his duty, guided the court into how the IGST law works and shared a brief history of the law and the changes happening before and after it. During his submissions, he also shared the Mega Exemption Notification no. 50/2017 under which several items are there which are exempted from putting IGST. The Amicus Curiae did note that such exemptions were also there before Mega Exemption Notification had come into play. However, DGFT (Directorate General of Foreign Trade) on 30-04-2021 issued notification No. 4/2015-2020 through which oxygenators had been exempted from Custom Duty/BCD (Basic Customs Duty. In another notification on no. 30/2021 dated 01.05.2021 it reduced IGST from 28% to 12% for personal use. If however the oxygenator was procured by a canalizing agency (bodies such as State Trading Corporation of India (STC) or/and Metals and Minerals Corporation (MMTC) and such are defined as canalising agents) then it will be fully exempted from paying any sort of IGST, albeit subject to certain conditions. What the conditions are were not shared in the open court. The Amicus Curiae further observed that it is contrary to practice where both BCD and IGST has been exempted for canalising agents and others, some IGST has to be paid for personal use. To share within the narrow boundaries of the topic, he shared entry no. 607A of General Exemption no.190 where duty and IGST in case of life-saving drugs are zero provided the life-saving drugs imported have been provided by zero cost from an overseas supplier for personal use. He further shared that the oxygen generator would fall in the same entry of 607A as it fulfills all the criteria as shared for life-saving medicines and devices. He also used the help of Drugs and Cosmetics Act 1940 which provides such a relief. The Amicus Curiae further noted that GOI amended its foreign trade policy (2015-2020) via notification no.4/2015-2020, dated 30.04.2021, issued by DGFT where Rakhi and life-saving drugs for personal use has been exempted from BCD till 30-07-2021. No reason not to give the same exemption to oxygenators which fulfill the same thing. The Amicus Curiae, further observes that there are exceptional circumstances provisions as adverted to in sub-section (2) of Section 25 of the Customs Act, whereby Covid-19 which is known and labelled as a pandemic where the distinctions between the two classes of individuals or agencies do not make any sense. While he did make the observation that exemption from duty is not a right, in the light of the pandemic and Article 14, it does not make sense to have distinctions between the two classes of importers. He further shared from Circular no. 9/2014-Customs, dated 19.08.2014 by CBEC (Central Board of Excise and Customs) which gave broad exemptions under Section 25 (2) of the same act in respect of goods and services imported for safety and rehabilitation of people suffering and effected by natural disasters and epidemics. He further submits that the impugned notification is irrational as there is no intelligible differentia rule applied or observed in classifying the import of oxygen concentrators into two categories. One, by the State and its agencies; and the other, by an individual for personal use by way of gift. So there was an absence of adequate determining principle . To bolster his argument, he shared the judgements of

a) Union of India vs. N.S. Rathnam & Sons, (2015) 10 SCC 681 (N.S. Ratnams and Sons Case) b) Shayara Bano vs. Union of India, (2017) 9 SCC 1 (Shayara Bano Case) The Amicus Curiae also rightly observed that the right to life also encompasses within it, the right to health. You cannot have one without the other and within that is the right to have affordable treatment. He further stated that the state does not only have a duty but a positive obligation is cast upon it to ensure that the citizen s health is secured. He again cited Navtej Singh Johars vs Union of India (Navtej Singh Johar Case) in defence of right to life. Mr. Datar also shared that unlike in normal circumstances, it is and should be enough to show distinct and noticeable burdensomeness which is directly attributable to the impugned/questionable tax. The gentleman cited Indian Express Newspapers (Bombay) Private Limited vs. Union of India, (1985) 1 SCC 641 (Indian Express case) 1985 which shared both about Article 19 (1) (a) and Article 21. Bloggers note At this juncture, I should point out which I am sharing the judgement and I would be sharing only the Amicus Curiae POV and then the judge s final observations. While I was reading it, I was stuck by the fact that the Amicus Curiae had cited 4 cases till now, 3 of them are pretty well known both in the legal fraternity and even among public at large. Another 3 which have been shared below which are also of great significance. Hence, felt the need to share the whole judgement. The Amicus Curiae further observed that this tax would have to be disproportionately will have to be paid by the old and the infirm, and they might find it difficult to pay the amounts needed to pay the customs duty/IGST as well as find the agent to pay in this pandemic. Blogger Note The situation with the elderly is something like this. Now there are a few things to note, only Central Govt. employees and pensioners get pensions which has been freezed since last year. The rest of the elderly population does not. The rate of interest has fallen to record lows from 5-6% in savings interest rate to 2% and on Fixed Deposits at 4.9% while the nominal inflation rate has up by 6% while CPI and real inflation rates are and would be much more. And this is when there is absolutely no demand in the economy. To add to all this, RBI shared a couple of months ago that fraud of 5 trillion rupees has been committed between 2015 and 2019 in banks. And this is different from the number of record NPA s that have been both in Public and Private Sector banks. To get out of this, the banks have squeezed their customers and are squeezing as well as asking GOI for bailouts. How much GOI is responsible for the frauds as well as NPA s would probably require its own space. And even now, RBI and banks have made heavy provisions as lockdowns are still a facet and are supposed to remain a facet till the end of the year or even next year (all depending upon when we get the vaccine). The Amicus Curiae further argued that the ventilators which are available locally are of bad quality. The result of all this has resulted in a huge amount of unsurmountable pressure on hospitals which they are unable to overcome. Therefore, the levy of IGST on oxygenators has direct impact on health of the citizen. So the examination of the law should not be by what intention it was but how it is affecting citizen rights now. For this he shared R.C.Cooper vs Union of India (another famous case R.C. Cooper vs Union of India) especially paragraph 49 and Federation of Hotel & Restaurant Association of India vs. Union of India, (1989) at paragraph 46 (Federation of Hotel Case) Mr. Datar further shared the Supreme Court order dated 18.12.2020, passed in Suo Moto Writ Petition(Civil) No.7/2020, to buttress the plea that the right to health includes the right to affordable treatment. Blogger s Note For those, who don t know Suo Moto is when the Court, whether Supreme Court or the High Courts take up a matter for public good. It could be in anything, law and order, Banking, Finance, Public Health etc. etc. This was the norm before 2014. The excesses of the executive were curtailed by both the Higher and the lower Judiciary. That is and was the reason that Judiciary is and was known as the third pillar of Indian democracy. A good characterization of Suo Moto can be found here. Before ending his submission, the learned Amicus Curiae also shared Jeeja Ghosh vs. Union of India, (2016) (Jeeja Ghosh Case, an outstanding case as it deals with people with disabilities and their rights and the observations made by the Division Bench of Hon ble Mr. Justice A. K. Sikri as well as Hon ble Mr. Justice R. K. Agrawal.) After Amicus Curiae completed his submissions, it was the turn of Mr. Sudhir Nandrajog, and he adopted the arguments and submissions made by the Amicus Curiae. The gentleman reiterated the facts of the case and how the impugned notification was violative of both Article 14 and 21 of the Indian Constitution. Blogger s Note The High Court s judgement which shows all the above arguments by the Amicus Curiae and the petitioner s lawyer also shared the State s view. It is only on page 24, where the Delhi High Court starts to share its own observations on the arguments of both sides. Judgement continued The first observation that the Court makes is that while the petitioner demonstrated that the impugned tax imposition would have a distinct and noticeable burdensomeness while the State did not state or share in any way how much of a loss it would incur if such a tax were let go and how much additional work would have to be done in order to receive this specific tax. It didn t need to do something which is down the wire or mathematically precise, but it didn t even care to show even theoretically how many people will be affected by the above. The counter-affidavit by the State is silent on the whole issue. The Court also contended that the State failed to prove how collecting IGST from the concerned individuals would help in fighting coronavirus in any substantial manner for the public at large. The High Court shared observations from the Navtej Singh Johar case where it is observed that the State has both negative and positive obligations to ensure that its citizens are able to enjoy the right to health. The High Court further made the point that no respectable person does like to be turned into a charity case. If the State contends that those who obey the law should pay the taxes then it is also obligatory on the state s part to lessen exactions such as taxes at the very least in times of war, famine, floods, epidemics and pandemics. Such an approach would lead a person to live a life of dignity which is part of Article 21 of the Constitution. Another point that was made by the State that only the GST council is able to make any changes as regards to exemptions rather than the State were found to be false as the State had made some exemptions without going to the GST council using its own powers under Section 25 of the Customs Act. The Court also points out that it does send a discriminatory pattern when somebody like petitioner has to pay the tax for personal use while those who are buying it for commercial use do not have to pay the tax. The Court agreed of the view of the Amicus Curiae, Mr. Datar that oxygenator should be taxed at NIL rate at IGST as it is part of life-saving drugs and oxygenator fits the bill as medical equipment as it is used in the treatment, mitigation and prevention of spread of Coronavirus. Mr. Datar also did show that oxygenator is placed at the same level as other life-saving drugs. The Court felt further emboldened as the observations by Supreme Court in State of Andhra Pradesh vs. Linde India Limited, 2020 ( State of Andhra Pradesh vs Linde Ltd.) The Court further shared many subsequent notifications from the State and various press releases by the State itself which does make the Court s point that oxygenators indeed are drugs as defined in the court case above. The State should have it as part of notification 190. This would preserve the start of the notification date from 03.05.2021 and the state would not have to issue a new notification. The Court further went to postulate that any persons similar to the petitioner could avail of the same, if they furnish a letter of undertaking to an officer designated by the State that the medical equipment would not be put to commercial use. Till the state does not do that, in the interim the importer could give the same undertaking to Joint Secretary, Customs or their nominee can hand over the same to custom officer. The Court also shared that it does not disagree with the State s arguments but the challenges which have arisen are in a unique time period/circumstances, so they are basing their judgement based on how the situation is. The Court also mentioned an order given by Supreme Court Diary No. 10669/2020 passed on 20.03.2020 where SC has taken pains to understand the issues faced by the citizens. The court also mentioned the Small Scale Industrial Manufactures Association Case (both of these cases I don t know) . So in conclusion, the Court holds the imposition of IGST on oxygenator which are imported by individuals as gifts from their relatives as unconstitutional. They also shared that any taxes taken by GOI in above scenario have to be returned. The relief to the state is they will not have to pay interest cost on the same. To check misuse of the same, the petitioner or people who are in similar circumstances would have to give a letter of undertaking to an officer designated by the State within 7 days of the state notifying the patient or anybody authorized by him/her to act on their behalf to share the letter of undertaking with the State. And till the State doesn t provide an officer, the above will continue. Hence, both the writ petition and the pending application are disposed off. The Registry is directed to release any money deposited by the petitioner along with any interest occurred on it (if any) . At the end they record appreciation of Mr. Arvind Datar, Mr. Zoheb Hossain, Mr. Sudhir Nandrajog as well as Mr. Siddharth Bambha. It is only due to their assistance that the court could reach the conclusion it did. For Delhi High Court RAJIV SHAKDHER, J. TALWANT SINGH, J. May 21, 2020 Blogger s Observations Now, after the verdict GOI does have few choices, either accept the verdict or appeal in the SC. A third choice is to make a committee and come to the same conclusions via the committee. GOI has done something similar in the past. If that happens and the same conclusions are reached as before, then the aggrieved may have no choice but to appear in the highest court of law. And this will put the aggrieved at a much more vulnerable place than before as SC court fees, lawyer fees etc. are quite high compared to High Courts. So, there is a possibility that the petitioner may not even approach unless and until some non-profit (NGO) does decide to fight and put it up as common cause or something similar. There is another judgement that I will share, probably tomorrow. Thankfully, that one is pretty short compared to this one. So it should be far more easier to read. FWIW, I did learn about the whole freeenode stuff and many channels who have shifted from freenode to libera. I will share my own experience of the same but that probably will take a day or two.
Zeeshan of IYC (India Youth Congress) along with Salman Khan s non-profit Being Human getting oxygenators
The above picture of Zeeshan. There have been a whole team of Indian Youth Congress workers (main opposition party to the ruling party) who have doing lot of relief effort. They have been buying Oxygenators from abroad with help of Being Human Foundation started by Salman Khan, an actor who works in A-grade movies in Bollywood.

28 May 2021

Jonathan McDowell: Trying to understand Kubernetes networking

I previously built a single node Kubernetes cluster as a test environment to learn more about it. The first thing I want to try to understand is its networking. In particular the IP addresses that are listed are all 10.* and my host s network is a 192.168/24. I understand each pod gets its own virtual ethernet interface and associated IP address, and these are generally private within the cluster (and firewalled out other than for exposed services). What does that actually look like?
$ ip route
default via 192.168.53.1 dev enx00e04c6851de
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
192.168.0.0/24 dev weave proto kernel scope link src 192.168.0.1
192.168.53.0/24 dev enx00e04c6851de proto kernel scope link src 192.168.53.147
Huh. No sign of any way to get to 10.107.66.138 (the IP my echoserver from the previous post is available on directly from the host). What about network interfaces? (under the cut because it s lengthy)
ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: enx00e04c6851de: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:e0:4c:68:51:de brd ff:ff:ff:ff:ff:ff
    inet 192.168.53.147/24 brd 192.168.53.255 scope global dynamic enx00e04c6851de
       valid_lft 41571sec preferred_lft 41571sec
3: wlp1s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 74:d8:3e:70:3b:18 brd ff:ff:ff:ff:ff:ff
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:18:04:9e:08 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
5: datapath: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether d2:5a:fd:c1:56:23 brd ff:ff:ff:ff:ff:ff
7: weave: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue state UP group default qlen 1000
    link/ether 12:82:8f:ed:c7:bf brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.1/24 brd 192.168.0.255 scope global weave
       valid_lft forever preferred_lft forever
9: vethwe-datapath@vethwe-bridge: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue master datapath state UP group default
    link/ether b6:49:88:d6:6d:84 brd ff:ff:ff:ff:ff:ff
10: vethwe-bridge@vethwe-datapath: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue master weave state UP group default
    link/ether 6e:6c:03:1d:e5:0e brd ff:ff:ff:ff:ff:ff
11: vxlan-6784: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65535 qdisc noqueue master datapath state UNKNOWN group default qlen 1000
    link/ether 9a:af:c5:0a:b3:fd brd ff:ff:ff:ff:ff:ff
13: vethwepl534c0a6@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue master weave state UP group default
    link/ether 1e:ac:f1:85:61:9a brd ff:ff:ff:ff:ff:ff link-netnsid 0
15: vethwepl9ffd6b6@if14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue master weave state UP group default
    link/ether 56:ca:71:2a:ab:39 brd ff:ff:ff:ff:ff:ff link-netnsid 1
17: vethwepl62b369d@if16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue master weave state UP group default
    link/ether e2:a0:bb:ee:fc:73 brd ff:ff:ff:ff:ff:ff link-netnsid 2
23: vethwepl6669168@if22: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue master weave state UP group default
    link/ether f2:e7:e6:95:e0:61 brd ff:ff:ff:ff:ff:ff link-netnsid 3
That looks like a collection of virtual ethernet devices that are being managed by the weave networking plugin, and presumably partnered inside each pod. They re bridged to the weave interface (the master weave bit). Still no clues about the 10.* range. What about ARP?
ip neigh
192.168.53.1 dev enx00e04c6851de lladdr e4:8d:8c:35:98:d5 DELAY
192.168.0.4 dev datapath lladdr da:22:06:96:50:cb STALE
192.168.0.2 dev weave lladdr 66:eb:ce:16:3c:62 REACHABLE
192.168.53.136 dev enx00e04c6851de lladdr 00:e0:4c:39:f2:54 REACHABLE
192.168.0.6 dev weave lladdr 56:a9:f0:d2:9e:f3 STALE
192.168.0.3 dev datapath lladdr f2:42:c9:c3:08:71 STALE
192.168.0.3 dev weave lladdr f2:42:c9:c3:08:71 REACHABLE
192.168.0.2 dev datapath lladdr 66:eb:ce:16:3c:62 STALE
192.168.0.6 dev datapath lladdr 56:a9:f0:d2:9e:f3 STALE
192.168.0.4 dev weave lladdr da:22:06:96:50:cb STALE
192.168.0.5 dev datapath lladdr fe:6f:1b:14:56:5a STALE
192.168.0.5 dev weave lladdr fe:6f:1b:14:56:5a REACHABLE
Nope. That just looks like addresses on the weave managed bridge. Alright. What about firewalling?
nft list ruleset
table ip nat  
	chain DOCKER  
		iifname "docker0" counter packets 0 bytes 0 return
	 
	chain POSTROUTING  
		type nat hook postrouting priority srcnat; policy accept;
		 counter packets 531750 bytes 31913539 jump KUBE-POSTROUTING
		oifname != "docker0" ip saddr 172.17.0.0/16 counter packets 1 bytes 84 masquerade 
		counter packets 525600 bytes 31544134 jump WEAVE
	 
	chain PREROUTING  
		type nat hook prerouting priority dstnat; policy accept;
		 counter packets 180 bytes 12525 jump KUBE-SERVICES
		fib daddr type local counter packets 23 bytes 1380 jump DOCKER
	 
	chain OUTPUT  
		type nat hook output priority -100; policy accept;
		 counter packets 527005 bytes 31628455 jump KUBE-SERVICES
		ip daddr != 127.0.0.0/8 fib daddr type local counter packets 285425 bytes 17125524 jump DOCKER
	 
	chain KUBE-MARK-DROP  
		counter packets 0 bytes 0 meta mark set mark or 0x8000 
	 
	chain KUBE-MARK-MASQ  
		counter packets 0 bytes 0 meta mark set mark or 0x4000 
	 
	chain KUBE-POSTROUTING  
		mark and 0x4000 != 0x4000 counter packets 4622 bytes 277720 return
		counter packets 0 bytes 0 meta mark set mark xor 0x4000 
		 counter packets 0 bytes 0 masquerade 
	 
	chain KUBE-KUBELET-CANARY  
	 
	chain INPUT  
		type nat hook input priority 100; policy accept;
	 
	chain KUBE-PROXY-CANARY  
	 
	chain KUBE-SERVICES  
		meta l4proto tcp ip daddr 10.96.0.10  tcp dport 9153 counter packets 0 bytes 0 jump KUBE-SVC-JD5MR3NA4I4DYORP
		meta l4proto tcp ip daddr 10.107.66.138  tcp dport 8080 counter packets 1 bytes 60 jump KUBE-SVC-666FUMINWJLRRQPD
		meta l4proto tcp ip daddr 10.111.16.129  tcp dport 443 counter packets 0 bytes 0 jump KUBE-SVC-EZYNCFY2F7N6OQA2
		meta l4proto tcp ip daddr 10.96.9.41  tcp dport 443 counter packets 0 bytes 0 jump KUBE-SVC-EDNDUDH2C75GIR6O
		meta l4proto tcp ip daddr 192.168.53.147  tcp dport 443 counter packets 0 bytes 0 jump KUBE-XLB-EDNDUDH2C75GIR6O
		meta l4proto tcp ip daddr 10.96.9.41  tcp dport 80 counter packets 0 bytes 0 jump KUBE-SVC-CG5I4G2RS3ZVWGLK
		meta l4proto tcp ip daddr 192.168.53.147  tcp dport 80 counter packets 0 bytes 0 jump KUBE-XLB-CG5I4G2RS3ZVWGLK
		meta l4proto tcp ip daddr 10.96.0.1  tcp dport 443 counter packets 0 bytes 0 jump KUBE-SVC-NPX46M4PTMTKRN6Y
		meta l4proto udp ip daddr 10.96.0.10  udp dport 53 counter packets 0 bytes 0 jump KUBE-SVC-TCOU7JCQXEZGVUNU
		meta l4proto tcp ip daddr 10.96.0.10  tcp dport 53 counter packets 0 bytes 0 jump KUBE-SVC-ERIFXISQEP7F7OF4
		 fib daddr type local counter packets 3312 bytes 198720 jump KUBE-NODEPORTS
	 
	chain KUBE-NODEPORTS  
		meta l4proto tcp  tcp dport 31529 counter packets 0 bytes 0 jump KUBE-MARK-MASQ
		meta l4proto tcp  tcp dport 31529 counter packets 0 bytes 0 jump KUBE-SVC-666FUMINWJLRRQPD
		meta l4proto tcp ip saddr 127.0.0.0/8  tcp dport 30894 counter packets 0 bytes 0 jump KUBE-MARK-MASQ
		meta l4proto tcp  tcp dport 30894 counter packets 0 bytes 0 jump KUBE-XLB-EDNDUDH2C75GIR6O
		meta l4proto tcp ip saddr 127.0.0.0/8  tcp dport 32740 counter packets 0 bytes 0 jump KUBE-MARK-MASQ
		meta l4proto tcp  tcp dport 32740 counter packets 0 bytes 0 jump KUBE-XLB-CG5I4G2RS3ZVWGLK
	 
	chain KUBE-SVC-NPX46M4PTMTKRN6Y  
		 counter packets 0 bytes 0 jump KUBE-SEP-Y6PHKONXBG3JINP2
	 
	chain KUBE-SEP-Y6PHKONXBG3JINP2  
		ip saddr 192.168.53.147  counter packets 0 bytes 0 jump KUBE-MARK-MASQ
		meta l4proto tcp   counter packets 0 bytes 0 dnat to 192.168.53.147:6443
	 
	chain WEAVE  
		# match-set weaver-no-masq-local dst  counter packets 135966 bytes 8160820 return
		ip saddr 192.168.0.0/24 ip daddr 224.0.0.0/4 counter packets 0 bytes 0 return
		ip saddr != 192.168.0.0/24 ip daddr 192.168.0.0/24 counter packets 0 bytes 0 masquerade 
		ip saddr 192.168.0.0/24 ip daddr != 192.168.0.0/24 counter packets 33 bytes 2941 masquerade 
	 
	chain WEAVE-CANARY  
	 
	chain KUBE-SVC-JD5MR3NA4I4DYORP  
		  counter packets 0 bytes 0 jump KUBE-SEP-6JI23ZDEH4VLR5EN
		 counter packets 0 bytes 0 jump KUBE-SEP-FATPLMAF37ZNQP5P
	 
	chain KUBE-SEP-6JI23ZDEH4VLR5EN  
		ip saddr 192.168.0.2  counter packets 0 bytes 0 jump KUBE-MARK-MASQ
		meta l4proto tcp   counter packets 0 bytes 0 dnat to 192.168.0.2:9153
	 
	chain KUBE-SVC-TCOU7JCQXEZGVUNU  
		  counter packets 0 bytes 0 jump KUBE-SEP-JTN4UBVS7OG5RONX
		 counter packets 0 bytes 0 jump KUBE-SEP-4TCKAEJ6POVEFPVW
	 
	chain KUBE-SEP-JTN4UBVS7OG5RONX  
		ip saddr 192.168.0.2  counter packets 0 bytes 0 jump KUBE-MARK-MASQ
		meta l4proto udp   counter packets 0 bytes 0 dnat to 192.168.0.2:53
	 
	chain KUBE-SVC-ERIFXISQEP7F7OF4  
		  counter packets 0 bytes 0 jump KUBE-SEP-UPZX2EM3TRFH2ASL
		 counter packets 0 bytes 0 jump KUBE-SEP-KPHYKKPVMB473Z76
	 
	chain KUBE-SEP-UPZX2EM3TRFH2ASL  
		ip saddr 192.168.0.2  counter packets 0 bytes 0 jump KUBE-MARK-MASQ
		meta l4proto tcp   counter packets 0 bytes 0 dnat to 192.168.0.2:53
	 
	chain KUBE-SEP-4TCKAEJ6POVEFPVW  
		ip saddr 192.168.0.3  counter packets 0 bytes 0 jump KUBE-MARK-MASQ
		meta l4proto udp   counter packets 0 bytes 0 dnat to 192.168.0.3:53
	 
	chain KUBE-SEP-KPHYKKPVMB473Z76  
		ip saddr 192.168.0.3  counter packets 0 bytes 0 jump KUBE-MARK-MASQ
		meta l4proto tcp   counter packets 0 bytes 0 dnat to 192.168.0.3:53
	 
	chain KUBE-SEP-FATPLMAF37ZNQP5P  
		ip saddr 192.168.0.3  counter packets 0 bytes 0 jump KUBE-MARK-MASQ
		meta l4proto tcp   counter packets 0 bytes 0 dnat to 192.168.0.3:9153
	 
	chain KUBE-SVC-666FUMINWJLRRQPD  
		 counter packets 1 bytes 60 jump KUBE-SEP-LYLDBZYLHY4MT3AQ
	 
	chain KUBE-SEP-LYLDBZYLHY4MT3AQ  
		ip saddr 192.168.0.4  counter packets 0 bytes 0 jump KUBE-MARK-MASQ
		meta l4proto tcp   counter packets 1 bytes 60 dnat to 192.168.0.4:8080
	 
	chain KUBE-XLB-EDNDUDH2C75GIR6O  
		 fib saddr type local counter packets 0 bytes 0 jump KUBE-MARK-MASQ
		 fib saddr type local counter packets 0 bytes 0 jump KUBE-SVC-EDNDUDH2C75GIR6O
		 counter packets 0 bytes 0 jump KUBE-SEP-BLQHCYCSXY3NRKLC
	 
	chain KUBE-XLB-CG5I4G2RS3ZVWGLK  
		 fib saddr type local counter packets 0 bytes 0 jump KUBE-MARK-MASQ
		 fib saddr type local counter packets 0 bytes 0 jump KUBE-SVC-CG5I4G2RS3ZVWGLK
		 counter packets 0 bytes 0 jump KUBE-SEP-5XVRKWM672JGTWXH
	 
	chain KUBE-SVC-EDNDUDH2C75GIR6O  
		 counter packets 0 bytes 0 jump KUBE-SEP-BLQHCYCSXY3NRKLC
	 
	chain KUBE-SEP-BLQHCYCSXY3NRKLC  
		ip saddr 192.168.0.5  counter packets 0 bytes 0 jump KUBE-MARK-MASQ
		meta l4proto tcp   counter packets 0 bytes 0 dnat to 192.168.0.5:443
	 
	chain KUBE-SVC-CG5I4G2RS3ZVWGLK  
		 counter packets 0 bytes 0 jump KUBE-SEP-5XVRKWM672JGTWXH
	 
	chain KUBE-SEP-5XVRKWM672JGTWXH  
		ip saddr 192.168.0.5  counter packets 0 bytes 0 jump KUBE-MARK-MASQ
		meta l4proto tcp   counter packets 0 bytes 0 dnat to 192.168.0.5:80
	 
	chain KUBE-SVC-EZYNCFY2F7N6OQA2  
		 counter packets 0 bytes 0 jump KUBE-SEP-JYW326XAJ4KK7QPG
	 
	chain KUBE-SEP-JYW326XAJ4KK7QPG  
		ip saddr 192.168.0.5  counter packets 0 bytes 0 jump KUBE-MARK-MASQ
		meta l4proto tcp   counter packets 0 bytes 0 dnat to 192.168.0.5:8443
	 
 
table ip filter  
	chain DOCKER  
	 
	chain DOCKER-ISOLATION-STAGE-1  
		iifname "docker0" oifname != "docker0" counter packets 0 bytes 0 jump DOCKER-ISOLATION-STAGE-2
		counter packets 0 bytes 0 return
	 
	chain DOCKER-ISOLATION-STAGE-2  
		oifname "docker0" counter packets 0 bytes 0 drop
		counter packets 0 bytes 0 return
	 
	chain FORWARD  
		type filter hook forward priority filter; policy drop;
		iifname "weave"  counter packets 213 bytes 54014 jump WEAVE-NPC-EGRESS
		oifname "weave"  counter packets 150 bytes 30038 jump WEAVE-NPC
		oifname "weave" ct state new counter packets 0 bytes 0 log group 86 
		oifname "weave" counter packets 0 bytes 0 drop
		iifname "weave" oifname != "weave" counter packets 33 bytes 2941 accept
		oifname "weave" ct state related,established counter packets 0 bytes 0 accept
		 counter packets 0 bytes 0 jump KUBE-FORWARD
		ct state new  counter packets 0 bytes 0 jump KUBE-SERVICES
		ct state new  counter packets 0 bytes 0 jump KUBE-EXTERNAL-SERVICES
		counter packets 0 bytes 0 jump DOCKER-USER
		counter packets 0 bytes 0 jump DOCKER-ISOLATION-STAGE-1
		oifname "docker0" ct state related,established counter packets 0 bytes 0 accept
		oifname "docker0" counter packets 0 bytes 0 jump DOCKER
		iifname "docker0" oifname != "docker0" counter packets 0 bytes 0 accept
		iifname "docker0" oifname "docker0" counter packets 0 bytes 0 accept
	 
	chain DOCKER-USER  
		counter packets 0 bytes 0 return
	 
	chain KUBE-FIREWALL  
		 mark and 0x8000 == 0x8000 counter packets 0 bytes 0 drop
		ip saddr != 127.0.0.0/8 ip daddr 127.0.0.0/8  ct status dnat counter packets 0 bytes 0 drop
	 
	chain OUTPUT  
		type filter hook output priority filter; policy accept;
		ct state new  counter packets 527014 bytes 31628984 jump KUBE-SERVICES
		counter packets 36324809 bytes 6021214027 jump KUBE-FIREWALL
		meta l4proto != esp  mark and 0x20000 == 0x20000 counter packets 0 bytes 0 drop
	 
	chain INPUT  
		type filter hook input priority filter; policy accept;
		 counter packets 35869492 bytes 5971008896 jump KUBE-NODEPORTS
		ct state new  counter packets 390938 bytes 23457377 jump KUBE-EXTERNAL-SERVICES
		counter packets 36249774 bytes 6030068622 jump KUBE-FIREWALL
		meta l4proto tcp ip daddr 127.0.0.1 tcp dport 6784 fib saddr type != local ct state != related,established  counter packets 0 bytes 0 drop
		iifname "weave" counter packets 907273 bytes 88697229 jump WEAVE-NPC-EGRESS
		counter packets 34809601 bytes 5818213726 jump WEAVE-IPSEC-IN
	 
	chain KUBE-KUBELET-CANARY  
	 
	chain KUBE-PROXY-CANARY  
	 
	chain KUBE-EXTERNAL-SERVICES  
	 
	chain KUBE-NODEPORTS  
		meta l4proto tcp  tcp dport 32196 counter packets 0 bytes 0 accept
		meta l4proto tcp  tcp dport 32196 counter packets 0 bytes 0 accept
	 
	chain KUBE-SERVICES  
	 
	chain KUBE-FORWARD  
		ct state invalid counter packets 0 bytes 0 drop
		 mark and 0x4000 == 0x4000 counter packets 0 bytes 0 accept
		 ct state related,established counter packets 0 bytes 0 accept
		 ct state related,established counter packets 0 bytes 0 accept
	 
	chain WEAVE-NPC-INGRESS  
	 
	chain WEAVE-NPC-DEFAULT  
		# match-set weave-;rGqyMIl1HN^cfDki~Z$3]6!N dst  counter packets 14 bytes 840 accept
		# match-set weave-P.B !ZhkAr5q=XZ?3 tMBA+0 dst  counter packets 0 bytes 0 accept
		# match-set weave-Rzff h:=]JaaJl/G;(XJpGjZ[ dst  counter packets 0 bytes 0 accept
		# match-set weave-]B*(W?)t*z5O17G044[gUo#$l dst  counter packets 0 bytes 0 accept
		# match-set weave-iLgO^ o=U/*%KE[@=W:l~ 9T dst  counter packets 9 bytes 540 accept
	 
	chain WEAVE-NPC  
		ct state related,established counter packets 124 bytes 28478 accept
		ip daddr 224.0.0.0/4 counter packets 0 bytes 0 accept
		# PHYSDEV match --physdev-out vethwe-bridge --physdev-is-bridged counter packets 3 bytes 180 accept
		ct state new counter packets 23 bytes 1380 jump WEAVE-NPC-DEFAULT
		ct state new counter packets 0 bytes 0 jump WEAVE-NPC-INGRESS
	 
	chain WEAVE-NPC-EGRESS-ACCEPT  
		counter packets 48 bytes 3769 meta mark set mark or 0x40000 
	 
	chain WEAVE-NPC-EGRESS-CUSTOM  
	 
	chain WEAVE-NPC-EGRESS-DEFAULT  
		# match-set weave-s_+ChJId4Uy_$ G;WdH ~TK)I src  counter packets 0 bytes 0 jump WEAVE-NPC-EGRESS-ACCEPT
		# match-set weave-s_+ChJId4Uy_$ G;WdH ~TK)I src  counter packets 0 bytes 0 return
		# match-set weave-E1ney4o[ojNrLk.6rOHi;7MPE src  counter packets 31 bytes 2749 jump WEAVE-NPC-EGRESS-ACCEPT
		# match-set weave-E1ney4o[ojNrLk.6rOHi;7MPE src  counter packets 31 bytes 2749 return
		# match-set weave-41s)5vQ^o/xWGz6a20N:~?# E src  counter packets 0 bytes 0 jump WEAVE-NPC-EGRESS-ACCEPT
		# match-set weave-41s)5vQ^o/xWGz6a20N:~?# E src  counter packets 0 bytes 0 return
		# match-set weave-sui%__gZ kX~oZgI_Ttqp=Dp src  counter packets 0 bytes 0 jump WEAVE-NPC-EGRESS-ACCEPT
		# match-set weave-sui%__gZ kX~oZgI_Ttqp=Dp src  counter packets 0 bytes 0 return
		# match-set weave-nmMUaDKV*YkQcP5s?Q[R54Ep3 src  counter packets 17 bytes 1020 jump WEAVE-NPC-EGRESS-ACCEPT
		# match-set weave-nmMUaDKV*YkQcP5s?Q[R54Ep3 src  counter packets 17 bytes 1020 return
	 
	chain WEAVE-NPC-EGRESS  
		ct state related,established counter packets 907425 bytes 88746642 accept
		# PHYSDEV match --physdev-in vethwe-bridge --physdev-is-bridged counter packets 0 bytes 0 return
		fib daddr type local counter packets 11 bytes 640 return
		ip daddr 224.0.0.0/4 counter packets 0 bytes 0 return
		ct state new counter packets 50 bytes 3961 jump WEAVE-NPC-EGRESS-DEFAULT
		ct state new mark and 0x40000 != 0x40000 counter packets 2 bytes 192 jump WEAVE-NPC-EGRESS-CUSTOM
	 
	chain WEAVE-IPSEC-IN  
	 
	chain WEAVE-CANARY  
	 
 
table ip mangle  
	chain KUBE-KUBELET-CANARY  
	 
	chain PREROUTING  
		type filter hook prerouting priority mangle; policy accept;
	 
	chain INPUT  
		type filter hook input priority mangle; policy accept;
		counter packets 35716863 bytes 5906910315 jump WEAVE-IPSEC-IN
	 
	chain FORWARD  
		type filter hook forward priority mangle; policy accept;
	 
	chain OUTPUT  
		type route hook output priority mangle; policy accept;
		counter packets 35804064 bytes 5938944956 jump WEAVE-IPSEC-OUT
	 
	chain POSTROUTING  
		type filter hook postrouting priority mangle; policy accept;
	 
	chain KUBE-PROXY-CANARY  
	 
	chain WEAVE-IPSEC-IN  
	 
	chain WEAVE-IPSEC-IN-MARK  
		counter packets 0 bytes 0 meta mark set mark or 0x20000
	 
	chain WEAVE-IPSEC-OUT  
	 
	chain WEAVE-IPSEC-OUT-MARK  
		counter packets 0 bytes 0 meta mark set mark or 0x20000
	 
	chain WEAVE-CANARY  
	 
 
Wow. That s a lot of nftables entries, but it explains what s going on. We have a nat entry for:
meta l4proto tcp ip daddr 10.107.66.138 tcp dport 8080 counter packets 1 bytes 60 jump KUBE-SVC-666FUMINWJLRRQPD
which ends up going to KUBE-SEP-LYLDBZYLHY4MT3AQ and:
meta l4proto tcp counter packets 1 bytes 60 dnat to 192.168.0.4:8080
So packets headed for our echoserver are eventually ending up in a container that has a local IP address of 192.168.0.4. Which we can see in our routing table via the weave interface. Mystery explained. We can see the ingress for the externally visible HTTP service as well:
meta l4proto tcp ip daddr 192.168.33.147 tcp dport 80 counter packets 0 bytes 0 jump KUBE-XLB-CG5I4G2RS3ZVWGLK
which ends up redirected to:
meta l4proto tcp counter packets 0 bytes 0 dnat to 192.168.0.5:80
So from that we d expect the IP inside the echoserver pod to be 192.168.0.4 and the IP address instead our nginx ingress pod to be 192.168.0.5. Let s look:
root@udon:/# docker ps   grep echoserver
7cbb177bee18   k8s.gcr.io/echoserver                 "/usr/local/bin/run. "   3 days ago   Up 3 days             k8s_echoserver_hello-node-59bffcc9fd-8hkgb_default_c7111c9e-7131-40e0-876d-be89d5ca1812_0
root@udon:/# docker exec -it 7cbb177bee18 /bin/bash
root@hello-node-59bffcc9fd-8hkgb:/# awk '/32 host/   print f    f=$2 ' <<< "$(</proc/net/fib_trie)"   sort -u
127.0.0.1
192.168.0.4
It s a slightly awkward method of determining the local IPs addresses due to the stripped down nature of the container, but it clearly shows the expected 192.168.0.4 address. I ve touched here upon the ability to actually enter a container and have a poke around its running environment by using docker directly. Next step is to use that to investigate what containers have actually been spun up and what they re doing. I ll also revisit networking when I get to the point of building a multi-node cluster, to examine how the bridging between different hosts is done.

25 May 2021

Vincent Bernat: Jerikan+Ansible: a configuration management system for network

There are many resources for network automation with Ansible. Most of them only expose the first steps or limit themselves to a narrow scope. They give no clue on how to expand from that. Real network environments may be large, versatile, heterogeneous, and filled with exceptions. The lack of real-world examples for Ansible deployments, unlike Puppet and SaltStack, leads many teams to build brittle and incomplete automation solutions. We have released under an open-source license our attempt to tackle this problem: Here is a quick demo to configure a new peering:
This work is the collective effort of C dric Hasco t, Jean-Christophe Legatte, Lo c Pailhas, S bastien Hurtel, Tchadel Icard, and Vincent Bernat. We are the network team of Blade, a French company operating Shadow, a cloud-computing product. In May 2021, our company was bought by Octave Klaba and the infrastructure is being transferred to OVHcloud, saving Shadow as a product, but making our team redundant. Our network was around 800 devices, spanning over 10 datacenters with more than 2.5 Tbps of available egress bandwidth. The released material is therefore a substantial example of managing a medium-scale network using Ansible. We have left out the handling of our legacy datacenters to make the final result more readable while keeping enough material to not turn it into a trivial example.

Jerikan The first component is Jerikan. As input, it takes a list of devices, configuration data, templates, and validation scripts. It generates a set of configuration files for each device. Ansible could cover this task, but it has the following limitations:
  • it is slow;
  • errors are difficult to debug;1 and
  • the hierarchy to look up a variable is rigid.
Jerikan inputs and outputs
Jerikan inputs and outputs
If you want to follow the examples, you only need to have Docker and Docker Compose installed. Clone the repository and you are ready!

Source of truth We use YAML files, versioned with Git, as the single source of truth instead of using a database, like NetBox, or a mix of a database and text files. This provides many advantages:
  • anyone can use their preferred text editor;
  • the team prepares changes in branches;
  • the team reviews changes using merge requests;
  • the merge requests expose the changes to the generated configuration files;
  • rollback to a previous state is easy; and
  • it is fast.
The first file is devices.yaml. It contains the device list. The second file is classifier.yaml. It defines a scope for each device. A scope is a set of keys and values. It is used in templates and to look up data associated with a device.
$ ./run-jerikan scope to1-p1.sk1.blade-group.net
continent: apac
environment: prod
groups:
- tor
- tor-bgp
- tor-bgp-compute
host: to1-p1.sk1
location: sk1
member: '1'
model: dell-s4048
os: cumulus
pod: '1'
shorthost: to1-p1
The device name is matched against a list of regular expressions and the scope is extended by the result of each match. For to1-p1.sk1.blade-group.net, the following subset of classifier.yaml defines its scope:
matchers:
  - '^(([^.]*)\..*)\.blade-group\.net':
      environment: prod
      host: '\1'
      shorthost: '\2'
  - '\.(sk1)\.':
      location: '\1'
      continent: apac
  - '^to([12])-[as]?p(\d+)\.':
      member: '\1'
      pod: '\2'
  - '^to[12]-p\d+\.':
      groups:
        - tor
        - tor-bgp
        - tor-bgp-compute
  - '^to[12]-(p ap)\d+\.sk1\.':
      os: cumulus
      model: dell-s4048
The third file is searchpaths.py. It describes which directories to search for a variable. A Python function provides a list of paths to look up in data/ for a given scope. Here is a simplified version:2
def searchpaths(scope):
    paths = [
        "host/ scope[location] / scope[shorthost] ",
        "location/ scope[location] ",
        "os/ scope[os] - scope[model] ",
        "os/ scope[os] ",
        'common'
    ]
    for idx in range(len(paths)):
        try:
            paths[idx] = paths[idx].format(scope=scope)
        except KeyError:
            paths[idx] = None
    return [path for path in paths if path]
With this definition, the data for to1-p1.sk1.blade-group.net is looked up in the following paths:
$ ./run-jerikan scope to1-p1.sk1.blade-group.net
[ ]
Search paths:
  host/sk1/to1-p1
  location/sk1
  os/cumulus-dell-s4048
  os/cumulus
  common
Variables are scoped using a namespace that should be specified when doing a lookup. We use the following ones:
  • system for accounts, DNS, syslog servers,
  • topology for ports, interfaces, IP addresses, subnets,
  • bgp for BGP configuration
  • build for templates and validation scripts
  • apps for application variables
When looking up for a variable in a given namespace, Jerikan looks for a YAML file named after the namespace in each directory in the search paths. For example, if we look up a variable for to1-p1.sk1.blade-group.net in the bgp namespace, the following YAML files are processed: host/sk1/to1-p1/bgp.yaml, location/sk1/bgp.yaml, os/cumulus-dell-s4048/bgp.yaml, os/cumulus/bgp.yaml, and common/bgp.yaml. The search stops at the first match. The schema.yaml file allows us to override this behavior by asking to merge dictionaries and arrays across all matching files. Here is an excerpt of this file for the topology namespace:
system:
  users:
    merge: hash
  sampling:
    merge: hash
  ansible-vars:
    merge: hash
  netbox:
    merge: hash
The last feature of the source of truth is the ability to use Jinja2 templates for keys and values by prefixing them with ~ :
# In data/os/junos/system.yaml
netbox:
  manufacturer: Juniper
  model: "~  model upper  "
# In data/groups/tor-bgp-compute/system.yaml
netbox:
  role: net_tor_gpu_switch
Looking up for netbox in the system namespace for to1-p2.ussfo03.blade-group.net yields the following result:
$ ./run-jerikan scope to1-p2.ussfo03.blade-group.net
continent: us
environment: prod
groups:
- tor
- tor-bgp
- tor-bgp-compute
host: to1-p2.ussfo03
location: ussfo03
member: '1'
model: qfx5110-48s
os: junos
pod: '2'
shorthost: to1-p2
[ ]
Search paths:
[ ]
  groups/tor-bgp-compute
[ ]
  os/junos
  common
$ ./run-jerikan lookup to1-p2.ussfo03.blade-group.net system netbox
manufacturer: Juniper
model: QFX5110-48S
role: net_tor_gpu_switch
This also works for structured data:
# In groups/adm-gateway/topology.yaml
interface-rescue:
  address: "~  lookup('topology', 'addresses').rescue  "
  up:
    - "~ip route add default via   lookup('topology', 'addresses').rescue ipaddr('first_usable')   table rescue"
    - "~ip rule add from   lookup('topology', 'addresses').rescue ipaddr('address')   table rescue priority 10"
# In groups/adm-gateway-sk1/topology.yaml
interfaces:
  ens1f0: "~  lookup('topology', 'interface-rescue')  "
This yields the following result:
$ ./run-jerikan lookup gateway1.sk1.blade-group.net topology interfaces
[ ]
ens1f0:
  address: 121.78.242.10/29
  up:
  - ip route add default via 121.78.242.9 table rescue
  - ip rule add from 121.78.242.10 table rescue priority 10
When putting data in the source of truth, we use the following rules:
  1. Don t repeat yourself.
  2. Put the data in the most specific place without breaking the first rule.
  3. Use templates with parsimony, mostly to help with the previous rules.
  4. Restrict the data model to what is needed for your use case.
The first rule is important. For example, when specifying IP addresses for a point-to-point link, only specify one side and deduce the other value in the templates. The last rule means you do not need to mimic a BGP YANG model to specify BGP peers and policies:
peers:
  transit:
    cogent:
      asn: 174
      remote:
        - 38.140.30.233
        - 2001:550:2:B::1F9:1
      specific-import:
        - name: ATT-US
          as-path: ".*7018$"
          lp-delta: 50
  ix-sfmix:
    rs-sfmix:
      monitored: true
      asn: 63055
      remote:
        - 206.197.187.253
        - 206.197.187.254
        - 2001:504:30::ba06:3055:1
        - 2001:504:30::ba06:3055:2
    blizzard:
      asn: 57976
      remote:
        - 206.197.187.42
        - 2001:504:30::ba05:7976:1
      irr: AS-BLIZZARD

Templates The list of templates to compile for each device is stored in the source of truth, under the build namespace:
$ ./run-jerikan lookup edge1.ussfo03.blade-group.net build templates
data.yaml: data.j2
config.txt: junos/main.j2
config-base.txt: junos/base.j2
config-irr.txt: junos/irr.j2
$ ./run-jerikan lookup to1-p1.ussfo03.blade-group.net build templates
data.yaml: data.j2
config.txt: cumulus/main.j2
frr.conf: cumulus/frr.j2
interfaces.conf: cumulus/interfaces.j2
ports.conf: cumulus/ports.j2
dhcpd.conf: cumulus/dhcp.j2
default-isc-dhcp: cumulus/default-isc-dhcp.j2
authorized_keys: cumulus/authorized-keys.j2
motd: linux/motd.j2
acl.rules: cumulus/acl.j2
rsyslog.conf: cumulus/rsyslog.conf.j2
Templates are using Jinja2. This is the same engine used in Ansible. Jerikan ships some custom filters but also reuse some of the useful filters from Ansible, notably ipaddr. Here is an excerpt of templates/junos/base.j2 to configure DNS and NTP servers on Juniper devices:
system  
  ntp  
 % for ntp in lookup("system", "ntp") % 
    server   ntp  ;
 % endfor % 
   
  name-server  
 % for dns in lookup("system", "dns") % 
      dns  ;
 % endfor % 
   
 
The equivalent template for Cisco IOS-XR is:
 % for dns in lookup('system', 'dns') % 
domain vrf VRF-MANAGEMENT name-server   dns  
 % endfor % 
!
 % for syslog in lookup('system', 'syslog') % 
logging   syslog   vrf VRF-MANAGEMENT
 % endfor % 
!
There are three helper functions provided:
  • devices() returns the list of devices matching a set of conditions on the scope. For example, devices("location==ussfo03", "groups==tor-bgp") returns the list of devices in San Francisco in the tor-bgp group. You can also omit the operator if you want the specified value to be equal to the one in the local scope. For example, devices("location") returns devices in the current location.
  • lookup() does a key lookup. It takes the namespace, the key, and optionally, a device name. If not provided, the current device is assumed.
  • scope() returns the scope of the provided device.
Here is how you would define iBGP sessions between edge devices in the same location:
 % for neighbor in devices("location", "groups==edge") if neighbor != device % 
   % for address in lookup("topology", "addresses", neighbor).loopback tolist % 
protocols bgp group IPV  address ipv  -EDGES-IBGP  
  neighbor   address    
    description "IPv  address ipv  : iBGP to   neighbor  ";
   
 
   % endfor % 
 % endfor % 
We also have a global key-value store to save information to be reused in another template or device. This is quite useful to automatically build DNS records. First, capture the IP address inserted into a template with store() as a filter:
interface Loopback0
 description 'Loopback:'
  % for address in lookup('topology', 'addresses').loopback tolist % 
 ipv  address ipv   address   address store('addresses', 'Loopback0') ipaddr('cidr')  
  % endfor % 
!
Then, reuse it later to build DNS records by iterating over store():4
 % for device, ip, interface in store('addresses') % 
   % set interface = interface replace('/', '-') replace('.', '-') replace(':', '-') % 
   % set name = ' . '.format(interface lower, device) % 
  name  . IN   'A' if ip ipv4 else 'AAAA'     ip ipaddr('address')  
 % endfor % 
Templates are compiled locally with ./run-jerikan build. The --limit argument restricts the devices to generate configuration files for. Build is not done in parallel because a template may depend on the data collected by another template. Currently, it takes 1 minute to compile around 3000 files spanning over 800 devices.
Jerikan outputs when building templates
Output of Jerikan after building configuration files for six devices
When an error occurs, a detailed traceback is displayed, including the template name, the line number and the value of all visible variables. This is a major time-saver compared to Ansible!
templates/opengear/config.j2:15: in top-level template code
    config.interfaces.  interface  .netmask   adddress   ipaddr("netmask")  
        continent  = 'us'
        device     = 'con1-ag2.ussfo03.blade-group.net'
        environment = 'prod'
        host       = 'con1-ag2.ussfo03'
        infos      =  'address': '172.30.24.19/21' 
        interface  = 'wan'
        location   = 'ussfo03'
        loop       = <LoopContext 1/2>
        member     = '2'
        model      = 'cm7132-2-dac'
        os         = 'opengear'
        shorthost  = 'con1-ag2'
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
value = JerkianUndefined, query = 'netmask', version = False, alias = 'ipaddr'
[ ]
        # Check if value is a list and parse each element
        if isinstance(value, (list, tuple, types.GeneratorType)):
            _ret = [ipaddr(element, str(query), version) for element in value]
            return [item for item in _ret if item]
>       elif not value or value is True:
E       jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'adddress'
We don t have general-purpose rules when writing templates. Like for the source of truth, there is no need to create generic templates able to produce any BGP configuration. There is a balance to be found between readability and avoiding duplication. Templates can become scary and complex: sometimes, it s better to write a filter or a function in jerikan/jinja.py. Mastering Jinja2 is a good investment. Take time to browse through our templates as some of them show interesting features.

Checks Optionally, each configuration file can be validated by a script in the checks/ directory. Jerikan looks up the key checks in the build namespace to know which checks to run:
$ ./run-jerikan lookup edge1.ussfo03.blade-group.net build checks
- description: Juniper configuration file syntax check
  script: checks/junoser
  cache:
    input: config.txt
    output: config-set.txt
- description: check YAML data
  script: checks/data.yaml
  cache: data.yaml
In the above example, checks/junoser is executed if there is a change to the generated config.txt file. It also outputs a transformed version of the configuration file which is easier to understand when using diff. Junoser checks a Junos configuration file using Juniper s XML schema definition for Netconf.5 On error, Jerikan displays:
jerikan/build.py:127: RuntimeError
-------------- Captured syntax check with Junoser call --------------
P: checks/junoser edge2.ussfo03.blade-group.net
C: /app/jerikan
O:
E: Invalid syntax:  set system syslog archive size 10m files 10 word-readable
S: 1

Integration into GitLab CI The next step is to compile the templates using a CI. As we are using GitLab, Jerikan ships with a .gitlab-ci.yml file. When we need to make a change, we create a dedicated branch and a merge request. GitLab compiles the templates using the same environment we use on our laptops and store them as an artifact.
Merge requests in GitLab for a change
Merge request to add a new port in USSFO03. The templates were compiled successfully but approval from another team member is still required to merge.
Before approving the merge request, another team member looks at the changes in data and templates but also the differences for the generated configuration files:
Differences for generated configuration files
The change configures a port on the Juniper device, adds records to DNS, and updates NetBox with the new IP addresses (not shown).

Ansible After Jerikan has built the configuration files, Ansible takes over. It is also packaged as a Docker image to avoid the trouble to maintain the right Python virtual environment and ensure everyone is using the same versions.

Inventory Jerikan has generated an inventory file. It contains all the managed devices, the variables defined for each of them and the groups converted to Ansible groups:
ob1-n1.sk1.blade-group.net ansible_host=172.29.15.12 ansible_user=blade ansible_connection=network_cli ansible_network_os=ios
ob2-n1.sk1.blade-group.net ansible_host=172.29.15.13 ansible_user=blade ansible_connection=network_cli ansible_network_os=ios
ob1-n1.ussfo03.blade-group.net ansible_host=172.29.15.12 ansible_user=blade ansible_connection=network_cli ansible_network_os=ios
none ansible_connection=local
[oob]
ob1-n1.sk1.blade-group.net
ob2-n1.sk1.blade-group.net
ob1-n1.ussfo03.blade-group.net
[os-ios]
ob1-n1.sk1.blade-group.net
ob2-n1.sk1.blade-group.net
ob1-n1.ussfo03.blade-group.net
[model-c2960s]
ob1-n1.sk1.blade-group.net
ob2-n1.sk1.blade-group.net
ob1-n1.ussfo03.blade-group.net
[location-sk1]
ob1-n1.sk1.blade-group.net
ob2-n1.sk1.blade-group.net
[location-ussfo03]
ob1-n1.ussfo03.blade-group.net
[in-sync]
ob1-n1.sk1.blade-group.net
ob2-n1.sk1.blade-group.net
ob1-n1.ussfo03.blade-group.net
none
in-sync is a special group for devices which configuration should match the golden configuration. Daily and unattended, Ansible should be able to push configurations to this group. The mid-term goal is to cover all devices. none is a special device for tasks not related to a specific host. This includes synchronizing NetBox, IRR objects, and the DNS, updating the RPKI, and building the geofeed files.

Playbook We use a single playbook for all devices. It is described in the ansible/playbooks/site.yaml file. Here is a shortened version:
- hosts: adm-gateway:!done
  strategy: mitogen_linear
  roles:
    - blade.linux
    - blade.adm-gateway
    - done
- hosts: os-linux:!done
  strategy: mitogen_linear
  roles:
    - blade.linux
    - done
- hosts: os-junos:!done
  gather_facts: false
  roles:
    - blade.junos
    - done
- hosts: os-opengear:!done
  gather_facts: false
  roles:
    - blade.opengear
    - done
- hosts: none:!done
  gather_facts: false
  roles:
    - blade.none
    - done
A host executes only one of the play. For example, a Junos device executes the blade.junos role. Once a play has been executed, the device is added to the done group and the other plays are skipped. The playbook can be executed with the configuration files generated by the GitLab CI using the ./run-ansible-gitlab command. This is a wrapper around Docker and the ansible-playbook command and it accepts the same arguments. To deploy the configuration on the edge devices for the SK1 datacenter in check mode, we use:
$ ./run-ansible-gitlab playbooks/site.yaml --limit='edge:&location-sk1' --diff --check
[ ]
PLAY RECAP *************************************************************
edge1.sk1.blade-group.net  : ok=6    changed=0    unreachable=0    failed=0    skipped=3    rescued=0    ignored=0
edge2.sk1.blade-group.net  : ok=5    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0
We have some rules when writing roles:
  • --check must detect if a change is needed;
  • --diff must provide a visualization of the planned changes;
  • --check and --diff must not display anything if there is nothing to change;
  • writing a custom module tailored to our needs is a valid solution;
  • the whole device configuration is managed;6
  • secrets must be stored in Vault;
  • templates should be avoided as we have Jerikan for that; and
  • avoid duplication and reuse tasks.7
We avoid using collections from Ansible Galaxy, the exception being collections to connect and interact with vendor devices, like cisco.iosxr collection. The quality of Ansible Galaxy collections is quite random and it is an additional maintenance burden. It seems better to write roles tailored to our needs. The collections we use are in ci/ansible/ansible-galaxy.yaml. We use Mitogen to get a 10 speedup on Ansible executions on Linux hosts. We also have a few playbooks for operational purpose: upgrading the OS version, isolate an edge router, etc. We were also planning on how to add operational checks in roles: are all the BGP sessions up? They could have been used to validate a deployment and rollback if there is an issue. Currently, our playbooks are run from our laptops. To keep tabs, we are using ARA. A weekly dry-run on devices in the in-sync group also provides a dashboard on which devices we need to run Ansible on.

Configuration data and templates Jerikan ships with pre-populated data and templates matching the configuration of our USSFO03 and SK1 datacenters. They do not exist anymore but, we promise, all this was used in production back in the days!
Network architecture for Blade datacenter
The latest iteration of our network infrastructure for SK1, USSFO03, and future data centers. The production network is using BGPttH using a spine-leaf fabric. The out-of-band network is using a simple L2 design, using the spanning tree protocol, as well as a set of console servers.
Notably, you can find the configuration for:
our edge routers
Some are running on Junos, like edge2.ussfo03, the others on IOS-XR, like edge1.sk1. The implemented functionalities are similar in both cases and we could swap one for the other. It includes the BGP configuration for transits, peerings, and IX as well as the associated BGP policies. PeeringDB is queried to get the maximum number of prefixes to accept for peerings. bgpq3 and a containerized IRRd help to filter received routes. A firewall is added to protect the routing engine. Both IPv4 and IPv6 are configured.
our BGP-based fabric
BGP is used inside the datacenter8 and is extended on bare-metal hosts. The configuration is automatically derived from the device location and the port number.9 Top-of-the-rack devices are using passive BGP sessions for ports towards servers. They are also serving a provisioning network to let them boot using DHCP and PXE. They also act as a DHCP server. The design is multivendor. Some devices are running Cumulus Linux, like to1-p1.ussfo03, while some others are running Junos, like to1-p2.ussfo03.
our out-of-band fabric
We are using Cisco Catalyst 2960 switches to build an L2 out-of-band network. To provide redundancy and saving a few bucks on wiring, we build small loops and run the spanning-tree protocol. See ob1-p1.ussfo03. It is redundantly connected to our gateway servers. We also use OpenGear devices for console access. See con1-n1.ussfo03
our administrative gateways
These Linux servers have multiple purposes: SSH jump boxes, rescue connection, direct access to the out-of-band network, zero-touch provisioning of network devices,10 Internet access for management flows, centralization of the console servers using Conserver, and API for autoconfiguration of BGP sessions for bare-metal servers. They are the first servers installed in a new datacenter and are used to provision everything else. Check both the generated files and the associated Ansible tasks.

  1. Ansible does not even provide a line number when there is an error in a template. You may need to find the problem by bisecting.
    $ ansible --version
    ansible 2.10.8
    [ ]
    $ cat test.j2
    Hello   name  !
    $ ansible all -i localhost, \
    >  --connection=local \
    >  -m template \
    >  -a "src=test.j2 dest=test.txt"
    localhost   FAILED! =>  
        "changed": false,
        "msg": "AnsibleUndefinedVariable: 'name' is undefined"
     
    
  2. You may recognize the same concepts as in Hiera, the hierarchical key-value store from Puppet. At first, we were using Jerakia, a similar independent store exposing an HTTP REST interface. However, the lookup overhead is too large for our use. Jerikan implements the same functionality within a Python function.
  3. The list of available filters is mangled inside jerikan/jinja.py. This is a remain of the fact we do not maintain Jerikan as a standalone software.
  4. This is a bit confusing: we have a store() filter and a store() function. With Jinja2, filters and functions live in two different namespaces.
  5. We are using a fork with some modifications to be able to validate our configurations and exposing an HTTP service to reduce the time spent on each configuration check.
  6. There is a trend in network automation to automate a configuration subset, for example by having a playbook to create a new BGP session. We believe this is wrong: with time, your configuration will get out-of-sync with its expected state, notably hand-made changes will be left undetected.
  7. See ansible/roles/blade.linux/tasks/firewall.yaml and ansible/roles/blade.linux/tasks/interfaces.yaml. They are meant to be called when needed, using import_role.
  8. We also have some datacenters using BGP EVPN VXLAN at medium-scale using Juniper devices. As they are still in production today, we didn t include this feature but we may publish it in the future.
  9. In retrospect, this may not be a good idea unless you are pretty sure everything is uniform (number of switches for each layer, number of ports). This was not our case. We now think it is a better idea to assign a prefix to each device and write it in the source of truth.
  10. Non-linux based devices are upgraded and configured unattended. Cumulus Linux devices are automatically upgraded on install but the final configuration has to be pushed using Ansible: we didn t want to duplicate the configuration process using another tool.

Pavit Kaur: Journey to GSoC

I am really excited that my Google Summer of Code proposal with Debian for the project Debian Continuous Integration improvements has been accepted. Through this blog, I am here to share about my Pre-GSoC journey. alt text I knew about GSoC since my first year of college but had this misconception that only great coders get selected for GSoC which did not let me apply to the program until my 3rd year of engineering. I applied this year not because I thought I have turned into one but because I actually wanted to give a fair try to this before the time I become ineligible to participate.

Finding Project Scrolling through the list of GSoC 2021 organizations, I was checking out projects of organizations I am familiar with. Debian is one of the huge Open Source communities that has always inspired developers around the world to contribute to Open Source. So as I checked through Debian projects, I got excited to find the Debian Continuous Integration improvements project (referred to as debci in this post) related to web development and more concerned with backend work which is something I am very much interested in. I joined the community, and as directed by the application tasks of the project I set up the debci on my machine and started with an issue labeled as a newcomer. Soon I was able to submit my first Merge Request and it was reviewed by Antonio Terceiro, the mentor of my GSoC Project. With his further guidance, I was able to turn MR into an acceptable patch, and voila it got merged! That really did boost my morale to contribute further to the project.

Student Application Period At the suggestion of the mentor, during the Student Application Period, I worked on more open issues which were helping me understand the codebase better and in turn the deliverables of the project for my proposal. For my every doubt, I first tried to tackle it myself, and if still not able to find a solution I turn to mentors who did their best to answer my queries and this is how I completed my proposal and got it reviewed by Antonio before finally submitting it on 13th April. I still cannot express that feeling of satisfaction I achieved on submitting the proposal. I finally successfully applied for GSoC.

After Proposal Submission I did not stop my contribution after the Student Application period ended and kept on working on more issues which helped me stay in touch with the project and also because I was enjoying it a lot. I had already made up my mind to contribute more in Open Source as I learned and enjoyed plenty during this process.

Day of Results On 17th May, I got my acceptance mail at around 11:30 pm at night and I remember screaming and waking everybody up in my home to announce the news to them. It was truly the happiest moment for me.

Moment of Truth I would admit that I got involved with GSoC because of the reputation associated with it but things I learned during this pre GSoC period have made me realize the fun and learning opportunities associated with working opensource and I am here to stay for sure. I plan to write more blogs regarding my project and keep you guys updated about my work. Stay tuned!

Shirish Agarwal: Pandemic, Toolkit and India

Pandemic Situation in India. I don t know from where I should start. This is probably a good start. I actually would recommend Indiacable as they do attempt to share some things happening in India from day to day but still there is a lot thatt they just can t cover, nobody can cover. There were two reports which kind of shook me all inside. One which sadly came from the UK publication Independent, probably as no Indian publication would dare publish it. The other from Rural India. I have been privileged in many ways, including friends who have asked me if I need any financial help. But seeing reports like above, these people need more help, guidance and help than I. While I m never one to say give to Foundations. If some people do want to help people from Maharashtra, then moneylifefoundation could be a good place where they could donate. FWIW, they usually use the foundation to help savers and investors be safe and help in getting money when taken by companies with dubious intentions. That is their drive. Two articles show their bent. The first one is about the Algo scam which I have written previously about the same in this blog. Interestingly, when I talk about this scam, all Modi supporters are silent. The other one does give some idea as to why the Govt. is indifferent. That is going to a heavy cross for all relatives to bear. There has been a lot that has been happening. Now instead of being limited to cities, Covid has now gone hinterland in a big way. One could ask also Praveen as he probably knows what would be good for Kerala and surrounding areas. The biggest change, however, has been that India is now battling not just the pandemic but also Mucormycosis also known as black fungus and its deadlier cousin the white fungus. Mucormycosis came largely due to an ill-advise given that applying cow dung gives protection to Corona. And many applied it due to faith. And people who know science do know that in fact it has that bacteria. Sadly, those of us who are and were more interested in law, computer science etc. has now also have to keep on top of what is happening in the medical field. It isn t that I hate it, but it has a lot of costs. From what I could gather on various social media and elsewhere, a single injection of anti-fungal for the above costs INR 3k/- and that needs to be 5 times in a day and that course has to be for three weeks. So even the relatively wealthy people can and will become poor in no time. No wonder thousands of those went to UK, US, Dubai or wherever they could find safe-harbor from the pandemic with no plans of arriving back soon. There was also the whole bit about FBS or Fetal Bovin Serum. India ordered millions of blood serum products from abroad and continues to. This was quickly shut down as news on Social Media. Apparently, it is only the Indian cow which is worthy of reverence. All other cows and their children are fair game according to those in power. Of course, that discussion was quickly shut down as was the discussion about IGP (Indian Genome Project). People over the years had asked me why India never participated for the HGP (Human Gnome Project). I actually had no answer for that. Then in 2020, there was idea of IGP which was put up and then it was quickly shot down as the results could damage a political party s image. In fact, a note to people who want to join Indian civil services tells the reason exactly. While many countries in the world are hypocrites, including the U.S. none can take the place that India has made for itself in that field.

The Online experience The vaccination process has been made online and has led to severe heartburn and trouble for many including many memes. For e.g.

Daily work, get up, have a bath, see if you got a slot on the app, sleep.
People trying desperately to get a slot, taken from Hindi Movie Dilwale Dulhania Le Jaygenge.
Just to explain what is happening, one has to go to the website of cowin. Sharing a screenshot of the same.
Cowin app. sceeenshot
I have deliberately taken a screenshot of the cowin app. in U.P. which is one of the areas where the ruling party, BJP has. I haven t taken my state for the simple reason, even if a slot is open, it is of no use as there are no vaccines. As have been shared in India Cable as well as in many newspapers, it is the Central Govt. which holds the strings for the vaccines. Maharashtra did put up an international tender but to no effect. All vaccine manufacturers want only Central Govt. for purchases for multiple reasons. And GOI is saying it has no money even though recently it got loans as well as a dividend from RBI to the tune of 99k crore. For what all that money is, we have no clue. Coming back though, to the issue at hand. the cowin app. is made an open api. While normally, people like us should and are happy when an API is open, it has made those who understand how to use git, compile, etc. better than others. A copy of the public repo. of how you can do the same can be found on Github. Now, obviously, for people like me and many others it has ethical issues.

Kiran s Interview in Times of India (TOI) There isn t much to say apart from I haven t used it. I just didn t want to. It just is unethical. Hopefully, in the coming days GOI does something better. That is the only thing we are surviving on, hope.

The Toolkit saga A few days before, GOI shared a toolkit apparently made by Congress to defame the party in power. That toolkit was shared before the press and Altnews did the investigation and promptly shredded the claims. Congress promptly made an FIR in Chhattisgarh where it is in power. The gentleman who made the claims Mr. Sambit Patra refused to appear against the police without evidence citing personal reasons and asking 1 week to appear before them. Apart from Altnews which did a great job, sadly many people didn t even know that there is something called WYSIWYG. I had to explain that so many Industries, whether it is politics, creative industries, legal, ad industries, medical transcription, and imaging all use this, and all the participants use the same version of the software. The reason being that in most Industries, there is a huge loss and issue of legal liabilities if something untoward happens. For e.g. if medical transcription is done in India is wrong (although his or her work will be checked by a superior in the West), but for whatever reason is not, and a wrong diagnosis is put (due to wrong color or something) then a patient could die and the firm who does that work could face heavy penalties which could be the death of them. There is another myth that Congress has unlimited wealth or huge wealth. I asked if that was the case, why didn t they shift to Mac. Of course, none have answers on this one. There is another reason why they didn t want to appear. The Rona Wilson investigation by Arsenal Experts also has made them cautious. Previously, they had a free run. Nowadays, software forensic tools are available to one and all. For e.g. Debian itself has a good variety of tools for the same. I remember Vipin s sharing few years back. For those who want to start, just install the apps. and try figuring out. Expertise on using the tools takes years though, as you use the tool day in night. Update 25/05/2021 Apparently because Twitter made and showcased few tweets as Manipulated Media , those in Govt. are and were dead against it. So they conducted a raid against Twitter India headquarters, knowing fully well that there would be nobody except security. The moment I read this, my mind went to the whole Fruit of the poisonous tree legal doctrine. Sadly though, India doesn t recognize it and in fact, still believes in the pre-colonial era that evidence however collected is good. A good explanation of the same can be found here. There are some exceptions to the rule, but they are done so fine that more often than not, they can t be used in the court of law in India. Although a good RTI was shared by Mr. Saket Gokhale on the same issue, which does raise some interesting points
Twitter India Raid, Saket Gokhale RTI 1
Saket Gokhale RTI query , Twitter India Raid 2
FWIW, Saket has been successful in getting his prayers heard either as answers to RTI queries or then following it up in the various High Courts of India. Of course, those who are in the ruling party ridicule him but are unable to find faults in his application of logic. And quite a few times, I have learned from his applications as well as nuances or whatever is there in law, a judgment or a guideline which he invokes in his prayer. For e.g. the Lalitha Kumari Guidelines which the gentleman has shared in his prayer can be found here. Hence now, it would be upto the Delhi Police Cell to prove their case in response to RTI. He has also trapped them as he has shared they can t give excuses/exemptions which they have tried before. As I had shared earlier, High Courts in India have woken up, whether it is Delhi, Mumbai, Aurangabad, Madhya Pradesh, Uttar Pradesh, Odisha or Kerala. Just today i.e. on 25th May 2021, Justices Bela Trivedi and Justice Kalra had asked how come all the hospitals don t have NOC from the Fire De[partment. They also questioned the ASG (Assistant Solicitor General) as how BU (Building Use Certificate) has been granted as almost all the 400 hospitals are in residential area. To which the ASG replies, it is the same state in almost 4000 schools as well as 6000 odd factories in Ahemdabad alone, leave the rest of the district and state alone. And this is when last year strict instuctions were passed. They chose to do nothing sadly. I will share a link on this when bar and bench gives me  The Hindu also shared the whole raid on twitter saga.

Conclusion In conclusion, I sincerely do not where we are headed. The only thing I know is that we cannot expect things to be better before year-end and maybe even after that. It all depends on the vaccines and their availability. After that ruralindia article, I had to see quite a few movies and whatnot just to get that out of my head. And this is apart from the 1600 odd teachers and workers who have died in the U.P. poll duty. Now, what a loss, not just to the family members of the victims, but a whole generation of school children who would not be able to get quality teaching and be deprived of education. What will be their future, God only knows. The only good Bollywood movie which I saw was Ramprasad ki Teravi . The movie was an accurate representation of most families in and around me. There was a movie called Sansar (1987) which showed the breakup of the joint family and into a nuclear family. This movie could very well have been a continuation of the same. Even Marathi movies which at one time were very progressive have gone back to the same boy, girl love story routine. Sameer, though released in late 2020, was able to see it only recently. Vakeel Saab was an ok copy of Pink . I loved Sameer as, unlike Salman Khan films, it showed pretty much an authentic human struggle of a person who goes to the Middle East without any qualifications and works as a laborer and the trials he goes through. Somehow, Malayalam movies have a knack for showing truth without much of budget. Most of the Indian web series didn t make an impact. I think many of them were just going through the motions, it seems as everybody is concerned with the well-being of their near and dear ones. There was also this (Trigger Warning: This story discusses organized campaigns glorifying and advocating sexual violence against Muslim women.) Hoping people somehow make it to the other side of the pandemic.

24 May 2021

Antoine Beaupr : Leaving Freenode

The freenode IRC network has been hijacked. TL;DR: move to libera.chat or OFTC.net, as did countless free software projects including Gentoo, CentOS, KDE, Wikipedia, FOSDEM, and more. Debian and the Tor project were already on OFTC and are not affected by this.

What is freenode and why should I care? freenode is the largest remaining IRC network. Before this incident, it had close to 80,000 users, which is small in terms of modern internet history -- even small social networks are larger by multiple orders of magnitude -- but is large in IRC history. The IRC network is also extensively used by the free software community, being the default IRC network on many programs, and used by hundreds if not thousands of free software projects. I have been using freenode since at least 2006. This matters if you care about IRC, the internet, open protocols, decentralisation, and, to a certain extent, federation as well. It also touches on who has the right on network resources: the people who "own" it (through money) or the people who make it work (through their labor). I am biased towards open protocols, the internet, federation, and worker power, and this might taint this analysis.

What happened? It's a long story, but basically:
  1. back in 2017, the former head of staff sold the freenode.net domain (and its related company) to Andrew Lee, "American entrepreneur, software developer and writer", and, rather weirdly, supposedly "crown prince of Korea" although that part is kind of complex (see House of Yi, Yi Won, and Yi Seok). It should be noted the Korean Empire hasn't existed for over a century at this point (even though its flag, also weirdly, remains)
  2. back then, this was only known to the public as this strange PIA and freenode joining forces gimmick. it was suspicious at first, but since the network kept running, no one paid much attention to it. opers of the network were similarly reassured that Lee would have no say in the management of the network
  3. this all changed recently when Lee asserted ownership of the freenode.net domain and started meddling in the operations of the network, according to this summary. this part is disputed, but it is corroborated by almost a dozen former staff which collectively resigned from the network in protest, after legal threats, when it was obvious freenode was lost.
  4. the departing freenode staff founded a new network, irc.libera.chat, based on the new ircd they were working on with OFTC, solanum
  5. meanwhile, bot armies started attacking all IRC networks: both libera and freenode, but also OFTC and unrelated networks like a small one I help operate. those attacks have mostly stopped as of this writing (2021-05-24 17:30UTC)
  6. on freenode, however, things are going for the worse: Lee has been accused of taking over a channel, in a grotesque abuse of power; then changing freenode policy to not only justify the abuse, but also remove rules against hateful speech, effectively allowing nazis on the network (update: the change was reverted, but not by Lee)
Update: even though the policy change was reverted, the actual conversations allowed on freenode have already degenerated into toxic garbage. There are also massive channel takeovers (presumably over 700), mostly on channels that were redirecting to libera, but also channels that were still live. Channels that were taken over include #fosdem, #wikipedia, #haskell... Instead of working on the network, the new "so-called freenode" staff is spending effort writing bots and patches to basically automate taking over channels. I run an IRC network and this bot is obviously not standard "services" stuff... This is just grotesque. At this point I agree with this HN comment:
We should stop implicitly legitimizing Andrew Lee's power grab by referring to his dominion as "Freenode". Freenode is a quarter-century-old community that has changed its name to libera.chat; the thing being referred to here as "Freenode" is something else that has illegitimately acquired control of Freenode's old servers and user database, causing enormous inconvenience to the real Freenode.
I don't agree with the suggested name there, let's instead call it "so called freenode" as suggested later in the thread.

What now? I recommend people and organisations move away from freenode as soon as possible. This is a major change: documentation needs to be fixed, and the migration needs to be coordinated. But I do not believe we can trust the new freenode "owners" to operate the network reliably and in good faith. It's also important to use the current momentum to build a critical mass elsewhere so that people don't end up on freenode again by default and find an even more toxic community than your typical run-of-the-mill free software project (which is already not a high bar to meet). Update: people are moving to libera in droves. It's now reaching 18,000 users, which is bigger than OFTC and getting close to the largest, traditionnal, IRC networks (EFnet, Undernet, IRCnet are in the 10-20k users range). so-called freenode is still larger, currently clocking 68,000 users, but that's a huge drop from the previous count which was 78,000 before the exodus began. We're even starting to see the effects of the migration on netsplit.de. Update 2: the isfreenodedeadyet.com site is updated more frequently than netsplit and shows tons more information. It shows 25k online users for libera and 61k for so-called freenode (down from ~78k), and the trend doesn't seem to be stopping for so-called freenode. There's also a list of 400+ channels that have moved out. Keep in mind that such migrations take effect over long periods of time.

Where do I move to? The first thing you should do is to figure out which tool to use for interactive user support. There are multiple alternatives, of course -- this is the internet after all -- but here is a short list of suggestions, in preferred priority order:
  1. irc.libera.chat
  2. irc.OFTC.net
  3. Matrix.org, which bridges with OFTC and (hopefully soon) with libera as well, modern IRC alternative
  4. XMPP/Jabber also still exists, if you're into that kind of stuff, but I don't think the "chat room" story is great there, at least not as good as Matrix
Basically, the decision tree is this:
  • if you want to stay on IRC:
    • if you are already on many OFTC channels and few freenode channels: move to OFTC
    • if you are more inclined to support the previous freenode staff: move to libera
    • if you care about matrix users (in the short term): move to OFTC
  • if you are ready to leave IRC:
    • if you want the latest and greatest: move to Matrix
    • if you like XML and already use XMPP: move to XMPP
Frankly, at this point, everyone should seriously consider moving to Matrix. The user story is great, the web is a first class user, it supports E2EE (although XMPP as well), and has a lot of momentum behind it. It even bridges with IRC well (which is not the case for XMPP) so if you're worried about problems like this happening again. (Indeed, I wouldn't be surprised if similar drama happens on OFTC or libera in the future. The history of IRC is full of such epic controversies, takeovers, sabotage, attacks, technical flamewars, and other silly things. I am not sure, but I suspect a federated model like Matrix might be more resilient to conflicts like this one.) Changing protocols might mean losing a bunch of users however: not everyone is ready to move to Matrix, for example. Graybeards like me have been using irssi for years, if not decades, and would take quite a bit of convincing to move elsewhere. I have mostly kept my channels on IRC, and moved either to OFTC or libera. In retrospect, I think I might have moved everything to OFTC if I would have thought about it more, because almost all of my channels are there. But I kind of expect a lot of the freenode community to move to libera, so I am keeping a socket open there anyways.

How do I move? The first thing you should do is to update documentation, websites, and source code to stop pointing at freenode altogether. This is what I did for feed2exec, for example. You need to let people know in the current channel as well, and possibly shutdown the channel on freenode. Since my channels are either small or empty, I took the radical approach of:
  • redirecting the channel to ##unavailable which is historically the way we show channels have moved to another network
  • make the channel invite-only (which effectively enforces the redirection)
  • kicking everyone out of the channel
  • kickban people who rejoin
  • set the topic to announce the change
In IRC speak, the following commands should do all this:
/msg ChanServ set #anarcat mlock +if ##unavailable
/msg ChanServ clear #anarcat users moving to irc.libera.chat
/msg ChanServ set #anarcat restricted on
/topic #anarcat this channel has moved to irc.libera.chat
If the channel is not registered, the following might work
/mode #anarcat +if ##unavailable
Then you can leave freenode altogether:
/disconnect Freenode unacceptable hijack, policy changes and takeovers. so long and thanks for all the fish.
Keep in mind that some people have been unable to setup such redirections, because the new freenode staff have taken over their channel, in which case you're out of luck... Some people have expressed concern about their private data hosted at freenode as well. If you care about this, you can always talk to NickServ and DROP your nick. Be warned, however, that this assumes good faith of the network operators, which, at this point, is kind of futile. I would assume any data you have registered on there (typically: your NickServ password and email address) to be compromised and leaked. If your password is used elsewhere (tsk, tsk), change it everywhere. Update: there's also another procedure, similar to the above, but with a different approach. Keep in mind that so-called freenode staff are actively hijacking channels for the mere act of mentioning libera in the channel topic, so thread carefully there.

Last words This is a sad time for IRC in general, and freenode in particular. It's a real shame that the previous freenode staff have been kicked out, and it's especially horrible that if the new policies of the network are basically making the network open to nazis. I wish things would have gone out differently: now we have yet another fork in the IRC history. While it's not the first time freenode changes name (it was called OPN before), now the old freenode is still around and this will bring much confusion to the world, especially since the new freenode staff is still claiming to support FOSS. I understand there are many sides to this story, and some people were deeply hurt by all this. But for me, it's completely unacceptable to keep pushing your staff so hard that they basically all (except one?) resign in protest. For me, that's leadership failure at the utmost, and a complete disgrace. And of course, I can't in good conscience support or join a network that allows hate speech. Regardless of the fate of whatever we'll call what's left of freenode, maybe it's time for this old IRC thing to die already. It's still a sad day in internet history, but then again, maybe IRC will never die...

22 May 2021

Ritesh Raj Sarraf: Setting Up a Secure Webapp

As a person who prefers full access to data in the simplest format, while at the same time having it useful with latest technologies, my quest for trying things out is an ongoing activity. Earlier, I blogged about my needs of collating news feeds in a simple format, readily accessible offline, while still being useful and aligned with the modern paradigm. In today s age, the other common aspect of our life, is digitization of moments. With the advent of great technology and affordable economics, the world now has access to great devices to capture moments in digital form. Most people, these days, are equipped with smart devices, like mobile phones, that come with pretty good image capturing devices. Our lives, our societies, how we interact; a lot of it is now built around the assumption of smart devices and digital services. A lot of good things have happened of it. We are now able to send messages to people, securely, in a matter of seconds. We are now able to capture moments, which otherwise we d often miss; all thanks to devices like smart mobile phones that most of all carry almost all along with us. In this regard, we are quite indebted to the technology revolution that has improved life styles all over. Of course, like every coin has a flip side, thus too, too much of obsession with digital life, also has its drawbacks. A simple reminder to self should always be about what a human life is designed for and try to live by its rules.

Moments Given the tools general availability, we are all more used to capturing moments. Once upon a time, taking a photo meant going to a photo studio. Then came a generation where we d have personal devices with roll films, to which we captured the moments. And then get it processed from a photo developing service provider. And that is when we d know which all moments came out correct and which ones were spoilt. In such cases, there s always the wish I could have that moment again. Thankfully, now, in our current generation, we have the liberty to take pictures and validate them almost immediately. We also have the flexibility to take multiple shots of the moment and do the filtering later. There s also many intelligent and invasive services, all mostly provided free, to help you organize these moments; at a small cost of keeping an eye on your life activity. In a summary, mankind, now, generates lots and lots of data. So much lot that now even mammoths like Google are forced to make a call whether it is more profitable to sniff user activity while providing them a service for free, or ask people to pay otherwise too.

Privacy Many of us are wary of the amount of personal data we generate, which is our asset; and how the big tech giants want a piece of it in the name of free services. And in such quest, free aside, there s always a lookout for privacy savvy tools that can help us draw a bold line in between Public and Private

Google Photos Google Photos has been a great tool. The way it organizes, siphons and present your data is simply amazing. No good would have been the photos taken, if they weren t easily searchable, organized, annotated, and presented. But Google Photos is proprietary and invasive. The amount of information they have access to should be a concern to all people. For services that invade so much, the world needs Free Services.

PhotoPrism I only came across this project around a month ago, while the project has been around since 2019. So far, in my exploration, this is one of the best Free Software tools to manage and organize my digital photos. The other favorite tool that I regularly depend on is Digikam and it still is a gem. In a gist, PhotoPrism aims to be the equivalent of Google Photos, for the privacy savvy people. As of date, it has a decent list of features available. And for some of the missing ones, I ve come up with a fairly okay workflow with other tools, which is one of the reasons of this blog post. PhotoPrism is a web app, written in the Go Programming Language. Its layout and workflow as a Photo Management application is similar to Google Photos. It is very performant compared to other applications. Since it is a Progressive Web App, access through a Laptop, Tablet or Mobile is almost the same. It uses Google TensorFlow for some of its features and thus you feel like using Google Photos in some regard. Some shortcomings and workarounds
  • Facial Recognition: To date, facial recognition is a planned feature. But this was easy to tackle given that Digikam has pretty solid facial recognition. So I use Digikam to detect, recognize and annotate faces. And then PhotoPrism is happy enough to use those annotations and present relevant data.
  • HTTP Web App: The upstream project has done a good job of making use of Docker container technology in presenting this web app solution. A software solution, the equivalent of a Google Photos, does need a heavyweight config. In free software, it is all about reusing available tools. PhotoPrism makes use of tools like MySQL, Vue.js, TensorFlow, GoLang, GoLibs and strives to provide a single package solution, all thanks to Docker containers.
In its current offering, PhotoPrism is run as a HTTP Web App. I wanted to have an added layer of security on top of it. And thus run an nginx reverse proxy on top of it. Along with it, I run the proxy service on HTTPS, thus making all traffic from clients to the proxy encrypted. I also wanted an added layer of HTTP AUTH on top, so I explored some options and finally settled down with http-auth-digest Also, in the current implementation, PhotoPrism doesn t have a strong notion of normal and private photos in its data organization. I wanted normal photos available under a standard auth realm and private photos under a different realm. And along with a different realm, I also wanted some added security directives for it. So far, it looks like I ve put together a decent solution with the help of nginx.
  • First of all, since the port from the host is forwarded to the docker instance, that needed to be controlled. Instead of the default of listening on all interfaces, I changed it to loopback only. Because my primary and only interface is going to be the nginx reverse proxy
  • Setup nginx with a self-signed certificate to have all communication encrypted.
  • Setup nginx as a reverse proxy to talk to PhotoPrism.
    server  
       listen 80;
       listen [::]:80;
       server_name lenovo;
       return 302 https://$host$request_uri;
       rewrite ^ https://$http_host$request_uri? permanent;    # force redirect http to https
        server_tokens off;
     
    server  
    listen 443 ssl http2 default_server;
    listen [::]:443 ssl http2 default_server;
    ssl_certificate_key /etc/ssl/private/ssl-cert-snakeoil.key;
    ssl_certificate /etc/ssl/certs/ssl-cert-snakeoil.pem;
    server_tokens off;
    server_name lenovo;
    client_max_body_size 500M;
        auth_digest_expires 900s;
        auth_digest_evasion_time 5s;
        auth_digest_replays 500;
    location /private  
        auth_digest 'abc';
        auth_digest_user_file /etc/nginx/htdigest;
        auth_digest_expires 900s;
        auth_digest_evasion_time 5s;
        auth_digest_replays 500;
        proxy_pass http://localhost:2342/private;
     
    location /discover  
        auth_digest 'abc';
        auth_digest_user_file /etc/nginx/htdigest;
        auth_digest_expires 900s;
        auth_digest_evasion_time 5s;
        auth_digest_replays 500;
        proxy_pass http://localhost:2342/discover;
     
    location /  
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $host;
        proxy_buffering off;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_pass http://localhost:2342;
          
        auth_basic "Priv";
        auth_basic_user_file /etc/nginx/htpasswd;
     
     
    
  • nginx packaging: I was so happy to see the simplicity of the nginx packaging in Debian. Becuase http-auth-digest is not upstream, it needs to be pulled in separately and compiled. I was happy to see how simple the packaging structure of nginx in Debian is. It was just a matter of putting in the module in the right modules location, which as structured such in the packaging falls under the debian/ packaging sub-folder, that any future upgrades will be quite easy to manage.
  • http-auth-digest: I d love to see http-auth-digest module be part of the upstream package. While I m a web n00b, I felt this module perfect for my use case. From what I ve understood, set up and tested so far, this module fills in all my requirements; which is more of like a session management.
With the combined setup in place, https://host is authenticated with a different set of credentials. On the other hand, https://host/discover and /private get covered with a different set of credentials and policies. While, this will continue to be an ongoing effort and audit of the services that I build, so far now, I feel this is in decent shape that I can use it as my daily driver. The end result is:
nginx auth prompt
nginx auth prompt
PhotoPrism Photo List
PhotoPrism Photo List
PhotoPrism Menu
PhotoPrism Menu

Updates
  • Things can really get frustrating at times. These days, it is Google that contributes to it. https://bugs.chromium.org/p/chromium/issues/detail?id=544244
    • It is is now fair to say that Google is big enough to rewrite all the standards. Or at least break them. Or best, their arrogance of it. This bug report is a fine example of what other ways it could have been dealt.

Next.

Previous.