Farmer on-going protests Before I start with the education system in India which I have talked about many times in the past, first let me share couple of pieces about the farmer movement which is still at Sanghu Delhi border. The above picture became somewhat viral as it showed Manjeet Kaur, who drove down from Patiala, Punjab to Sanghu border along with her friends to take part in the on-going protests. The picture not only shares how the women are part and parcel of this protest but also they are independently taking part in the protest. The other were two articles I read today, first was an article in tribune which questions that if the policy worked, why it didn t work in the state of Bihar. The other by a young law student who had to go from Chandigarh to Delhi with family for some work and her experience with the ongoing protest. In fact, an interesting observation was made by the CJI in the many suits against farmer protests in the SC. This makes for much more interesting read when you see an RTI query filed by Saket Gokhale to NHAI , a Central Govt. agency which is supposed to be independent and asks if they had filed an FIR and asked compensation from Haryana State and Haryana State Police which had dug up National Highway 44 and if any permission was asked for the same from NHAI. And NHAI unable to take any action for the same. If this isn t shameful, I dunno what is

NHAI response to Saket Gokhale s query.

Sadly, the way the response has been worded makes it impossible for NHAI to discharge its own responsibilities and this becomes a precedent for other states now that know that NHAI is vulnerable. A pretty sad turn on events. Indian Education can t go online There was a recent article on scroll which shared how Indian education can t go online as only a few have computers with decent netlink speeds as well as other factors which are needed for online education. But there are also many things that the article doesn t take into account which actually make the task more difficult and raise the boundary more. Now in most schools and colleges, the number of students to teacher ratio could be anywhere between 70-150 or even more. In the last few years, a lot of schools have been closed down by various Governments, including and not limited to the ruling Govt. They have in fact intensified closures of public schools wherever their Govt. has been in power. Closing to 5000+ schools in one state in a year is a dramatic shift and such has been happening time and again. In fact, the rising costs of Indian education has made many to leave Indian shores and do studies abroad. And once they do their masters or whatever, the chances of them coming back to India become more and more remote. In India the costs have been becoming so bad that NBFC s have started products targeting the same. How NBFC and Banks have (both public and private) have fared with respect to Indian consumers needs its own blog post but one word to describe it is bad . But as shared above, needs its own blog post. Coming to the Indian context though, what has not been captured in that article is that the responsibility of making new content also raises huge barriers for teachers. My own experience in teacher s trainings for ICT usage has shown that most teachers do not know and use internet effectively both to sustain their own curiosity as well as their students. Part of which is whether you are private employee or a public school teacher, the teacher is not paid enough. I have had multiple conversations with friends over the years who are teachers who shared that they get 50% salary in-hand while they sign for 100%. This is more in the case or private schools though. In Govt. schools, the teachers apart from their regular administrative duties apart from teaching duties are also unpaid labor for Govt. policy. Take the recent covid crisis, it was the teachers who for months together went from door-to-door asking if they had a covid patient. This was all over India. Even for voter registration, census, polio and various other immunization efforts, the teachers are roped in. So apart from that, they somehow have to figure out how to make ends meet and also boost student morale. Hence the attention is only limited to the first couple of benches rather than the whole as a 45-minute to an hr. session is just not enough to go through a class of 70-150 school students giving individual attention. And this is when for most teachers, teaching is a means to an end and not the end itself. I am going to take one example of science and sort of break-it-down in multiple steps and how I would have approached that topic for say the 5th-6th standard students in say a public school in Pune and especially if Covid would not have been an issue so you have face-to-face meetup. There was recent news about a mysterious radio signal which came from one of our closest galactic neighbors Proxima Centauri. Now let s say there was a class I was teaching/sharing which I had shared before and there already is trust formed. So before coming to the news, I would tell the students about frequency and more generally the notation of why we like to measure things and how we measure things. There is so much beautiful history which could be acted and enacted which can show and remains in mind why measurement is needed. Once that is understood, discussed and an underpinning is established, we could move to human perception or the lack of it. We know that humans have lots of limits in almost everything, whether it is talking, touching, hearing, all of our five senses are pretty limited with what we know of spectrum available in the immediate family kingdom as well as in the Universe. I would start with how far can a person throw his voice and be heard without using any other means. There does come a point where they need to use anything from a megaphone to a loudspeaker and what it actually does. The other thing I would then talk is about the radio and ask the students to find more about the internals of a radio. If possible to bring an old radio to school where it could be disassembled. After they are familiar with some names of the electronic components and what they do, take them to the electronics market where they try to source all the things needed to make a radio and whatever they encounter. This would allow the students to try and do bargain shopping as well as learn from where to source things. Some might even get a copy or two of electronic projects where the shop themselves sell blueprints to hobbyists so that they can tinker. If there is a place in the school where soldering can be done, then all can try and sooner or later we come to know if something works or not. There is also possibility of talking about noise cancellation and then the topic of ITU can also be bought up and how they do frequency allocation. Last but not the least then the topic can be approached about an alien civilization and an unknown radio signal and what it means and what it can mean. Now if just one topic can give such a wide range of things to do and develop an understanding about not the subject itself but surrounding subjects as well I see no reason why teachers can t do this except they are handcuffed to lot of policy as well as real-life constraints. For e.g. I remember in my school days, we used to go out once or twice a year and that used to be either a school picnic or something similar. The only other I know is going to Mumbai for Nehru planetarium and Nehru science Centre. Unfortunately, I didn t go at that time because the school was taking students via air and the tickets were super costly at the time and that too for a 10 minute journey between the two cities. Those were different days, today you can t have a direct flight between the two cities as it doesn t make an economic sense. It makes more sense to go to Mumbai via train or bus as you will reach Mumbai in about couple of hrs. Of course, Pune does have its own planetarium at New English school and there are a few amateur astronomy clubs in Pune but nothing on the scale that what Mumbai has, but then this is getting off-topic. Now, again in an online world could this be done? Not without both the teacher and the student both spending lot of resources online and even then will be a lower understanding as both the hands-on experience as well as interacting with other students and learning from other students (aping) would be hugely limited. Even the social skills that students develop in a school setting will be rusted. My own social skills probably have weakened and rusted as I have very limited interaction with people over the past few months due to Covid fears and would be at least for the next few years till a large enough population is not vaccinated.

Following up to yesterday s article about how NNCP rehabilitates asynchronous communication with modern encryption and onion routing, here is the first of my posts showing how to put it into action. Email is a natural fit for async; in fact, much of early email was carried by UUCP. It is useful for an airgapped machine to be able to send back messages; errors from cron, results of handling incoming data, disk space alerts, etc. (Of course, this would apply to a non-airgapped machine also). The NNCP documentation already describes how to do this for Postfix. Here I will show how to do it for Exim. A quick detour to UUCP land When you encounter a system such as email that has instructions for doing something via UUCP, that should be an alert to you that here is some very relevant information for doing this same thing via NNCP. The syntax is different, but broadly, here s a table of similar NNCP commands:

Purpose	UUCP	NNCP
Connect to remote system	uucico -s, uupoll	nncp-call, nncp-caller
Receive connection (pipe, daemon, etc)	uucico (-l or similar)	nncp-daemon
Request remote execution, stdin piped in	uux	nncp-exec
Copy file to remote machine	uucp	nncp-file
Copy file from remote machine	uucp	nncp-freq
Process received requests	uuxqt	nncp-toss
Move outbound requests to dir (for USB stick, airgap, etc)	N/A	nncp-xfer
Create streaming package of outbound requests	N/A	nncp-bundle

If you used UUCP back in the day, you surely remember bang paths. I will not be using those here. NNCP handles routing itself, rather than making the MTA be aware of the network topology, so this simplifies things considerably. Sending from Exim to a smarthost One common use for async email is from a satellite system: one that doesn t receive mail, or have local mailboxes, but just needs to get email out to the Internet. This is a common situation even for conventionally-connected systems; in Exim speak, this is a satellite system that routes mail via a smarthost. That is, every outbound message goes to a specific target, which then is responsible for eventual delivery (over the Internet, LAN, whatever). This is fairly simple in Exim. We actually have two choices for how to do this: bsmtp or rmail mode. bsmtp (batch SMTP) is the more modern way, and is essentially a derivative of SMTP that explicitly can be queued asynchronously. Basically it s a set of SMTP commands that can be saved in a file. The alternative is rmail (which is just an alias for sendmail these days), where the data is piped to rmail/sendmail with the recipients given on the command line. Both can work with Exim and NNCP, but because we re doing shiny new things, we ll use bsmtp. These instructions are loosely based on the Using outgoing BSMTP with Exim HOWTO. Some of these may assume Debianness in the configuration, but should be easily enough extrapolated to other configs as well. First, configure Exim to use satellite mode with minimal DNS lookups (assuming that you may not have working DNS anyhow). Then, in the Exim primary router section for smarthost (router/200_exim4-config_primary in Debian split configurations), just change transport = remote_smtp_smarthost to transport = nncp. Now, define the NNCP transport. If you are on Debian, you might name this transports/40_exim4-config_local_nncp:

nncp:
  debug_print = "T: nncp transport for $local_part@$domain"
  driver = pipe
  user = nncp
  batch_max = 100
  use_bsmtp
  command = /usr/local/nncp/bin/nncp-exec -noprogress -quiet hostname_goes_here rsmtp
.ifdef REMOTE_SMTP_HEADERS_REWRITE
  headers_rewrite = REMOTE_SMTP_HEADERS_REWRITE
.endif
.ifdef REMOTE_SMTP_RETURN_PATH
  return_path = REMOTE_SMTP_RETURN_PATH
.endif

This is pretty straightforward. We pipe to nncp-exec, run it as the nncp user. nncp-exec sends it to a target node and runs whatever that node has called rsmtp (the command to receive bsmtp data). When the target node processes the request, it will run the configured command and pipe the data in to it. More complicated: Routing to various NNCP nodes Perhaps you would like to be able to send mail directly to various NNCP nodes. There are a lot of ways to do that. Fundamentally, you will need a setup similar to the UUCP example in Exim s manualroute manual, which lets you define how to reach various hosts via UUCP/NNCP. Perhaps you have a star topology (every NNCP node exchanges email with a central hub). In the NNCP world, you have two choices of how you do this. You could, at the Exim level, make the central hub the smarthost for all the side nodes, and let it redistribute mail. That would work, but requires decrypting messages at the hub to let Exim process. The other alternative is to configure NNCP to just send to the destinations via the central hub; that takes advantage of onion routing and doesn t require any Exim processing at the central hub at all. Receiving mail from NNCP On the receiving side, first you need to configure NNCP to authorize the execution of a mail program. In the section of your receiving host where you set the permissions for the client, include something like this:

      exec:  
        rsmtp: ["/usr/sbin/sendmail", "-bS"]
       

The -bS option is what tells Exim to receive BSMTP on stdin. Now, you need to tell Exim that nncp is a trusted user (able to set From headers arbitrarily). Assuming you are running NNCP as the nncp user, then add MAIN_TRUSTED_USERS = nncp to a file such as /etc/exim4/conf.d/main/01_exim4-config_local-nncp. That s it! Some hosts, of course, both send and receive mail via NNCP and will need configurations for both.

Over the course of the last year and a half, I ve been doing some self-directed learning on how radios work. I ve gone from a very basic understanding of wireless communications (there s usually some sort of antenna, I guess?) all the way through the process of learning about and implementing a set of libraries to modulate and demodulate data using my now formidable stash of SDRs. I ve been implementing all of the RF processing code from first principals and purely based on other primitives I ve written myself to prove to myself that I understand each concept before moving on. I figured that there was a fun capstone to be done here - the blind reverse engineering and implementation of the protocol my cheep Amazon power switch uses to turn on and off my Christmas Tree. All the work described in this post was done over the course of a few hours thanks to help during the demodulation from Tom Bereknyei and hlieberman.

Going in blind When I first got my switch, I checked it for any FCC markings in order to look up the FCC filings to determine the operational frequency of the device, and maybe some other information such as declared modulation or maybe even part numbers and/or diagrams. However, beyond a few regulatory stickers, there were no FCC ids or other distinguishing IDs on the device. Worse yet, it appeared to be a whitelabeled version of another product, so searching Google for the product name was very unhelpful. Since operation of this device is unlicensed, I figured I d start looking in the ISM band. The most common band used that I ve seen is the band starting at 433.05MHz up to 434.79MHz. I fired up my trusty waterfall tuned to a center frequency of 433.92MHz (since it s right in the middle of the band, and it let me see far enough up and down the band to spot the remote) and pressed a few buttons. Imagine my surprise when I realize the operational frequency of this device is 433.920MHz, exactly dead center. Weird, but lucky! After taking a capture, I started to look at understanding what the modulation type of the signal was, and how I may go about demodulating it. Using inspectrum, I was able to clearly see the signal in the capture, and it immediately stuck out to my eye to be encoded using OOK / ASK. Next, I started to measure the smallest pulse, and see if I could infer the symbols per second, and try to decode it by hand. These types of signals are generally pretty easy to decode by eye. This wound up giving me symbol rate of 2.2 Ksym/s, which is a lot faster than I expected. While I was working by hand, Tom demodulated a few messages in Python, and noticed that if you grouped the bits into groups of 4, you either had a 1000 or a 1110 which caused me to realize this was encoded using something I saw documented elsewhere, where the 0 is a short pulse, and a 1 is a long pulse, not unlike morse code, but where each symbol takes up a fixed length of time (monospace morse code?). Working on that assumption, I changed my inspectrum symbol width, and demodulated a few more by hand. This wound up demodulating nicely (and the preamble / clock sync could be represented as repeating 0s, which is handy!) and gave us a symbol rate of 612(ish) symbols per second a lot closer to what I was expecting. If we take the code for on in the inspectrum capture above and demodulate it by hand, we get 0000000000110101100100010 (treat a short pulse as a 0, and a long pulse as a 1). If you re interested in following along at home, click on the inspectrum image, and write down the bits you see, and compare it to what I have! Right, so it looks like from what we can tell so far that the packet looks something like this:

preamble / sync

stuff

Next, I took a capture of all the button presses and demodulated them by hand, and put them into a table to try and understand the format of the messages:

Button	Demod'd Bits
On	0000000000110101100100010
Off	00000000001101011001010000
Dim Up	0000000000110101100110100
Dim Down	0000000000110101100100100
Timer 1h	0000000000110101100110010
Timer 2h	0000000000110101100100110
Timer 4h	0000000000110101100100000
Dim 100%	0000000000110101000101010
Dim 75%	00000000001101010001001100
Dim 50%	00000000001101010001001000
Dim 25%	0000000000110101000100000

Great! So, this is enough to attempt to control the tree with, I think so I wrote a simple modulator. My approach was to use the fact that I can break down a single symbol into 4 sub-symbol components which is to say, go back to representing a 1 as 1110, and a 0 as 1000. This let me allocate IQ space for the symbol, break the bit into 4 symbols, and if that symbol is 1, write out values from a carrier wave (cos in the real values, and sin in the imaginary values) to the buffer. Now that I can go from bits to IQ data, I can transmit that IQ data using my PlutoSDR or HackRF and try and control my tree. I gave it a try, and the tree blinked off! Success! But wait that s not enough for me I know I can t just demodulate bits and try and replay the bits forever there s stuff like addresses and keys and stuff, and I want to get a second one of these working. Let s take a look at the bits to see if we spot anything fun & interesting. At first glance, a few things jumped out at me as being weird? First is that the preamble is 10 bits long (fine, let s move along - maybe it just needs 8 in a row and there s two to ensure clocks sync?). Next is that the messages are not all the same length. I double (and triple!) checked the messages, and it s true, the messages are not all the same length. Adding an extra bit at the end didn t break anything, but I wonder if that s just due to the implementation rather than the protocol. But, good news, it looks like we have a stable prefix to the messages from the remote must be my device s address! The stable 6 bits that jump out right away are 110101. Something seems weird, though, 6 bits is a bit awkward, even for a bit limited embedded device. Why 6? But hey, wait, we had 10 bits in the preamble, what if we have an 8 bit address meaning my device is 00110101, and the preamble is 8 0 symbols! Those are numbers that someone working on an 8 bit aligned platform would pick! To test this, I added a 0 to the preamble to see if the message starts at the first 1, or if it requires all the bits to be fully decoded, and lo and behold, the tree did not turn on or off. This would seem to me to confirm that the 0s are part of the address, and I can assume we have two 8 bit aligned bytes in the prefix of the message.

preamble / sync

address

stuff

Now, when we go through the 9-10 bits of stuff , we see all sorts of weird bits floating all over the place. The first 4 bits look like it s either 1001 or 0001, but other than that, there s a lot of chaos. This is where things get really squishy. I needed more information to try and figure this out, but no matter how many times I sent a command it was always the same bits (so, no counters), and things feel very opaque still. The only way I was going to make any progress is to get another switch and see how the messages from the remote change. Off to Amazon I went, and ordered another switch from the same page, and eagerly waited its arrival.

Switch #2 The second switch showed up, and I hurriedly unboxed the kit, put batteries into the remote, and fired up my SDR to take a capture. After I captured the first button ( Off ), my heart sunk as I saw my lights connected to Switch #1 flicker off. Apparently the new switch and the old switch have the same exact address. To be sure, I demodulated the messages as before, and came out with the exact same bit pattern. This is a setback and letdown I was hoping to independently control my switches, but it also means I got no additional information about the address or button format. The upside to all of this, though, is that because the switches are controlled by either remote, I only needed one remote, so why not pull it apart and see if I can figure out what components it s using to transmit, and find any datasheets I can. The PCB was super simple, and I wound up finding a WL116SC IC on the PCB. After some googling, I found a single lone datasheet, entirely in Chinese. Thankfully, Google Translate seems to have worked well enough on technical words, and I was able to put together at least a little bit of understanding based on the documentation that was made available. I took a few screenshots below - I put the google translated text above the hanzi. From that sheet, we can see we got the basics of the 1 and 0 symbol encoding right (I was halfway expecting the bits to be flipped), and a huge find by way of a description of the bits in the message! It s a bummer that we missed the clock sync / preamble pulse before the data message, but that s OK somehow. It also turns out that 8 or 10 bit series of of 0"s wasn t clock sync at all - it was part of the address! Since it also turns out that all devices made by this manufacturer have the hardcoded address of []byte 0x00, 0x35 , that means that the vast majority of bits sent are always going to be the same for any button press on any remote made by this vendor. Seems like a waste of bits to me, but hey, what do I know. Additionally, this also tells us the trailing zeros are not part of the data encoding scheme, which is progress!

address

keycode

Now, working on the assumptions validated by the datasheet, here s the updated list of scancodes we ve found:

Button	Scancode Bits	Integer
On	10010001	145 / 0x91
Off	10010100	148 / 0x94
Dim Up	10011010	154 / 0x9A
Dim Down	10010010	146 / 0x92
Timer 1h	10011001	154 / 0x99
Timer 2h	10010011	147 / 0x93
Timer 4h	10010000	144 / 0x90
Dim 100%	00010101	21 / 0x15
Dim 75%	00010011	19 / 0x13
Dim 50%	00010010	18 / 0x12
Dim 25%	00010000	16 / 0x10

Interestingly, I think the Dim keys may have a confirmation that we have a good demod the codes on the bottom are missing the most significant bit, and when I look back at the scancode table in the datasheet, they make an interesting pattern the bottom two rows, right and left side values match up! If you take a look, Dim 100% is S1 , Dim 75% is S19 , Dim 50% is S8 , and Dim 25% is S20 . Cool! Since none of the other codes line up, I am willing to bet the most significant bit is a Combo indicator, and not part of the button (leaving 7 bits for the keycode). And even more interestingly, one of our scancodes ( Off , which is 0x94) shows up just below this table, in the examples. Over all, I think this tells us we have the right bits to look at for determining the scan code! Great news there!

Back to the modulation! So, armed with this knowledge, I was able to refactor my code to match the timings and understanding outlined by the datasheet and ensure things still work. The switch itself has a high degree of tolerance, so being wildly off frequency or a wildly wrong symbol rate may actually still work. It s hard to know if this is more or less correct, but matching documentation seems like a more stable foundation if nothing else. This code has been really reliable, and tends to work just as well as the remote from what I ve been able to determine. I ve been using incredibly low power to avoid any interference, and it s been very robust - a testament to the engineering that went into the outlet hardware, even though it cost less than of a lot of other switches! I have a lot of respect for the folks who built this device - it s incredibly simple, reliable and my guess is this thing will keep working even in some fairly harsh RF environments. The only downside is the fact the manufacturer used the same address for all their devices, rather than programming a unique address for each outlet and remote when the underlying WL116SC chip supports it. I m sure this was done to avoid complexity in assembly (e.g. pairing the remote and outlet, and having to keep those two items together during assembly), but it s still a bummer. I took apart the switch to see if I could dump an EEPROM and change the address in ROM, but the entire thing was potted in waterproof epoxy, which is a very nice feature if this was ever used outdoors. Not good news for tinkering, though!

Unsolved Mysteries At this point, even though I understand the protocol enough to control the device, it still feels like I hit a dead end in my understanding. I m not able to figure out how exactly the scancodes are implemented, and break them down into more specific parts. They are stable and based on the physical wiring of the remote, so I think I m going to leave it a magic number. I have what I was looking for, and these magic constants appear to be the right one to use, even if I did understand how to create the codes itself. This does leave us with a few bits we never resolved, which I ll memorialize below just to be sure I don t forget about them. Question #1: According to the datasheet there should be a preamble. Why do I not see one leading the first message? My hunch is that the trailing 0 at the end of the payload is actually just the preamble for the next message (always rendering the first message invalid?). This would let us claim there s an engineering reason why we are ignoring the weird bit, and also explain away something from the documentation. It s just weird that it wouldn t be present on the first message. This theory is mostly confirmed by measuring the timing and comparing it to the datasheet, but it s not exactly in line with the datasheet timings either (specifically, it s off by 200 s, which is kinda a lot for a system using 400 s timings). I think I could go either way on the last 0 being the preamble for the next message. It could be that the first message is technically invalid, or it could also be that this was not implemented or actively disabled by the vendor for this specific application / device. It s really hard to know without getting the source code for the WL116SC chip in this specific remote or the source in the outlet itself. Question #2: Why are some keycodes 8 bits and others 9 bits? I still have no idea why there sometimes 8 bits (for instance, On ) and other times there are 9 bits (for instance, Off ) in the 8 bit keycode field. I spent some time playing with the trailing zeros, when I try and send an Off with the most significant 8 bits (without the least significant / last 9th bit, which is a 0 ), it does not turn the tree off. If I send an On with 9 bits (an additional 0 after the least significant bit), it does work, but both On and Off work when I send 10, 11 or 12 bits padded with trailing zeros. I suspect my outlet will ignore data after the switch is done reading bits regardless of trailing zeros. The docs tell me there should only be 8 bits, but it won t work unless I send 9 bits for some commands. There s something fishy going on here, and the datasheet isn t exactly right either way. Question #3: How in the heck do those scancodes work? This one drove me nuts. I ve spent countless hours on trying to figure this out, including emailing the company that makes the WL116SC (they re really nice!), and even though they were super kind and generous with documentation and example source, I m still having a hard time lining up their documentation and examples with what I see from my remote. I think the manufacturer of my remote and switch has modified the protocol enough to where there s actually something different going on here. Bummer. I wound up in my place of last resort asking friends over Signal to try and see if they could find a pattern, as well as making multiple pleas to the twittersphere, to no avail (but thank you to Ben Hilburn, devnulling, Andreas Bombe and Larme for your repiles, help and advice!) I still don t understand how they assemble the scan code for instance, if you merely add, you won t know if a key press of 0x05 is 0x03 + 0x02 or if it s 0x01 + 0x04. On the other hand, treating it as two 4-bit integers won t work for 0x10 to 0x15 (since they need 5 bits to represent). It s also likely the most significant bit is a combo indicator, which only leaves 7 bits for the actual keypress data. Stuffing 10 bits of data into 7 bits is likely resulting in some really intricate bit work. On a last ditch whim, I tried to XOR the math into working, but some initial brute forcing to make the math work given the provided examples did not result in anything. It could be a bitpacked field that I don t understand, but I don t think I can make progress on that without inside knowledge and much more work. Here s the table containing the numbers I was working off of:

Keys	Key Codes	Scancode
S3 + S9	0x01 + 0x03	0x96
S6 + S12	0x07 + 0x09	0x94
S22 + S10	0x0D + 0x0F	0x3F

If anyone has thoughts on how these codes work, I d love to hear about it! Send me an email or a tweet or something - I m a bit stumped. There s some trick here that is being used to encode the combo key in a way that is decodeable. If it s actually not decodeable (which is a real possibility!), this may act as a unique button combo hash which allows the receiver to not actually determine which keys are pressed, but have a unique button that gets sent when a combo is used. I m not sure I know enough to have a theory as to which it may be.

Review: The Biggest Bluff, by Maria Konnikova

Publisher:	Penguin Press
Copyright:	2020
ISBN:	0-525-52263-8
Format:	Kindle
Pages:	335

After a particularly unlucky year for her family, Maria Konnikova was reading about the balance between luck and control in life and discovered, to her surprise, that John von Neumann, one of the foundational thinkers of computer science, was fascinated by poker. He found most card games boring because they relied on luck. Poker, however, he thought was the perfect balance between luck and skill: enough skill to make its effect undeniable, but enough luck that one could not control the game fully regardless of skill. Konnikova decided on a research project: spend one year learning No Limit Texas Hold'em from one of the best poker players in the world, with a goal of competing in the World Series of Poker. She had studied the description-experience gap during her doctoral research in psychology and wanted to see if the experience of randomness in poker would teach her something the description of the randomness of life could not. Before starting this project, she didn't know the basic rules and had never watched a game. Erik Seidel agreed to mentor her, and The Biggest Bluff is her account of that experience. This book is simultaneously frustrating and fascinating in ways that I don't think can be untangled without making it a far different book. Fitting, I suppose, for a book about how our brains entangle luck and skill. First, if you're looking for a book about poker play, this is not one. Konnikova rarely talks about specific hands or tournaments in more detail than her overall trajectory. That was a disappointment. In the few places she does describe some of her betting decisions, analysis of the other players, and tournament strategy, her accounts are engrossing and suspenseful. I would have happily read a book chronicling her poker tournaments and the decisions she made, but this is not that book. What The Biggest Bluff is instead is a psychological and philosophical examination of the process of learning poker. Konnikova uses her experiences as launching points into philosophical digressions. Even the lessons that have limited surface utility outside of poker, such as learning to suppress body language to avoid giving away information, turn into digressions about interpersonal dynamics and personality types. There are interesting tidbits here, but I've read a lot of popular psychology and was more interested in the poker. My frequent reading experience was impatiently waiting for Konnikova to finish lecturing and get back to her narrative. What Konnikova does do though, at a level that I haven't seen before before in a book of this type, is be brutally honest about her mistakes and her learning process. And I do mean brutally: The book opens with her throwing up in a casino bathroom. (This is not a fun book to read if you don't like reading about stress reactions and medical problems. There aren't many of them, but they're... memorable.) Konnikova knows and can explain the psychological state she's trying to reach, but still finds it hard to do in the moment. Correctly reacting to probabilities, cutting losses, and neither being too over-confident or too scared is very hard. Most books of this type elide over the repeated failures in a sort of training montage, which makes the process look easier than it feels. Konnikova tries to realistically show the setbacks and failures, and I think succeeds. That relentless introspection and critical honesty is the best part of this book, but I think it's also behind the stream of consciousness digressions about psychology and philosophy. It's a true portrayal of how Konnikova makes sense of the world. A more polished and streamlined book about the poker would have been more dramatically engrossing, but it might have lost the deep examination of how she combines poker with her knowledge of psychology to change her thinking. The one place where I think that self-reflection may fall a bit short, which I want to mention because I thought it was a missed opportunity, is around knowledge of hand probabilities. Konnikova makes a point, early in the book, of not approaching poker through the memorization and mathematics route and instead trying to find a play style that focuses on analyzing the other players and controlling her own emotions. This is a good hook, but by the end of the book it's not entirely true. The point that I think she was trying to make is that her edge against other poker players at her same level comes more from psychology than from calculation of precise odds in rare situations. This is true. But by the time she reaches high levels of play, she is using statistical simulators, practice tools, and intensive study just like any professional poker player. There is a minimum level of pure knowledge and memorization required that cannot be avoided. It's clear from the few things she says about this that those tools became more interesting to her as she became better at poker, and I wish she would have dug more into why and how that happened. How much of her newfound ability to make decisions and stick to a plan comes from emotional changes, and how much from that background store of confident knowledge? Or maybe those are different ways of looking at the same change? I did appreciate Konnikova's explicit acknowledgment at the end of the book that poker did not, in the end, provide some deep insight into the balance of luck and skill in real life. Learning poker instead gave Konnikova more personal ability to make a plan for the things that she can control and let go of the things she can't. I'm glad that worked for her, but since reading this book I have noticed former poker players who think life is more knowable than it is. Poker combines random chance with psychological play against other people, but it does so in a way and to an extent that is quantifiable. You can make the correct play and still lose a hand, but you can also know when this has happened. When you're used to analyzing the world through that frame and real life fails to provide that certainty, it's tempting to impose it anyway and insist your simplified models are more accurate because they're more comprehensible. But poker is a game, not a model; being more predictable and more constrained than real life is part of what makes it fun. The skill that Konnikova learned from it has a potential downside that she doesn't talk about. I'm not sure how to sum up this book. Konnikova's internal analysis and honesty is truly admirable and illuminating, but it left me wanting to read a different book that was more focused on poker narration. I know there are lots of those books out there, but I doubt they would be written with Konnikova's self-awareness and lack of ego. However, they would probably also lack the moments that made me cringe or that were deeply uncomfortable to read. My feelings are mixed. But if you want popular psychology wrapped around a deeply honest account of the process of learning poker, I suspect this book is one of a kind. Rating: 7 out of 10

Have you ever been traveling, shot a ton of photos and videos, but were annoyed to find it was saturating the terrible wifi you had access to? Maybe you d wish the upload to pause until you get somewhere else, but then pausing syncing on your Nextcloud/Syncthing/Dropbox would also pause other syncing you didn t want to pause. Or you have trouble backing up your laptop when not at home, in a way that won t accidentaly eat up your cell phone data. There are ways to help with this: asynchronous transfer. Here s a lot of background. If you want to see how encrypted, onion-routed UUCP looks, skip ahead to the NNCP section! There is an old saying: When all you have is a hammer, every problem looks like a nail. We have this wonderful tool called ssh available, and it is pervasive and well-understood, so we tend to use it. But we ve missed out on some benefits of asynchronous processing that we actually used to have more frequently. Of course, we are all used to some asynchronous services in our lives. Email is a popular example: most mail clients work offline and will transmit stored messages when the mail server becomes reachable. Mail servers themselves work that way, too. Many instant messaging platforms do as well. Even some backup systems do. Bacula/Bareos, for instance, spools all backup data to disk on the system connected to the tape drive, and from there to the tape itself. They do this for several reasons, but primarily the fact that if tape drives are not fed with data at their design speed, it can cause physical damage to the tape or even the drive. It causes the drive to have to pause, and seek backwards to reposition for the next write. This creates excessive travel of the tape over the write heads, causing a condition known as tape shine where the tape is damaged prematurely. Here are some problems people often run into when sending data across a network (or the Internet) synchronously:

• One side of the communication is much faster than the other
• Internet issues interrupting communications mid-stream
• Slow Internet causing processes to take much longer than planned, resulting in unexpected results or locking issues
• Physical damage due to performance issues

Of course, there are plenty of situations where synchronous communication is a must. For instance:

• When the status of the transaction at the remote end must be known immediately
• When there is insufficient space to spool a job s data

I suspect that the reason we don t do more asynchronous processing these days, despite it being strong in the Unix heritage, is the lack of modern tools to do it. Let s explore some more. Some of my use cases I run ZFS on all my systems that support it: file server, laptops, workstations, etc. It is only natural to use ZFS send/receive to do backups, and I do. However, when I am traveling, my laptop never gets backed up, because the backups are pulled from the backup system. Sure, there are ways around that; a VPN, for instance. But then we have the situation where sometimes I do not want to send the backup even if I have a working Internet connection: perhaps I m tethered to a mobile connection and it would be expensive to do so, or I m on hotel Wifi that is flaky and slow and I don t want to give up any of its meager bandwidth. I have another backup-related problem. I have a remote server, which until recently was using extremely slow disks. If I made significant changes, the backup would take the better part of a day. That s annoying when I try to back up hourly. So of course I had to implement locking, but then that means none of my other machines would back up that day either. Once I needed to transmit about 2TB of data. My home Internet connection was terribly slow, and I calculated it would take multiple months to do this. So I took to manually copying parts of the data to my laptop, and whenever I d find an airport or coffee shop with faster Internet than at home, I d send off those bits from it. But it took a ton of work. The bespoke asynchronous problem And that ton of work is perhaps why we aren t doing more of this. There s been no great standard solution, so it s all roll your own when you need to. So we just use ssh, because it s easier and usually good enough . But as I wrote in my recent article on airgapped backups, there are reasons to go async. Solutions Wouldn t it be great to be able to queue up data for a machine, and let it get there in whatever way it can? Maybe a fast Internet connection is found, or via Tor, or via copying to a USB stick, or via radio broadcast? It would make many of these scenarios a lot easier. And there are ways for this now, with modern security! We have some tools on Linux for this: git-annex for storage and migration, syrep for synchronization, and NNCP for file transfer and remote execution (could be combined with some of these other tools). Let s dive in to NNCP. NNCP If you already know UUCP, think of NNCP as UUCP brought into the modern era, with modern security and tools. Basically, NNCP permits you to send files to a remote system, request files from a remote system, and pipe data to an NNCP command that requests execution remotely. So you could, say, pipe a zfs send to NNCP which sends it to the remote and pipes it to zfs receive when it gets there. NNCP has a delay-tolerant, resumable protocol that can run over just about any reliable connection: TCP, serial, Tor, radios of various kinds, you name it. But that s not all; it also can dump its queue onto something like a USB stick for transport, or even make a tar-style stream that could be munged however you like. If you want to get fancy, you can assign priorities to data packets, so that, for instance, outbound email will always get sent before that 1TB file you ve got to send also. You can also configure it so that certain carriers handle certain priorities of data; your cell phone would only handle the most urgent, but a USB stick would take anything. NNCP is source-routed; you can tell it that the way that Bob reaches Alice is via Carl, then Betty. Bob can generate a message that will be sent along that route, fully encrypted and authenticated at each step of the way; Carl can t see the content of the message or even anything about it other than its next hop. How this helps Let s revisit some of my scnearios with NNCP. For the laptop being backed up, while traveling it can queue up its backups, or photos, or videos, or whatever. They could be triggered by a command when on a good connection, or automatically. The data could be copied to USB and given to a friend to transmit; perfectly safe due to encryption. Or it could all wait until arriving at home, safely out of your other syncing directories. The NNCP documentation has an example of this. For the server being backed up slowly, that s easily solved; the slow backup would simply be queued up, and transmitted and processed when it s ready. This wouldn t interrupt other backups. How about the 2TB transmission problem? That s also made a lot easier. A command could be run to fill up a USB stick with parts of the queue, then that USB stick plugged in and transmitted whenever at a fast location. Repeat as needed while the slow system continues its upload of the remaining bits. NNCP has a lot of interesting use cases documented as well. If you are already familiar with how public keys work in SSH, then NNCP should be immediately familiar as well. It is a similar concept (though arguably somewhat easier to set up). I am working on setting up a NNCP network, and will have more posts on how to do so once I ve got it going. In the meantime, the documentation for the project is also pretty good.

I have updated my debpkg plugin for IDEA (e.g. IntelliJ, PyCharm, Android Studios) to v0.0.8. Here are some of the changes since last time I wrote about the plugin. New file types supported

• .dsc (as a generic deb822 file)
• .buildinfo and .changes (as a generic deb822 provided the base name looks like it follows the package_version_arch pattern)
• debian/copyright (as a DEP-5 / machine-readable copyright file if the Format field exists).

Links for URLs and bug closes There are often links in deb822 files or the debian/changelog and as of v0.0.8, the plugin will now highlight them and able you to easily open them via your browser. In the deb822 case, they generally appear in the Homepage field, the Vcs-* fields or the Format field of the debian/copyright field. For the changelog file, they often appear in the form of bug Closes statements such as the #123456 in Closes: #123456 , which is a reference to https://bugs.debian.org/123456. Improvements to debian/control The dependency validator now has per-field knowledge. This enables it to flag dependency relations in the Provides field that uses operators other than = (which is the only operator that is supported in that field). It also knows which fields support build-profile restrictions. It in theory also do Architecture restrictions, but I have not added it among other because it gets a bit spicy around binary packages. (Fun fact, you can have Depends: foo [amd64] but only for arch:any packages.) The plugin now suggests adding a Rules-Requires-Root field to the Source stanza along with a quick fix for adding the field. Admittedly, it was mostly done as exercise for me to learn how to do that kind of feature. Support for machine-readable debian/copyright The plugin now has a dedicated file type for debian/copyright that follows the machine-readable format. It should auto-detect it automatically based on the presence of the Format field being set to https://www.debian.org/doc/packaging-manuals/copyright-format/1.0. Sadly, I have not found the detection reliable in all cases, so you may have to apply it manually. With the copyright format, the plugin now scans the Files fields for common issues like pointing on non-existing paths and invalid escape sequences. When the plugin discovers a path that does not match anything, it highlights the part of the path that cannot be found. As an example, consider the pattern src/foo/data.c and that src/foo exist but data.c does not exist, then the plugin will only flag the data.c part of src/foo/data.c as invalid. The plugin will also suggest a quick fix if you a directory into the Files field to replace it with a directory wildcard (e.g. src/foo -> src/foo/* ), which is how the spec wants you to reference every file beneath a given directory. Finally, when the plugin can identify part of the path, then it will turn it into a link (reference in IDEA lingo). This means that you can CTRL + click on it to jump to the file. As a side-effect, it also provides refactoring assistance for renaming files, where renaming a file will often be automatically reflected in debian/copyright. This use case is admittedly mostly relevant people, who are both upstream and downstream maintainer. Folding support improvement for .dsc/.changes/.buildinfo files The new field types appeared with two cases, where I decided to improve the folding support logic. The first was the GPG signature (if present), which consists of two parts. The top part with is mostly a single line marker but often followed by a GPG armor header (e.g. Hash: SHA512 ) and then the signature blob with related marker lines around it. Both cases are folded into a single marker line by default to reduce their impact on content in the editor view. The second case was the following special-case pattern:

Files:
 <md5> <size> filename
Checksums-Sha256:
 <sha256> <size> filename

In the above example, where there is exactly on file name, those fields will by default now be folded into:

Files: <md5> <size> filename
Checksums-Sha256: <sha256> <size> filename

For all other multi-line fields, the plugin still falls back to a list of known fields to fold by default as in previous versions. Spellchecking improvements The plugin already supported selective spell checking in v0.0.3, where it often omitted spell checking for fields (in deb822 files) where it did not make sense. The spell check feature has been improved by providing a list of known packaging terms/jargo used by many contributors (so autopkgtests is no longer considered a typo). This applies to all file types (probably also those not handled by the plugin as it is just a dictionary). Furthermore, the plugin also attempts discover common patterns (e.g. file names or command arguments) and exempt these from spell checking in the debian/changelog. This also includes manpage references such as foo.1 or foo(1) . It is far from perfect and relies on common patterns to exclude spell checking. Nonetheless, it should reduce the number of false positive considerably. Feedback welcome Please let me know if you run into bugs or would like a particular feature implemented. You can submit bug reports and feature requests in the issue tracker on github.

Review: Anti-Social, by Nick Pettigrew

Publisher:	Century
Copyright:	2020
ISBN:	1-4735-7639-3
Format:	Kindle
Pages:	375

Nick Pettigrew was an anti-social behavior officer for council housing for over a decade. This book is a diary of one year in the life of that job. According to the author, the dates, names, and some of the identifying details have been changed but not exaggerated. While it's hard to verify that, and I'm sure a few of the stories have been tweaked to make better narratives, it has a messiness and chaos of half-resolved drama mixed with mundane office tedium that makes it ring true. The subject matter requires some explanation for US readers. Council housing is the UK's version of what the US would call public housing or "the projects," although it's a far larger percentage of the housing stock in the UK than it is in the US. These are buildings built by the government and run by local governments or non-profits to provide housing to working class and poor people at reduced cost compared to the private market. As in the US, new council housing construction has dropped off dramatically; as in the US, a bunch of it has been privatized, although in the UK the Right to Buy law at least sells the housing to the people living in it, as opposed to slumlords. The 1998 UK Crime and Disorder Act collects under anti-social behavior a bunch of low-grade irritations that in the US would fall under public nuisance laws, noise ordinances, and other vague misdemeanors like "creating a public disturbance": essentially, all the irritating and obnoxious things that people subject their neighbors to, but which fall short of obvious criminal behavior like theft or assault. As Pettigrew puts it:

"I just don't want to hear the word 'cunt' shouted repeatedly at 3am every night any more. I don't think that's unreasonable."

That law required the organizations managing council estates to have a procedure to deal with anti-social behavior, and thus the job of an anti-social behavior officer was created. In the US, these sorts of problems are dealt with by the local police, which is a little like trying to get rid of a raccoon by running it over with a balky, thirty-year-old car. The best outcome is that the raccoon gets freaked out by your efforts to get the car to start and leaves. The worst outcome is that you accidentally run over your neighbor's kid while aiming for the raccoon. Either way, neither you nor the car nor your neighbors are going to be happy about the process. An anti-social behavior officer is someone with some authority to do things like install cameras, tell people who don't live somewhere that they need to stop coming there, and start eviction proceedings, but who doesn't carry guns, can't arrest people, and enforces those rules when necessary via court proceedings or calling the police. In other words, it's one of the types of alternative responders whose job is to solve problems rather than stop crimes that we've been talking about in the United States. And although Pettigrew is quite grumpy and cynical about his job (for very understandable reasons explained in this book), my takeaway from another country with even worse problems is that the system sort of works. It's underfunded and poorly managed and nebulous and sometimes very sad, but it also achieves some of what it sets out to do, and you can see the outlines of a better system beneath it. My favorite thing about this book is that it forces the reader to grapple with the actual job in all its messiness, rather than our ideals of what we want the job to be. For example, if you're going to hire a bunch of people to do this job, you need some way to figure out whether they're any good at it, but it's not obvious how to measure success.

One infuriating KPI we're measured on is the final question a resident is asked once their case is closed: "Are you happy with the outcome of your case?" Because unless you evict the problem neighbor within a fortnight, erect a security wall around the complainant's house manned by armed guards, and bake them a cake, the answer will almost certainly be no.

Pettigrew has a background in comedy so he's sarcastically funny rather than strident or didactic, which for me was just the tone I needed to enjoy reading this enough to think deeply about it. The thing that most surprised me, even though it probably shouldn't, was just how many of the problems Pettigrew had to deal with stemmed from drugs. My guess is that a good 75% of the issues he described were a direct result of someone dealing drugs, growing drugs, stealing things to buy drugs, throwing parties with druggie friends, doing bizarre and dangerous things because they were on drugs, not taking care of their property because they were too busy doing drugs, or letting their property be taken over by other people doing or selling drugs. It's just never-ending. The case load is clearly made worse by chronic understaffing and underfunding, and spending more money on giving people more assistance is part of any reasonable solution, but so is reducing the demand. If the pattern of complaints in the US is at all similar (and I bet it is), it's no wonder that the police perceive drugs as central to their job. Pettigrew talks about this at some length in the epilogue, and I think one of his conclusions is worth quoting here:

Years of interacting with class A drug users has shown me that they really don't want to be out hustling to make enough money for their next score of diluted heroin. They want to sit at home, take drugs and when those drugs wear off, take some more. You may view this as a wasted life, and you may well be right, but here's the thing: they are going to do this anyway, whether we decide to criminalize them or not. If, after a lengthy period of honest, mature drug education across the country via schools, further education, government advertising and so on, class A drugs were made legally available on prescription to adults via pharmacies, the vast majority of users would pick up their prescriptions, go home, take their drugs and leave everyone else alone. They couldn't be arsed to break into your car if they didn't have to.

The decriminalization argument is a lot larger than the space of this book review, so I won't get into it further, except to say that if you think that conclusion sounds unreasonable or dangerous or destructive to society, consider reading this book. One of the things that becomes obvious from Pettigrew's diary of incidents is that most people just want to feel safe, and a truly stunning amount of crime and anti-social behavior that makes them feel not safe is a direct or indirect result of people trying to get drugs or to get money to get drugs. The other pattern that's obvious from this book (and obvious from most writing by anyone involved in the justice system) is that the system doesn't really work.

The best way to describe going to court is that it's like paying a visit to a run-down hospital where nobody gets better.

Some of that is because it's not set up to solve problems, and a lot of that is because solving problems is a lot more expensive than we want to admit and we don't want to pony up the resources. I'm probably making this book sound hopelessly depressing, but it's not. It's angry and cynical, but it's also very funny (particularly if you like sarcasm), extremely quotable, insightful, and deeply empathetic. There are people trying their best to help other people despite a system that doesn't reward them for doing so, and surprisingly often they manage to do so. The path to helping more people isn't even that difficult to find. The epilogue of this book is full of solid and straightforward ideas that don't require some novel insight into humanity, just a bunch of hard work. And, while reading about the work of helping people and being exasperated by people, you'll also get to laugh at the bits of office nonsense that are universal.

The old form for monthly one-to-ones used to have two questions "What did you do over the last month?" and "What do you plan to do in the next month?" and my line manager at the time seemed unimpressed when I answered "my job" to both.

I loved this book. It was one of my favorite random finds of the year, and, at least judging from how difficult it is to find in Amazon search results, it's not gotten enough attention in the United States. Highly recommended. Rating: 9 out of 10

I recently took part in an interview with Vladimir Bejdo, an intern at the Software Freedom Conservancy, in order to talk about the Reproducible Builds project, my own participation in software freedom, the importance of reproducibility in software development and to have a brief discussion on the issues facing free software as a whole:

VB: To start off with, it might be useful to first ask you this how would you relate the importance of reproducibility to a user who is non-technical? CL: I sometimes use the analogy of the food supply chain to quickly relate our work to non-technical audiences. The multiple stages of how our food reaches our plates today (such as seeding, harvesting, picking, transportation, packaging, etc.) can loosely translate to how software actually ends up on our computers, particularly in the way that if any of the steps in the multi-stage food supply chain has an issue then it quickly becomes a serious problem. For example, even if we could guarantee that only the most wholesome apples were picked in our orchards, if they became tainted on the way to the supermarket it will be a real problem for us at the end of the day. We may not even be able to even tell by simply inspecting our Pink Ladies or Honeycrisps, and washing them thoroughly under the tap may not be enough either. In an ideal world, we would be able to personally inspect the provenance of our food at all of the stages of manufacturing and transportation. But at some point, we must place our trust in the process and in brands, as well as various regulatory bodies to ensure that potential problems in our food are minimized, possibly even paying a time/effort premium by growing our own or buying direct from local markets in order to minimize the number of steps, etc. However, when we use free software we can do better: Reproducible builds are a set of software development practices, ideas and tools that create an independently-verifiable path all the way from the original source code to what actually runs on our machines. Reproducible builds can reveal the injection of back-doors introduced by the hacking of developers own computers, build servers and package repositories, and also expose where volunteers or companies have been coerced into making changes via blackmail, court order, and so on. With reproducible builds, there is no longer any need to trust any particular source of authority. In the same way that, say, a Mr Smith might check that his calculator is giving him the right answer to 2+2=4 by asking enough of his friends to check theirs too, users and developers of a reproducible build can verify the software they are using by creating a collective consensus instead.

The full interview can be found on the Conservancy webpages.

Another note to myself before I forget about this nifty usage of socat again. I was looking for something to mock a serial device, similar to a microcontroller which usually ends up as /dev/ttyACM0 and might output some text. What I found is a very helpful post on stackoverflow showing an example utilizing socat.

$ socat -d -d pty,rawer pty,rawer
2020/12/20 21:37:53 socat[29130] N PTY is /dev/pts/8
2020/12/20 21:37:53 socat[29130] N PTY is /dev/pts/11
2020/12/20 21:37:53 socat[29130] N starting data transfer loop with FDs [5,5] and [7,7]

Write whatever you need to the second pty, here /dev/pts/11, e.g.

$ i=0; while :; do echo "foo: $ i " > /dev/pts/11; let i++; sleep 5; done

Now you can listen with whatever you like, e.g. some tool you work on, on the fist pty, here /dev/pts/8. For demonstration purpose just use cat:

$ cat /dev/pts/8
foo: 0
foo: 1

socat is an awesome tool, looking through the manpage you need some knowledge about sockets, but it's incredibly vesatile.

Review: Can't Even, by Anne Helen Petersen

Publisher:	Houghton Mifflin
Copyright:	2020
ISBN:	0-358-31659-6
Format:	Kindle
Pages:	230

Like many other people, I first became aware of Anne Helen Petersen's journalism when her Buzzfeed article "How Millennials Became the Burnout Generation" went viral. Can't Even is the much-awaited (at least by me) book-length expansion of that thesis: The United States is, as a society, burning out, and that burnout is falling on millennials the hardest. We're not recognizing the symptoms because we think burnout looks like something dramatic and flashy. But for most people burnout looks less like a nervous breakdown and more like constant background anxiety and lack of energy.

Laura, who lives in Chicago and works as a special ed teacher, never wants to see her friends, or date, or cook she's so tired, she just wants to melt into the couch. "But then I can't focus on what I'm watching, and end up unfocused again, and not completely relaxing," she explained. "Here I am telling you I don't even relax right! I feel bad about feeling bad! But by the time I have leisure time, I just want to be alone!"

Petersen explores this idea across childhood, education, work, family, and parenting, but the core of her thesis is the precise opposite of the pervasive myth that millennials are entitled and lazy (a persistent generational critique that Petersen points out was also leveled at their Baby Boomer parents in the 1960s and 1970s). Millennials aren't slackers; they're workaholics from childhood, for whom everything has become a hustle and a second (or third or fourth) job. The struggle with "adulting" is a symptom of the burnout on the other side of exhaustion, the mental failures that happen when you've forced yourself to keep going on empty so many times that it's left lingering damage. Petersen is a synthesizing writer who draws together the threads of other books rather than going deep on a novel concept, so if you've been reading about work, psychology, stress, and productivity, many of the ideas here will be familiar. But she's been reading the same authors that I've been reading (Tressie McMillan Cottom, Emily Guendelsberger, Brigid Schulte, and even Cal Newport), and this was the book that helped me pull those analyses together into a coherent picture. That picture starts with the shift of risk in the 1970s and 1980s from previously stable corporations with long-lasting jobs and retirement pensions onto individual employees. The corresponding rise in precarity and therefore fear led to a concerted effort to re-establish a feeling of control. Baby Boomers doubled down on personal responsibility and personal capability, replacing unstructured childhood for their kids with planned activities and academic achievement. That generation, in turn, internalized the need for constant improvement, constant grading, and constant achievement, accepting an implied bargain that if they worked very hard, got good grades, got into good schools, and got a good degree, it would pay off in a good life and financial security. They were betrayed. The payoff never happened; many millennials graduated into the Great Recession and the worst economy since World War II. In response, millennials doubled down on the only path to success they were taught. They took on more debt, got more education, moved back in with their parents to cut expenses, and tried even harder.

Even after watching our parents get shut out, fall from, or simply struggle anxiously to maintain the American Dream, we didn't reject it. We tried to work harder, and better, more efficiently, with more credentials, to achieve it.

Once one has this framework in mind, it's startling how pervasive the "just try harder" message is and how deeply we've internalized it. It is at the center of the time management literature: Getting Things Done focuses almost entirely on individual efficiency. Later time management work has become more aware of the importance of pruning the to-do list and doing fewer things, but addresses that through techniques for individual prioritization. Cal Newport is more aware than most that constant busyness and multitasking interacts poorly with the human brain, and has taken a few tentative steps towards treating the problem as systemic rather than individual, but his focus is still primarily on individual choices. Even when tackling a problem that is clearly societal, such as the monetization of fear and outrage on social media, the solutions are all individual: recognize that those platforms are bad for you, make an individual determination that your attention is being exploited, and quit social media through your personal force of will. And this isn't just productivity systems. Most of public discussion of environmentalism in the United States is about personal energy consumption, your individual carbon footprint, household recycling, and whether you personally should eat meat. Discussions of monopoly and monopsony become debates over whether you personally should buy from Amazon. Concerns about personal privacy turn into advocacy for using an ad blocker or shaming people for using Google products. Articles about the growth of right-wing extremism become exhortations to take responsibility for the right-wing extremist in your life and argue them out of their beliefs over the dinner table. Every major systemic issue facing society becomes yet another personal obligation, another place we are failing as individuals, something else that requires trying harder, learning more, caring more, doing more. This advice is well-meaning (mostly; sometimes it is an intentional and cynical diversion), and can even be effective with specific problems. But it's also a trap. If you're feeling miserable, you just haven't found the right combination of time-block scheduling, kanban, and bullet journaling yet. If you're upset at corporate greed and the destruction of the environment, the change starts with you and your household. The solution is in your personal hands; you just have try a little harder, work a little harder, make better decisions, and spend money more ethically (generally by buying more expensive products). And therefore, when we're already burned out, every topic becomes another failure, increasing our already excessive guilt and anxiety. Believing that we're in control, even when we're not, does have psychological value. That's part of what makes it such a beguiling trap. While drafting this review, I listened to Ezra Klein's interview with Robert Sapolsky on poverty and stress, and one of the points he made is that, when mildly or moderately bad things happen, believing you have control is empowering. It lets you recast the setback as a larger disaster that you were able to prevent and avoid a sense of futility. But when something major goes wrong, believing you have control is actively harmful to your mental health. The tragedy is now also a personal failure, leading to guilt and internal recrimination on top of the effects of the tragedy itself. This is why often the most comforting thing we can say to someone else after a personal disaster is "there's nothing you could have done." Believing we can improve our lives if we just try a little harder does work, until it doesn't. And because it does work for smaller things, it's hard to abandon; in the short term, believing we're at the mercy of forces outside our control feels even worse. So we double down on self-improvement, giving ourselves even more things to attempt to do and thus burning out even more. Petersen is having none of this, and her anger is both satisfying and clarifying.

In writing that article, and this book, I haven't cured anyone's burnout, including my own. But one thing did become incredibly clear. This isn't a personal problem. It's a societal one and it will not be cured by productivity apps, or a bullet journal, or face mask skin treatments, or overnight fucking oats. We gravitate toward those personal cures because they seem tenable, and promise that our lives can be recentered, and regrounded, with just a bit more discipline, a new app, a better email organization strategy, or a new approach to meal planning. But these are all merely Band-Aids on an open wound. They might temporarily stop the bleeding, but when they fall off, and we fail at our new-found discipline, we just feel worse.

Structurally, Can't Even is half summaries of other books and essays put into this overall structure and half short profiles and quotes from millennials that illustrate her point. This is Petersen's typical journalistic style if you're familiar with her other work. It gains a lot from the voices of individuals, but it can also feel like argument from anecdote. If there's a epistemic flaw in this book, it's that Petersen defends her arguments more with examples than with scientific study. I've read enough of the other books she cites, many of which do go into the underlying studies and statistics, to know that her argument is well-grounded, but I think Can't Even works better as a roadmap and synthesis than as a primary source of convincing data. The other flaw that I'll mention is that although Petersen tries very hard to incorporate poorer and non-white millennials, I don't think the effort was successful, and I'm not sure it was possible within the structure of this book. She frequently makes a statement that's accurate and insightful for millennials from white, middle-class families, acknowledges that it doesn't entirely apply to, for example, racial minorities, and then moves on without truly reconciling those two perspectives. I think this is a deep structural problem: One's experience of American life is very different depending on race and class, and the phenomenon that Petersen is speaking to is to an extent specific to those social classes who had a more comfortable and relaxing life and are losing it. One way to see the story of the modern economy is that white people are becoming as precarious as everyone else already was, and are reacting by making the lives of non-white people yet more miserable. Petersen is accurately pointing to significant changes in relationships with employers, productivity, family, and the ideology of individualism, but experiencing that as a change is more applicable to white people than non-white people. That means there are, in a way, two books here: one about the slow collapse of the white middle class into constant burnout, and a different book about the much longer-standing burnout of being non-white in the United States and our systemic failure to address the causes of it. Petersen tries to gesture at the second book, but she's not the person to write it and those two books cannot comfortably live between the same covers. The gestures therefore feel awkward and forced, and while the discomfort itself serves some purpose, it lacks the insight that Petersen brings to the rest of the book. Those critiques aside, I found Can't Even immensely clarifying. It's the first book that explained to me in a way I understood what's so demoralizing and harmful about Instagram and its allure of cosplaying as a successful person. It helped me understand how productivity and individual political choices fit into a system that emphasizes individual action as an excuse to not address collective problems. And it also gave me a strange form of hope, because if something can't go on forever, it will, at some point, stop.

Millennials have been denigrated and mischaracterized, blamed for struggling in situations that set us up to fail. But if we have the endurance and aptitude and wherewithal to work ourselves this deeply into the ground, we also have the strength to fight. We have little savings and less stability. Our anger is barely contained. We're a pile of ashes smoldering, a bad memory of our best selves. Underestimate us at your peril: We have so little left to lose.

Nothing will change without individual people making different decisions and taking different actions than they are today. But we have gone much too far down the path of individual, atomized actions that may produce feelings of personal virtue but that are a path to ineffectiveness and burnout when faced with systemic problems. We need to make different choices, yes, but choices towards solidarity and movement politics rather than personal optimization. There is a backlash coming. If we let it ground itself in personal grievance, it could turn ugly and take a racist and nationalist direction. But that's not, by in large, what millennials have done, and that makes me optimistic. If we embrace the energy of that backlash and help shape it to be more inclusive, just, and fair, we can rediscover the effectiveness of collective solutions for collective problems. Rating: 8 out of 10

What did I learn about WebRTC in 2020. Because of remote life I needed new tools for every day life, around video and recording. I figured that HTML5 had most of ffmpeg and OBS features that I needed, and thought I could make equivalent tools with just web technology. Learning curve was relatively steep because many of examples on the web that I could find was stale and nonfunctional. Most sample code are still not ES2015. What year is this? This is 2020. Busy rewriting var to const, functions to await.

For a while I ve had a problem with my Thinkpad X1 Carbon Gen 1 [1] where storage pauses for 30 seconds, it s become more common recently and unfortunately everything seems to depend on storage (ideally a web browser playing a video from the Internet could keep doing so with no disk access, but that s not what happens).

echo 3 > /sys/block/sda/device/timeout

I ve put the above in /etc/rc.local which should make it only take 3 seconds to recover from storage problems instead of 30 seconds, the problem hasn t recurred since that change so I don t know if it works as desired. The Thinkpad is running BTRFS and no data is reported as being lost, so it s not a problem to use it at this time. The bottom of this post has an example of the errors I m getting. A friend said that his searches for such errors shows people claiming that it s a SATA controller or cable problem, which for a SATA device bolted directly to the motherboard means it s likely to be a motherboard problem (IE more expensive to fix than the $289 I paid for the laptop almost 3 years ago). A Lenovo forum discussion says that the X1 Carbon Gen1 uses an unusual connector that s not M.2 SATA and not mSATA [2]. This means that a replacement is going to be expensive, $100US for a replacement from eBay when a M.2 SATA or NVMe device would cost about $50AU from my local store. It also means that I can t use a replacement for anything else. If my laptop took a regular NVMe I d buy one and test it out, if it didn t solve the problem a spare NVMe device is always handy to have. But I m not interested in spending $100US for a part that may turn out to be useless. I bought the laptop expecting that there was nothing I could fix inside it. While it is theoretically possible to upgrade the CPU etc in a laptop most people find that the effort and expense makes it better to just replace the entire laptop. With the ultra light category of laptops the RAM is soldered on the motherboard and there are even less options for replacing things. So while it s annoying that Lenovo didn t use a standard interface for this device (they did so for other models in the range) it s not a situation that I hadn t expected. Now the question is what to do next. The Thinkpad has a SD socket and micro SD cards are getting large capacities nowadays. If it won t boot from a SD card I could boot from USB and then run from SD. So even if the internal SATA device fails entirely it should still be useful. I wonder if I could find a Thinkpad with a similar problem going cheap as I m pretty sure that a Windows user couldn t make any storage device usable the way I can with Linux. I ve looked at prices on auction sites but haven t seen a Thinkpad X1 Carbon going for anywhere near the price I got this one. The nearest I ve seen is around $350 for a laptop with significant damage. While I am a little unhappy at Lenovo s choice of interface, I ve definitely got a lot more than $289 of value out of this laptop. So I m not really complaining.

[315041.837612] ata1.00: status:   DRDY  
[315041.837613] ata1.00: failed command: WRITE FPDMA QUEUED
[315041.837616] ata1.00: cmd 61/20:48:28:1e:3e/00:00:00:00:00/40 tag 9 ncq dma 
16384 out
                         res 40/00:01:00:00:00/00:00:00:00:00/e0 Emask 0x4 
(timeout)                
[315041.837617] ata1.00: status:   DRDY  
[315041.837618] ata1.00: failed command: READ FPDMA QUEUED
[315041.837621] ata1.00: cmd 60/08:50:e0:26:84/00:00:00:00:00/40 tag 10 ncq 
dma 4096 in
                         res 40/00:01:00:00:00/00:00:00:00:00/e0 Emask 0x4 
(timeout)                
[315041.837622] ata1.00: status:   DRDY  
[315041.837625] ata1: hard resetting link
[315042.151781] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[315042.163368] ata1.00: ACPI cmd ef/02:00:00:00:00:a0 (SET FEATURES) 
succeeded
[315042.163370] ata1.00: ACPI cmd f5/00:00:00:00:00:a0 (SECURITY FREEZE LOCK) 
filtered out
[315042.163372] ata1.00: ACPI cmd ef/10:03:00:00:00:a0 (SET FEATURES) filtered 
out
[315042.183332] ata1.00: ACPI cmd ef/02:00:00:00:00:a0 (SET FEATURES) 
succeeded
[315042.183334] ata1.00: ACPI cmd f5/00:00:00:00:00:a0 (SECURITY FREEZE LOCK) 
filtered out
[315042.183336] ata1.00: ACPI cmd ef/10:03:00:00:00:a0 (SET FEATURES) filtered 
out
[315042.193332] ata1.00: configured for UDMA/133
[315042.193789] sd 0:0:0:0: [sda] tag#10 FAILED Result: hostbyte=DID_OK 
driverbyte=DRIVER_SENSE
[315042.193791] sd 0:0:0:0: [sda] tag#10 Sense Key : Illegal Request [current] 
[315042.193793] sd 0:0:0:0: [sda] tag#10 Add. Sense: Unaligned write command
[315042.193795] sd 0:0:0:0: [sda] tag#10 CDB: Read(10) 28 00 00 84 26 e0 00 00 
08 00
[315042.193797] print_req_error: I/O error, dev sda, sector 8660704
[315042.193810] ata1: EH complete

• [1] https://etbe.coker.com.au/2018/02/08/thinkpad-x1-carbon/
• [2] https://tinyurl.com/y7bsjarz

Review: Because Internet, by Gretchen McCulloch

Publisher:	Riverhead Books
Copyright:	2019
ISBN:	0-7352-1095-0
Format:	Kindle
Pages:	276

If you're familiar with linguistics as a field of scientific study (as opposed to the tool-based fields of grammar or writing advice), you'll be familiar with the dichotomy between written and spoken language. We may spend more time thinking about written language since it is central to most types of education and carries much of the intellectual and social weight of society. Linguists, however, see spoken language as more fundamental, since speech is wired into our brains and universal in human societies. Written language is a recent and somewhat artificial invention. One also learns from linguistics that spoken language does not follow many of the rules of written language that we painstakingly memorized in school. In casual speech, people split infinitives, speak in partial and run-on sentences, ignore nit-picking pronoun case rules, and rarely notice or care about the difference between less and fewer. Spoken language does have rules, but they're more subtle and nuanced than the grammar rules we learn in school. (I think the real fun of linguistics is separating the rules that native speakers follow effortlessly from the artificial rules used as education markers.) This is, in part, because nearly all spoken language is informal, whereas nearly all written language is formal. Enter the Internet, and enter this book. For the first time in human history we have both an explosion of informal writing and easy availability of that writing to linguists for study. Informal writing is not entirely new, of course. We've had personal letters for nearly as long as we've had writing, not to mention private notes, diaries, and other writing intended for tiny audiences. But consider who wrote private letters and, on top of that historical filter, whose private letters were preserved for linguistic research. Until relatively recently, only the upper classes were literate and had access to the infrastructure to write and send letters. Someone's letters or private notes were unlikely to be preserved unless they were someone famous and important, and thus often well-educated and more likely to take a more formal tone in writing. If you compare this to the Internet-driven blizzard of work and personal email, SMS conversations, chatrooms, and social media posts, the difference is obvious in both volume and level of informality. We're all on the Internet, we all read and write with a frequency that would be staggering to the average person from even fifty years ago, and while one may take a bit of additional care with a tricky email to one's manager, the SMS message to one's friend is as informal of a use of language as a conversation over coffee. Gretchen McCulloch is a professional linguist and Because Internet is about exactly this phenomenon: the new conventions of informal writing, how it has changed and evolved, and the new subtleties and shortcuts we've invented to make written communication easier. That goes beyond words and grammar to encompass punctuation, emoji and emoticons, memes and reaction gifs, and even the subtleties of timing, whitespace, and the construction of virtual places via our choices in how and where we write. This topic is my catnip, so it's not surprising I love this book. I've been heavily involved with online communities that communicate in writing since 1993 (making me, in McCulloch's classification, an Old Internet Person; each wave of introduction to the Internet has its own conventions that can be in conflict with later waves). I've now spent more than half my life carrying out most of my social activity and most of my closest friendships primarily in writing, so I found a lot of satisfaction in a linguistic study that takes that seriously rather than treating it as a curiosity. But, even better, I was amazed at how much I didn't know, in part because I am from a specific wave. I have a deep intuition for the Usenet conventions, but not as good of an understanding of the ones from AIM and LiveJournal one wave later (the Full Internet People). And I had a lot to learn about the conventions of the Instagram and Snapchat cluster (the Post Internet People, who have never known life without the Internet). One of the things that struck me while reading this book is how most of the language innovations that McCulloch describes are addressing the old complaint that written communication is inferior to face-to-face conversation because it lacks emotional nuance. My knee-jerk reply is that, no, written communication is full of emotional nuance and the complainer is just bad at reading it, but that's somewhat unfair. A better statement of the problem is that there is not a standardized language for emotional nuance in written communication, in part because it's so new in human history. Most humans are extremely good at reading facial expressions and body language for emotional cues, and those physical expressions are largely subconscious, reliable, and similar among different people (particularly within a culture; one can get in trouble with body language variations across cultures). This is not true of writing. With friends I've talked to over chat for twenty-five years, I can read volumes about their emotional state in a couple of short lines of text. But with strangers, despite decades of Internet communications, I will still misread cues and misinterpret simple intentions. The other standard response to this complaint is that it is possible to put extensive emotional nuance into formal writing. Just get better at writing! This is true, but unhelpful. There's a reason why we give book contracts to people who are very good at investing formal writing with emotional nuance. It's difficult, time-consuming, and requires a great deal of practice. That may be appropriate for formal, paid writing, but it won't do for informal writing, which by definition needs to be as effortless as possible. It's therefore unsurprising that once millions of people were using the Internet regularly for informal writing, they started adding new mechanisms, shortcuts, and conventions for emotional nuance. The standardization is growing, but conventions still vary widely between waves of Internet users. One of the most fascinating parts of this book for me was McCulloch's explanation of why periods (and, to a lesser extent, capital letters) in short chat messages are perceived by younger users as harsh or passive-aggressive. I still have the formal writing mindset of treating proper capitalization and punctuation as a point of pride, but McCulloch makes an excellent argument for letting go of my biases and understanding how and why language is changing. The realization I had while reading this is that many of the changes that look like sloppiness or laziness to someone trained in formal writing have the effect of giving language greater dynamic range. If one always uses periods uniformly, the period becomes meaningless except as a sentence boundary (which is redundant with newlines in most short informal chat messages). If one normally doesn't use it, and then suddenly starts using it, the period can carry semantic weight. It can convey a snippy tone of voice, a note of annoyance, or other subtle shades of meaning. I still use periods in most of my Slack messages because habits are hard to break, but I'm remembering to leave them off some of the time and paying more attention to what emotional weight they're carrying when present. Because Internet is therefore the rare book that meets the bar of changing my day-to-day behavior. "lol" is another excellent example that McCulloch spends some time on. It started life as LOL, an abbreviation for "laughing out loud," and that's still how it's stuck in my head. But, as McCulloch explains, it no longer means that to newer waves of Internet users. It now carries a far more complicated and nuanced meaning that has very little to do with physical laughter and that doesn't easily translate to a single word or sentence. I went from being mildly irritated by and mildly superior towards the ubiquitous "lol" to realizing that it's a fascinating new word that carries primarily emotional nuance and that I don't understand well enough to read or use properly (yet). One more example of the type of analysis McCulloch brings to this book: emoji. The tendency when talking about emoji is to treat them as rebuses (pictures that stand in for a word, or at least a specific concept). They are sometimes used that way, but McCulloch argues that they more often function in the same role that gestures play in informal speech, including the gestures that have no simple name and no independent meaning outside of the context of the words being said at the same time. This seems obvious in retrospect, but before reading Because Internet I had never thought about what a gesture is, what function it plays in speech, and how that could be translated into informal written communication. If you're as interested in this area as I am, this is great stuff. I'd seen several mentions of this book go past on Twitter and kept holding off because I had lots of things to read and was worried it would only cover the superficial things I already knew as a long-time Internet user who has listened to a few lectures on linguistics. That was not the case at all. I learned so much from this book and had a delightful time reading it. If you're also interested in these topics, recommended. Rating: 9 out of 10

There is a whole science to manipulating our attention. And because there is a lot of money to be made by doing this well, it means we all encounter attempts to manipulate what we pay attention to each day. What is this, and how is it harmful? This post will be the first on a series on the topic. Why is attention so important? When people use Facebook, they use it for free. Facebook generally doesn t even try to sell them anything, yet has billions in revenues. What, then, is Facebook s product? Well, really, it s you. Or, more specifically, your attention. Facebook sells your attention to advertisers. Everything they do is in service to that. They want you to spend more time on the site so they can show you more ads. (I should say here that I m using Facebook as an example, but this applies to other social media companies too.) Seeking to maximize attention So if your attention is so important to their profit, it follows naturally that they would seek ways to get people to spend more time on their site. And they do. They track all sorts of metrics, including engagement (if you click like , comment, share, or otherwise interact with content). They know which sorts of things are likely to capture your (and I mean you in specific!) attention and show you that. Your neighbor may have different interests and Facebook judges different things are likely to capture their attention. Manipulating your attention Attention turning into money isn t unique for social media. In fact, in the article If It Bleeds, It Leads: Understanding Fear-Based Media, Psychology Today writes:

In previous decades, the journalistic mission was to report the news as it actually happened, with fairness, balance, and integrity. However, capitalistic motives associated with journalism have forced much of today s television news to look to the spectacular, the stirring, and the controversial as news stories. It s no longer a race to break the story first or get the facts right. Instead, it s to acquire good ratings in order to get advertisers, so that profits soar. News programming uses a hierarchy of if it bleeds, it leads. Fear-based news programming has two aims. The first is to grab the viewer s attention. In the news media, this is called the teaser. The second aim is to persuade the viewer that the solution for reducing the identified fear will be in the news story. If a teaser asks, What s in your tap water that YOU need to know about? a viewer will likely tune in to get the up-to-date information to ensure safety.

You ve probably seen fear-based messages a lot on Facebook. They will highlight messages to liberals about being afraid of what Trump is doing, and to conservatives about being afraid of what Biden is doing. They may or may not even intentionally be doing this; it is their algorithm predicts that those would maximize time and engagement for certain people, so that s what they see. Fear leads to controversy It s not just fear, though. Social media also loves controversy. There s nothing that makes people really want to stay on Facebook like anger. See something controversial and you ll see hundreds or thousands of people are there arguing about it and in the process, giving Facebook their attention. A quick Internet search will show you numerous articles on how marketing companes can leverage controvery to get attention and engagement with their campaigns. Consequences of maximizing fear and controversy What does it mean to society at large and to you personally that large companies make a lot of money by maximizing fear and controversy? The most obvious way is it leads to less common ground. If the posts and reactions that show common ground are never seen because they don t drive engagement, it poisons the well; left and right hate each other with ever more vigor a profitable outcome to Facebook, but a poisonous one to all of us. I have had several friendships lost because I a liberal in agreement with these friends on political matters still talk to Trump voters. On the other side, we ve seen people storm the Michigan statehouse with weapons. How did that level of disagreement and even fear behind it get so firmly embedded in our society? Surely the fact that social media shows us things designed to stimulate fear and anger must play a role. What does it do to our ability to have empathy for, and understand, others? The Facebook groups I ve been in for like-minded people have largely been flooded with memes calling the President rump and other things clearly designed to make people angry or fearful. It s a worthless experience, and not just that, but it s a harmful experience. When our major media TV and social networks all are optimizing for fear, anger, and controvesry, we have a society beholden to fear, anger, and controvesy. In my next installment, I m going to talk about what to do about this, including the decentralized social networks of the Fediverse that are specifically designed to put you back in charge of your attention. Update 2020-12-16: There are two followup articles for this: how to join the Fediverse and non-creepy technology purchasing and gifting guides. The latter references the FSF s page on software manipulation towards addiction, which is particularly relevant to this topic.

Issues with Farm Laws While I have written about the farm laws a bit sometime back. The issue is still in the nation s eye and that is due to the policies which have been done. I have been reading up on it quite a bit and also have been seeing what has been happening in here and now. The problems are with the three bills themselves which I have shared as below Click to access farmers-produce-trade-and-commerce-promotion-and-facilation-bill.pdf Click to access farmers-empowerment-and-protection-bill.pdf Click to access essential-commodities-bill-2020.pdf Biggest issue with the laws While there are many issues with the laws themselves but for me the biggest issue is that the fundamental right of the farmer to get justice via civil courts has been railroaded. From the laws itself. Standard disclaimer not a lawyer, please consult one for any issues per-se.

Farmers-produce-trade-and-commerce (promotion and facilitation-bill) 2020 Page 4 Chapter 3 Section 8 (1)8. (1) In case of any dispute arising out of a transaction between the farmer and a trader under section 4, the parties may seek a mutually acceptable solution through conciliation by filing an application to the Sub-Divisional Magistrate who shall refer such dispute to a Conciliation Board to be appointed by him for facilitating the binding settlement of the dispute. (2) Every Board of Conciliation appointed by the Sub-Divisional Magistrate under sub-section (1), shall consist of a chairperson and such members not less than two and not more than four, as the Sub-Divisional Magistrate may deem fit.10 (5) If the parties to the transaction under sub-section (1) are unable to resolve the dispute within thirty days in the manner set out under this section, they may approach the Sub-Divisional Magistrate concerned who shall be the Sub-Divisional Authority for settlement of such dispute. (8) Any party aggrieved by the order of the Sub-Divisional Authority may prefer an appeal before the Appellate Authority (Collector or Additional Collector nominated by the Collector) within thirty days of such order who shall dispose of the appeal within thirty days from the date of filing of such appeal. 10. (1) Any person aggrieved by an order under section 9 may, prefer an appeal within sixty days from the date of such order, to an officer not below the rank of Joint Secretary to the Government of India to be nominated by the Central Government for this purpose: Page 6 of the bill. 13. No suit, prosecution or other legal proceedings shall lie against the Central Government or the State Government, or any officer of the Central Government or the State Government or any other person in respect of anything which is in good faith done or intended to be done under this Act or of any rules or orders made thereunder. Page 7 of the bill, 15. No civil court shall have jurisdiction to entertain any suit or proceedings in respect of any matter, the cognizance of which can be taken and disposed of by any authority empowered by or under this Act or the rules made thereunder. Now the same laws have been reiterated for the farmers (Empowerment and Protection) Agreement on Price Assurance and Farm Services Bill, 2020. The problem is that too much power is being put into the hands of the executive. All the three, whether it is SDM (Sub-Divisional Magistrate) , the Appellate Authority or the Government Secretary directly are subservient to the whims and fancies of the Central Govt. They after all get their salaries from the Govt. itself. So there will be no independent oversight to any injustices done to the farmer. The third bill i.e. the Essential Commodities Bill, 2020 does away with stock limits on traders and big players like Adani and Ambani. This means that both these players can take and keep produce at their end thereby forcing consumers like you and me who at the retail end would have to pay higher prices for fruits and vegetables while from the producer they will take at the lowest price possible. While I have shared is just one of the points. That is the reason why even the Supreme Court bar association which almost never takes part in politics has been forced to take sides with the farmers. In many ways, one is forced to remember the Emergency Update 11/12/20 Came across this article on the wire which tells how everybody s rights, not just the farmer s rights are being shod over. I think it depicts correctly the signs of time to come. While arguing on SM, also came to know about Article 300 (1), thanks to Sachin Kumar which shows multiple instances where Government was sued because somebody was working in official capacity and did mistakes, malafide or otherwise and it was the state who was made to pay. FWIW, today farmers from Maharashtra, my state arrived at Delhi border where they were also kept at bay. I did come across an infographic which shows how the various states have fared. Most tellingly, is the state of Bihar. It was in 2006 (one of the most backward states) where APMC was taken off. While others have tried to paint a flattering picture of Bihar, they have failed to share that in the interim 15 odd years, there hasn t been any sort of infrastructure created for farmers which is the reason it is still the lowest earner. These are the last available figures we have about the farmer s income. From 2014 to 2020 there hasn t been any update.

Situation Assessment Survey of Agricultural Household 2013 Copyright GOI,

This concludes just one portion of the bill. I will take other parts of the bill. I may dwell on some other parts as and when I have the time. A cartoon which depicts the current issue

I stand with farmers Copyright Sanitary Panels

Too much democracy Amitabh Kant Yesterday, the Niti Aayog chief Amitabh Kant remarked that we are too much of a democracy at an event called for Atmanirbhar Bharat which is basically a coinage for import substitution. Whether this is desirable or not I have argued and if needed will re-argue the same later as well. What is and was interesting were the gentleman s context, the media reactions and our overall Democracy Index which has been going downhill for quite some years. Now the gentleman who is the Niti Aayog chief and who is supposed to have the ear of the Prime Minister had opined it in an event organized by Swarajya Magazine (a far-right magazine) known to be Islamophobic and all things undemocratic. It has been a target of defundthehate campaign and with good reason. But that s a different story altogether. His full statement was as below

Tough reforms are very difficult in the Indian context, as we are too much of a democracy but the government has shown courage and determination in pushing such reforms across sectors, including mining, coal, labour and agriculture. Niti Aayog chief. The upper quotation remarks and the statement has been from the article in Indian Express which I have linked to. I have archived it as a pdf just in case the link goes dead. Yesterday, after the statement became viraled, tweets of media houses which shared the tweet suddenly become unavailable. Seems too much democracy, became too little democracy all of a sudden. I think Mr. Amitabh Kant didn t visualize as the opposition as well as most people who are on Twitter to share their opinion on the same. Few examples

Too much Democracy copyright Satish Acharya

Too much democracy Illustration and Copyright Alok

Sterlite protest 13 dead, 100 injured Copyright Business Standard too much democracy

Erosion of Democracy V-dem institute Copyright The Hindu Web Team

The last one requires a bit more information. This comes from V-Dem Institute which is an independent research institute based out of Department of Political Science, University of Gothenburg, Sweden. I am gonna leave the methods they use for another day as the blog post itself has become rather big/large. Apart from that is the Economists own Democracy Index -2019 Click to access democracy-index-2019.pdf Now for many people, both the V-Dem report and the Economist Index are some sort of attack against India. Doesn t matter that in V-Dem 200+ countries have been taken a variety of indicators and data or the Economist which has data from 150+- countries. Somehow India is supposed to be bigger than all these countries, they do think that other countries data specifically our neighbor China or any other neighbor, those are all accurate. How the dissonance is, has to be gauged from statements of various people. Update 11/12/20 Sadly, the newest V-Dem report marks India as getting into authoritarianism. Gag on Press and Media owners I had shared about the gag on the press especially with respect to western media or reports or anything. This news made its way to straitstimes which normally covers a wide-range of stories covering East Asia vis-a-vis India/South-East Asia. What has also been a big worry that most of the media has been in the hands of a few people. Caravan ran a story on the same in 2016, it has been four years, god only knows what the current situation might be. Any wonder that there is dearth of investigative journalism in India.

India media ownership 2016 Copyright Caravan

Incidentally, a reporter called Akarshan Uppal, who is a reporter on a channel called IBN24 had showecased just few days back how Adani has got land which was shot down for land change use in 2017 to 2020 around 100 acres. There seem to be very less details as to how the land was acquired, whose land it was etc. etc. The reporter was supposedly following a story on drugs on which he was attacked and is now lying in hospital.

Akarshan Uppal Reporter, IBN24 Copyright IBN24

While it would take a whole article/blog post to talk about either Adani or Ambani, in the recent case, the land that has been taken over by Adani is 100 acres and there are private rail lines. And all of this was secret till few days back. The place where these massive godowns/silos have been made are Panipat s Jondhan Kalan and Naultha villages in Haryana. This is Adani AgiLogistics. Almost 7 odd companies have registered and come up in the last couple of years. As can be seen, almost all have come up within the last 2-3 years. Seems to be a lot of coincidence, isn t it?

Personal Anecdote on Data Collection and child marriages in India.

Around 1995 -96 when Internet had started to become a thing in India, there had been quite a few non-profits which were working on various issues. One of those which I initially came in contact with and which I found to be a bit absurd was non-profit which was working in the field of women against Violence. Now it is and was not the concept or the idea which was absurd to me, it was what these women were doing. Instead of the traditional ways in which you counsel women and try and figure out issues, these women were collecting data points from newspapers and magazines. This was way way before data science became a thing in India. They had their own structure where a story about violence against women which would be above the fold would be 5 points, the one below 2.5 points, in inner pages, it would be less and less. Patriarchy at that time was so strong, even today is but at that time it was such, that it felt a waste of time. I did consult them but never said that but did privately feel the above. In hindsight, they were doing the right thing and yet even today crimes against women goes unreported and is suppressed by both State and Central Governments as well as NCRB (National Crime Records Bureau). Interestingly, just few days back, the case against M.J. Akbar by Priya Ramani had taken a back seat and the defamation case by M.J. Akbar was taken forward. Even then, Priya Ramani s counsel s arguments were such that the court wound up in half an hour when they were expecting to do a whole day hearing. The next hearing would be happening today which I will look at in few hours from now. Why Priya Ramani was singled out rather than other tweets may probably be because she is an NRI and most NRI s usually do not want to be part of the bureaucratic Indian court system. This is also the reason that most companies from outside India especially those who are into startups prefer to change ownership, IPR etc. to their own or any country outside India which does make a loss to the exchequer. But this again is a story for another day. At the end, while I did not want to end on a negative note, it seems in many ways status-quo remains. For e.g. 2 years back, a BJP candidate (part of the ruling dispensation) had made a controversy saying that if they win the police won t interfere in child marriages. This is and was in Rajasthan where they have been trying to eradicate it forever. Till date, neither the NHRC (National Human Rights Commission) nor NCW (National Commission for Women) has taken cognizance of the statement. This is our state of democracy.

A few years ago, the advertising industry introduced the ads.txt project in order to defend against widespread domain spoofing vulnerabilities in programmatic advertising. I decided to use this technology to opt out of having ads sold for my domains, at least through ad exchanges which perform this check, by hosting a text file containing this:

contact=ads@fmarier.org

at the following locations:

• https://feeding.cloud.geek.nz/ads.txt
• https://fmarier.org/ads.txt

(In order to get this to work on my blog, running Ikiwiki on Branchable, I had to disable the txt plugin in order to get ads.txt to be served as a plain text file instead of being automatically rendered as HTML.)

Specification The key parts of the specification for our purposes are:

[3.1] If the server response indicates the resource does not exist (HTTP Status Code 404), the advertising system can assume no declarations exist and that no advertising system is unauthorized to buy and sell ads on the website. [3.2.1] Some publishers may choose to not authorize any advertising system by publishing an empty ads.txt file, indicating that no advertising system is authorized to buy and sell ads on the website. So that consuming systems properly read and interpret the empty file (differentiating between web servers returning error pages for the /ads.txt URL), at least one properly formatted line must be included which adheres to the format specification described above.

As you can see, the specification sadly ignores RFC8615 and requires that the ads.txt file be present directly in the root of your web server, like the venerable robots.txt file, but unlike the newer security.txt standard. If you don't want to provide an email address in your ads.txt file, the specification recommends using the following line verbatim:

placeholder.example.com, placeholder, DIRECT, placeholder

Validation A number of online validators exist, but I used the following to double-check my setup:

• https://adstxt.guru/validator/url/?url=https%3A%2F%2Ffeeding.cloud.geek.nz%2Fads.txt
• https://www.adstxtgenerator.com/validator/results/?site=feeding.cloud.geek.nz

TL;DR: Some mildly tricky tricks allow you to select subgoals by name, like in Isabelle. This leads to more readable proofs, and proofs that are more robust against changes.

The need for case names Consider the following Coq proof, right out of the respected Software Foundations book, Chapter Inductively Defined Propositions (and slightly simplified):

Lemma star_app: forall T (s1 s2 : list T) (re : reg_exp T),
  s1 =~ Star re ->
  s2 =~ Star re ->
  s1 ++ s2 =~ Star re.
Proof.
  intros T s1 s2 re H1.
  remember (Star re) as re'.
   generalize dependent s2.
  induction H1.
  - (* MEmpty *) discriminate.
  - (* MChar *) discriminate.
  - (* MApp *) discriminate.
  - (* MUnionL *) discriminate.
  - (* MUnionR *) discriminate.
  - (* MStar0 *)
    injection Heqre' as Heqre''. intros s H. apply H.
  - (* MStarApp *)
    injection Heqre' as Heqre''.
    intros s3 H1. rewrite <- app_assoc.
    apply MStarApp.
    + apply H1_.
    + apply IHexp_match2.
      * rewrite Heqre''. reflexivity.
      * apply H1.
Qed.

This is a very typical example for a proof by induction over an inductively defined relation (in this case, the relations s =~ re which indicates that the string s matches the regular expression re): After some preparation, the invocation of a tactic like induction (could also be inversion or others) creates a number of new goals, one of for each rule of the relation. Then the modern(!) Coq user uses proof bullets to focus the goals, one after another, in order, to solve them. We can see from this example that proof bullets (which were a big improvement) are not quite satisfactory, and the author of this proof felt the need to annotate each subproof with the case it corresponds to. But these are mere comments! This is unsatisfactory, as there is nothing keeping them correct as my proof changes. And anyone who came to Coq from Isabelle is just flabbergasted that the state of art in Coq is still at that level So, wouldn t it be great if

• I could focus goals by their name (e.g. MEmpty)
• Focus them in any order (so that refactoring of my data types do not break my proofs)
• This even worked when simultaneously inducting over or inverting multiple relations?

Simulating named goals in Coq Well, with a few tricks we can! It just takes three simple steps:

1. I load a small file full of hacks (explained below):

   Require Import NamedCases.

2. This allows me to prefix the type of the constructors of the inductive relations (i.e. the rules of the relation) with case <name>, , as follows:

   Reserved Notation "s =~ re" (at level 80).
   Inductive exp_match  T  : list T -> reg_exp T -> Prop :=
       MEmpty:
       case empty,
       [] =~ EmptyStr
       MChar:
       case char,
       forall x,
       [x] =~ (Char x)
       MApp:
       case app,
       forall s1 re1 s2 re2
       (H1 : s1 =~ re1)
       (H2 : s2 =~ re2),
       (s1 ++ s2) =~ (App re1 re2)
       MUnionL:
       case union_l,
       forall s1 re1 re2
       (H1 : s1 =~ re1),
       s1 =~ (Union re1 re2)
       MUnionR:
       case union_r,
       forall re1 s2 re2
       (H2 : s2 =~ re2),
       s2 =~ (Union re1 re2)
       MStar0:
       case star0,
       forall re,
       [] =~ (Star re)
       MStarApp:
       case star_app,
       forall s1 s2 re
       (H1 : s1 =~ re)
       (H2 : s2 =~ (Star re)),
       (s1 ++ s2) =~ (Star re)
     where "s =~ re" := (exp_match s re).

   This is the definition from the Software Foundations book; only the case <name>, lines are added.
3. Now I can change the proof: After the induction tactic I use ; name_cases to name the cases, and then I can use the relatively new named goal focusing feature ([name]: , which I suggested two years ago, and was implemented by Th o Zimmermann) to focus a specific case:

   Lemma star_app2: forall T (s1 s2 : list T) (re : reg_exp T),
     s1 =~ Star re ->
     s2 =~ Star re ->
     s1 ++ s2 =~ Star re.
   Proof.
     intros T s1 s2 re H1.
     remember (Star re) as re'.
      generalize dependent s2.
     induction H1; name_cases.
     [empty]:  
        discriminate.
      
     [char]:  
        discriminate.
      
     [app]:  
        discriminate.
      
     [unionL]:  
        discriminate.
      
     [unionR]:  
       discriminate.
      
     [star0]:  
       injection Heqre' as Heqre''. intros s H. apply H.
      
     [starApp]:  
       injection Heqre' as Heqre''.
       intros s3 H1. rewrite <- app_assoc.
       apply MStarApp; clear_names.
       + apply H1_.
       + apply IHexp_match2.
         * rewrite Heqre''. reflexivity.
         * apply H1.
       
   Qed.

   The comments have turned into actual names!

Goal focusing is now just much more reliable. For example, let me try to discharge the boring cases by using ; try discriminate after the induction. This means there are suddenly only two goals left. With bullets, the first bullet would now suddenly focus a different goal, and my proof script would likely fail somewhere. With named goals, I get a helpful error message No such goal: empty and I can t even enter the first subproof. So let me delete these goals and, just to show off, focus the remaining goals in the wrong order:

Lemma star_app: forall T (s1 s2 : list T) (re : reg_exp T),
  s1 =~ Star re ->
  s2 =~ Star re ->
  s1 ++ s2 =~ Star re.
Proof.
  intros T s1 s2 re H1.
  remember (Star re) as re'.
   generalize dependent s2.
  induction H1; name_cases; try discriminate.
  [starApp]:  
    injection Heqre' as Heqre''.
    intros s3 H1. rewrite <- app_assoc.
    apply MStarApp; clear_names.
    + apply H1_.
    + apply IHexp_match2.
      * rewrite Heqre''. reflexivity.
      * apply H1.
    
  [star0]:  
    injection Heqre' as Heqre''. intros s H. apply H.
   
Qed.

I did not explain the clear_names tactic yet that I had to use after apply MStarApp. What I am showing here is a hack, and it shows these case names are extra hypothesis (of a trivially true type CaseName), and this tactic gets rid of it.

Even better than Isabelle The above even works when there are multiple inductions/inversions involved. To give an example, let s introduce another inductive predicate, and begin proving a lemma that one can do by induction on one and inversion on the other predicate.

Inductive Palindrome  T  : list T -> Prop :=
  EmptyPalin:
  case emptyP,
  Palindrome []
  SingletonPalin:
  case singletonP,
  forall x,
  Palindrome [x]
  GrowPalin:
  case growP,
  forall l x,
  Palindrome l -> Palindrome ([x] ++ l ++ [x])
.
Lemma palindrome_star_of_two:
  forall T (s : list T) x y,
  Palindrome s -> s =~ Star (App (Char x) (Char y)) ->
  s = [] \/ x = y.
Proof.
  intros T s x y HPalin HRe.
  induction HPalin; inversion HRe; name_cases.

At this point we have four open goals. Can you tell why there are four, and which ones they are? Luckily, Show Existentials will tell us their names (and Coq 8.13 will just show them in the proof state window). These names not only help us focus the goal in a reliable way; they even assist us in understanding what goal we have to deal with. For example, the last goal arose from the GrowP rule and the StarApp rule. Neat!

  [emptyP_star0]:  
    left. reflexivity.
   
  [emptyP_starApp]:  
     left. reflexivity.
   
  [singletonP_starApp]:  
    inversion HRe; clear HRe.
    inversion H5; clear H5.
    inversion H10; clear H10.
    inversion H11; clear H11.
    subst.
    inversion H3.
   
 [growP_starApp]:  
    admit. (* I did not complete the proof *)
  

Under the hood (the hack) The above actually works, but it works with a hack, and I wish that Coq s induction tactic would just do the right thing here. I hope eventually this will be the case, and this section will just be curiosity but until, it may actually be helpful.

1. The trick¹ is to leave markers in the proof state. We use a dedicate type CaseName, and a hypothesis name : CaseName indicates that this goal arose from the that case.

   Inductive CaseName := CaseNameI.

   With this, we can write StarApp: forall (starApp : CaseName), to name this case.
2. For a little bit of polish, we can introduce notation for that:

   Notation "'case' x , t" := (forall  x : CaseName , t) (at level 200).

   This allows us to use the syntax you saw above.
3. To easily discharge these CaseName assumptions when using the introduction forms, I wrote these tactics:

   Ltac clear_names := try exact CaseNameI.
   Ltac named_constructor  := constructor; [ exact CaseNameI   idtac .. ].
   Ltac named_econstructor := econstructor; [ exact CaseNameI   idtac .. ].

4. Finally I wrote the tactic name_cases. What it does is relatively simple:

   For each subgoal: Take all hypotheses of type CaseName, concatenate their names, change the name of the current goal to that, and clear these hypotheses

   Unfortunately, my grasp of Coq tactic programming is rather rudimentary, and the only way I manged to solve this is a horrible monster of Ltac2, including hand-rolled imperative implementations of string concatenation and a little nested Ltac1 tactic within. You have been warned:

   From Ltac2 Require Import Ltac2.
   From Ltac2 Require Option.
   Set Default Proof Mode "Classic".
   Ltac name_cases := ltac2:(
     (* Horribly manual string manipulations. Does this mean I should
        go to the Ocaml level?
     *)
     let strcpy s1 s2 o :=
       let rec go := fun n =>
         match Int.lt n (String.length s2) with
           true => String.set s1 (Int.add o n) (String.get s2 n); go (Int.add n 1)
           false => ()
         end
       in go 0
     in
     let concat := fun s1 s2 =>
       let l := Int.add (Int.add (String.length s1) (String.length s2)) 1 in
       let s := String.make l (Char.of_int 95) in
       strcpy s s1 0;
       strcpy s s2 (Int.add (String.length s1) 1);
       s
     in
     Control.enter (let rec go () :=
       lazy_match! goal with
         [ h1 : CaseName, h2 : CaseName  - _ ] =>
         (* Multiple case names? Combine them (and recurse) *)
         let h := concat (Ident.to_string h1) (Ident.to_string h2) in
         Std.clear [h1; h2];
         let h := Option.get (Ident.of_string h) in
         assert ($h := CaseNameI);
         go ()
         [ _ : CaseName  - _ ] =>
         (* A single case name? Set current goal name accordingly. *)
         ltac1:(
           (* How to do this in ltac2? *)
           lazymatch goal with
             [ H : CaseName  - _ ] => refine ?[H]; clear H
           end
         )
         [  - _ ] =>
         Control.backtrack_tactic_failure "Did not find any CaseName hypotheses"
       end
     in go)
   ).

   If someone feels like re-implementing this in a way that is more presentable, I ll be happy to update this code (and learn something along the way

That s all it takes!

Conclusion With some mild hackery, we can get named cases in Coq, which I find indispensable in managing larger, evolving proofs. I wonder why this style is not more common, and available out of the box, but maybe it is just not known well enough? Maybe this blog post can help a bit here.

---

1. I have vague recollections of seeing at least parts of this trick on some blog some years ago, and I do not claim originality here.

The Linux Foundation has published its annual report today. LWN calls it glossy and yeah, boy, it is shiny. So shiny that people that work in the publishing industry immediately see this has been produced with the Adobe toolchain which - unfortunately - is one of the big suites of software not yet available for Linux. Checking the PDF file metadata reveals the keywords "open source, open standards, open hardware, open data". That is what the Linux Foundation is about. Good stuff. The PDF producer meta data for the annual report PDF has been set to "Linux kernel 0.12.1 for Workgroups" and the PDF creator meta data element to "Sharp Zaurus XR-5000 (Maemo5) Edition". Somebody thought to better hide the real data and had some tongue-in-cheek ideas. Kudos. But nicer would have been to use Open Source software to produce the report, not? Running strings 2020-Linux-Foundation-Annual-Report_113020.pdf grep Adobe wc -l gives us 1229 lines and confirms the suspicion of the toolchain. A stale /Title (Annual Report 2020) /Producer (macOS Version 10.15.7 \(Build 19H15\) Quartz PDFContext) has been forgotten in the document to tell us about the platform. So, ladies and gentlemen, the Linux Foundation 2020 annual report has been produced on a Mac. Running Adobe Creative Cloud on MacOS Catalina 10.15.7. Which is proprietary software. Its kernel (and some userland pieces) are based on BSD. Not Linux.

---

The image on the front page also struck me as a bit odd ... using a ballpoint pen on the laptop screen? Unbranded laptop. Unbranded cup in the foreground. Kid in the background not paying attention to his tablet. All of that cries stock image so loud it hurts. Google currently finds ~560 uses of the picture and any editorial use nicely tells us that it is Dragana Gordic / Shutterstock. The image is "Smiling mom working at home with her child on the sofa while writing an email. Young woman working from home, while in quarantine isolation during the Covid-19 health crisis". See the Daily Mail for a wonderful example of the working mum in context. I hope, if her laptop had been powered on, it would have run Linux. I mean, what else would still run on an old white MacBook with an Intel "Core 2 Duo" processor from 2008? Continue reading "No dog food today - the Linux Foundation annual report"

In previous blog posts I talked about QEMU s qcow2 file format and how to make it faster. This post gives an overview of how the data is structured inside the image and how that affects performance, and this presentation at KVM Forum 2017 goes further into the topic. This time I will talk about a new extension to the qcow2 format that seeks to improve its performance and reduce its memory requirements. Let s start by describing the problem. Limitations of qcow2 One of the most important parameters when creating a new qcow2 image is the cluster size. Much like a filesystem s block size, the qcow2 cluster size indicates the minimum unit of allocation. One difference however is that while filesystems tend to use small blocks (4 KB is a common size in ext4, ntfs or hfs+) the standard qcow2 cluster size is 64 KB. This adds some overhead because QEMU always needs to write complete clusters so it often ends up doing copy-on-write and writing to the qcow2 image more data than what the virtual machine requested. This gets worse if the image has a backing file because then QEMU needs to copy data from there, so a write request not only becomes larger but it also involves additional read requests from the backing file(s). Because of that qcow2 images with larger cluster sizes tend to:

• grow faster, wasting more disk space and duplicating data.
• increase the amount of necessary I/O during cluster allocation,
  reducing the allocation performance.

Unfortunately, reducing the cluster size is in general not an option because it also has an impact on the amount of metadata used internally by qcow2 (reference counts, guest-to-host cluster mapping). Decreasing the cluster size increases the number of clusters and the amount of necessary metadata. This has direct negative impact on I/O performance, which can be mitigated by caching it in RAM, therefore increasing the memory requirements (the aforementioned post covers this in more detail). Subcluster allocation The problems described in the previous section are well-known consequences of the design of the qcow2 format and they have been discussed over the years. I have been working on a way to improve the situation and the work is now finished and available in QEMU 5.2 as a new extension to the qcow2 format called extended L2 entries. The so-called L2 tables are used to map guest addresses to data clusters. With extended L2 entries we can store more information about the status of each data cluster, and this allows us to have allocation at the subcluster level. The basic idea is that data clusters are now divided into 32 subclusters of the same size, and each one of them can be allocated separately. This allows combining the benefits of larger cluster sizes (less metadata and RAM requirements) with the benefits of smaller units of allocation (less copy-on-write, smaller images). If the subcluster size matches the block size of the filesystem used inside the virtual machine then we can eliminate the need for copy-on-write entirely. So with subcluster allocation we get:

• Sixteen times less metadata per unit of allocation, greatly reducing the amount of necessary L2 cache.
• Much faster I/O during allocating when the image has a backing file, up to 10-15 times more I/O operations per second for the same cluster size in my tests (see chart below).
• Smaller images and less duplication of data.

This figure shows the average number of I/O operations per second that I get with 4KB random write requests to an empty 40GB image with a fully populated backing file. Things to take into account:

• The performance improvements described earlier happen during allocation. Writing to already allocated (sub)clusters won t be any faster.
• If the image does not have a backing file chances are that the allocation performance is equally fast, with or without extended L2 entries. This depends on the filesystem, so it should be tested before enabling this feature (but note that the other benefits mentioned above still apply).
• Images with extended L2 entries are sparse, that is, they have holes and because of that their apparent size will be larger than the actual disk usage.
• It is not recommended to enable this feature in compressed images, as compressed clusters cannot take advantage of any of the benefits.
• Images with extended L2 entries cannot be read with older versions of QEMU.

How to use this? Extended L2 entries are available starting from QEMU 5.2. Due to the nature of the changes it is unlikely that this feature will be backported to an earlier version of QEMU. In order to test this you simply need to create an image with extended_l2=on, and you also probably want to use a larger cluster size (the default is 64 KB, remember that every cluster has 32 subclusters). Here is an example:

$ qemu-img create -f qcow2 -o extended_l2=on,cluster_size=128k img.qcow2 1T

And that s all you need to do. Once the image is created all allocations will happen at the subcluster level. More information This work was presented at the 2020 edition of the KVM Forum. Here is the video recording of the presentation, where I cover all this in more detail: You can also find the slides here. Acknowledgments This work has been possible thanks to Outscale, who have been sponsoring Igalia and my work in QEMU. And thanks of course to the rest of the QEMU development team for their feedback and help with this!

I joined the Debian project in late 1994, well before the first stable release was issued, and have been involved in various ways continuously ever since. Over the years, I adopted a number of packages that are, or at least were at one time, fundamental to the distribution. But, not surprisingly, my interests have shifted over time. In the more than quarter century I've contributed to Debian, I've adopted existing packages that needed attention, packaged new software I wanted to use that wasn't yet in Debian, offered packages up for others to adopt, and even sometimes requested the removal of packages that became obsolete or replaced by something better. That all felt completely healthy. But over the last couple weeks, I realized I'm still "responsible" for some packages I'd had for a very long time, that generally work well but over time have accumulated bugs in functionality I just don't use, and frankly haven't been able to find the motivation to chase down. As one example, I just noticed that I first uploaded the gzip package 25 years ago today, on 2 December 1995. And while the package works fine for me and most other folks, there are 30 outstanding bugs and 3 forwarded bugs that I just can't muster up any energy to address. So, I just added gzip to a short list of packages I've offered up for adoption recently. I'm pleased that tar already has a new maintainer, and hope that both sudo and gzip will get more attention soon. It's not that I'm less interested in Debian. I've just been busy recently packaging up more software I use or want to use in designing high power model rockets and the solid propellant motors I fly in them, and would rather spend the time I have available for Debian maintaining those packages and all their various build dependencies than continuing to be responsible for core packages in the distribution that "work fine for me" but could use attention. I'm writing about this partly to mark the passing of more than a quarter century as a package maintainer for Debian, partly to encourage other Debian package maintainers with the right skills and motivation to consider adopting some of the packages I'm giving up, and finally to encourage other long-time participants in Debian to spend a little time evaluating their own package lists in a similar way.