Search Results: "cas"

19 April 2020

Russ Allbery: Review: Beyond the Galaxy

Review: Beyond the Galaxy, by Ethan Siegel
Publisher: World Scientific
Copyright: 2016
ISBN: 981-4667-16-1
Format: Kindle
Pages: 370
In the preface of this book, Ethan Siegel recounts his experience as a new college professor in 2009, thrilled to learn that he would be teaching the college's introductory astronomy class. The only disappointment was that he could not find a textbook that described astronomy the way that he wanted to teach it. Siegel did not want to simply describe the current astronomical understanding. He wanted to start at the beginning and tell a story. What questions did we start with? How did we answer them? What new questions did those answers provoke? I enjoyed Sabine Hossenfelder's Lost in Math, but I'm even happier that she included in it a recommendation of Beyond the Galaxy as a good introduction to modern astronomy. I am completely in agreement with Siegel: Astronomy is interesting however it is described, but the best presentation for the interested lay person who does not intend to become a professional astronomy is chronological. (I feel the same way about physics and mathematics.) The narrative structure of scientific discovery draws me in every time. It's so satisfying to see our knowledge growing, breakthrough by breakthrough, only to meet the next puzzling inconsistency. It also is a natural fit for bringing someone up to speed on a field, since the history of the field starts with simpler questions and problems, and the complications of later discoveries build on the gaps left by earlier theories. Siegel tells that story well. Beyond the Galaxy starts with observations of seasonal changes in the path of the Sun through the sky and phases of the moon, quickly moves on to Eratosthenes's calculation of the circumference of the Earth, and continues at pace through early astronomy. The first chapter brings the reader up to the start of the 20th century, after which Siegel slows down and spends the remaining 10 chapters on the last 100 years: special and general relativity, the expansion of the universe, the Big Bang, dark matter, dark energy, the inflationary period, and more. For me, this was perfect. The story of early astronomy and the fight over the heliocentric model of the universe is worth telling, but it's been told many times before and features prominently in history books outside of science classes. The weight of material in this book matched my interest: What new developments have there been after general relativity and the Big Bang theory? What inconsistencies are dark matter and dark energy explaining? What is the inflationary period and how did it modify the Big Bang theory? (My half-formed impression of this part of astronomy from news articles turned out to be almost entirely wrong.) And how do we know all of this? The question of how we know as much as we do about the early origins of the universe turned out to be far more interesting than I had realized. It's one of those pieces of science where, when given only the conclusions, it's hard to imagine how our theories could be as complete as they are. Siegel is a bit repetitive in how he talks about theory development and occasionally skims over the details to avoid getting too deep into techniques and math, but he does a great job explaining the predictions of a theory, the design of experiments to prove or disprove those predictions, and the implications for the boundaries of what we know. This is the first time someone has explained the theorized origin of matter-antimatter asymmetry in sufficient detail that I was able to understand and internalize it. Dark matter and dark energy are the current topics of astronomy that get the most press, and the coverage of both is very good here. I was particularly fascinated by the link between dark matter and the superstructures of the universe that we've discovered. Astronomy does scale in a way that no other science can manage, and the scale of structure we've discovered and been able to map has gone up considerably since the last time I refreshed my astronomical knowledge. If you're looking for a math-light, comprehensive introduction to modern astronomy, I can wholeheartedly recommend this book. That goes double if you, like me, prefer a chronological view of scientific discovery so that you can understand what questions people were trying to answer and how our understanding changed over time. One note on format: Astronomy being what it is, this book is very heavy on pictures. It is not at all suitable for a dedicated ebook reader. The Kindle tries, but the combination of a small screen and black and white e-ink fail miserably. I switched to reading on a tablet, and that was a far better experience. The Kindle app still struggles to put the figures next to the text that discusses them, but that's a typesetting problem that I often have with physical books as well. If you don't have a tablet and you aren't willing to read on a computer screen, you probably want a physical copy of this one (which, sadly, means academic textbook pricing). Rating: 8 out of 10

17 April 2020

Julien Danjou: Being in Charge

Being in ChargeIf you never heard of the 10x engineer myth, it's a pretty great concept. It boils down to the idea where an engineer could be 10x more efficient than a random engineer. I find this fantastically twisted.Last week, I sat and chat with Alexis Monville in Le Podcast a podcast that equips you to make positive change in your organization. We talked about that 10x Engineer myth, and from there we digressed on how to grow your career and handle the different aspects of it.This was a very interesting exchange. Alexis is actually going to publish a new book next month (May 2020) entitled I am a Software Engineer and I am in charge.
Being in Charge
Lucky me, this week, I had the chance to be able to read the book before everybody else which means I actually read after our recording. I understood why Alexis said that a lot of what I was talking about during our podcast resonated with him. I send a detailed review of the book to Alexis and Michael if you're curious. I'm definitely recommending this book if you want to stop complaining about your job and start understanding how to pull the strings.I wish I had this book available 10 years ago!

16 April 2020

Raphaël Hertzog: Freexian s report about Debian Long Term Support, March 2020

A Debian LTS logo Like each month, here comes a report about the work of paid contributors to Debian LTS. Individual reports In March, 252 work hours have been dispatched among 14 paid contributors. Their reports are available: Evolution of the situation March was a strange month for many people all over the globe. Here we ll just express our hopes that you are and will be well! LTS gained a new contributor in March, Anton Gladky, however he then decided to become active later this year. Similarly Hugo Lefeuvre notified us that he ll be inactive in April. In case you missed it (or missed to act), please read this post about keeping Debian 8 Jessie alive for longer than 5 years. If you expect to have Debian 8 servers/devices running after June 30th 2020, and would like to have security updates for them, please get in touch with Freexian. Hurry up: the end of Jessie LTS is coming in less than three months! The security tracker currently lists 25 packages with a known CVE and the dla-needed.txt file has 23 packages needing an update. Thanks to our sponsors New sponsors are in bold.

No comment Liked this article? Click here. My blog is Flattr-enabled.

15 April 2020

Ian Jackson: Adapter to use camera tripod as a microphone stand

People complained that my laptop sound was buzzy, so I bought a proper microphone (thanks to mdw for advice, and loan of some kit). Proper microphones come with a holder that accepts a screw from your microphone stand. In my case, 3/8" "BSW" - similar to 3/8" US standard "coarse" (or if you unscrew a supplied insert, a special 27tpi 5/8" UNS). I don't have a microphone stand and I didn't want to buy one. I have two camera tripods - a small one and a big one. My camera tripods have 1/4" coarse (20tpi) UNC screws. Also, we have a 3D printer at home. And nowadays you can download a configurable screw thread from the internet. So I made myself an adapter. After a few iterations I have a pretty good object which can be used to fit my new microphone to either camera tripod. (I found that a UNC thread was close enough to fit the microphone's BSW.) Of coure, this is Open Hardware and the (short) source code is available. (To build it you'll want to git clone to get the included files.) Pictures below the cut. The flange at the bottom is because tripods usually have a soft top which is supposed to stop scratching the camera; a narrow hexagonal base would make gouges, hence the wide base. ( Read more... )

comment count unavailable comments

14 April 2020

Emmanuel Kasper: Recommended keyboard settings for Productivity and Usability, for European Programmers

TLDR: setxkbmap -layout us -variant altgr-intl and become a happier programmer.
The case for QWERTY for European ProgrammersIf you re working on Unix / Linux, or C based programming languages, it can make sense to switch to the qwerty(us) keyboard layout. Why ?
Unix, C, Perl, Java, and most of programming languages have been conceived on QWERTY keyboards.
So when the designers choose special characters to use for the language synthax, they simply choose what was easy to access on their own keyboard. This has been historically documented for the vi editor.

To give an example, using an Unix shell you have to type the dot . and slash / symbols quite often to navigate the filesystem. The two keys producing these symbols, are nicely aligned on a QWERTY layout and do not require a key combination to be entered. So you can quickly enter something like ../.. using a single hand.
Now using a QWERTZ layout, like in Germany / Austria, you have the . symbol easily accessible, but you need to combine two keys ( Shift + 7 ) to get a / .
And if you are a poor soul using an AZERTY layout, to get the . and / symbol you need each time a key combo.
The need of key combos is bad not only for speed (multiple keys to lookup) but also for usability, as you have to stretch your fingers to reach the key if using a single hand, provoking repetitive strain injury. You might be smiling but this is commonly known amongst Emacs Users, due to the prominent use of commands using Ctrl and Alt combos, and led to the creation of an Emacs Ergonomic wiki.

This goes as well for many symbol commonly used in programming languages, think for instance about the semicolon ; for terminating a statement, the [ , and ] symbols for defining an array, and the backslash \ for escaping.
All these keys are accessible via a single keypress on qwerty and require key combos on qwertz and azerty. No wonder Linux, Minix and BSD were invented on non-azerty layouts: in France we were still busy typing the path to the source code, when in other parts of the world people already had the file open in their editor.
You don t need to throw away your existing keyboards when learning the qwerty (us) layout: for a couple of euros, you can find on ebay alternate keys stickers to put on your laptop.
Accessing keys with diacritics with the AltGr International variantNow if you want to switch to a qwerty keyboard layout, and you re French or German, you might wonder how to access the characters with diacritics, the and of French and and of German.
Fortunately there is a very clever keyboard variant for the us layout who uses the AltGr key, to make all these keys accessible, just hiding them behind the AltGr key.
Need ? that s just AltGr + e away. Needs ? That s just AltGr + , Needs ? AltGr +p
See the coolness of that ? You can type all international diacritics in an easy way, and there are even keyboard stickers for that too.

Now how to access to this layout of wonder ?
You can configure the layout and variant in Xorg, in debian/ubuntu that would be entering in /etc/default/keyboard
XKBLAYOUT="us"
XKBVARIANT="altgr-intl"
After restarting the X server, you can check that the settings have been applied with
setxkbmap -print -verbose 10
If using Gnome, you can also set the keyboard layout and variant by changing the schema org.gnome.desktop-inputsources, which will override the desktop-agnostic settings of /etc/default/keyboard.
For this you can either call
dconf write /org/gnome/desktop/input-sources/sources "[('xkb', 'us+altgr-intl')]"
or navigate with the gui tool dconf-settings to org.gnome.desktop-inputsources and set the value there.

If you want to further improve your keyboard layout, you can also have a look at swapping Ctrl and Alt, as described here (French language article) Happy hacking !

Daniel Lange: I think we need more creativity in statistics

" 'Boa constrictors swallow their prey whole, without chewing it. After that they are not able to move, and they sleep through the six months that they need for digestion.' I pondered deeply, then, over the adventures of the jungle. And after some work with a colored pencil I succeeded in making my first drawing. My Drawing Number One. It looked something like this: Boa Constrictor by Antoine de Saint Exup ry I showed my masterpiece to the grown-ups, and asked them whether the drawing frightened them. But they answered: 'Frighten? Why should any one be frightened by a hat?' My drawing was not a picture of a hat. It was a picture of a boa constrictor digesting an elephant. But since the grown-ups were not able to understand it, I made another drawing: I drew the inside of a boa constrictor, so that the grown-ups could see it clearly. They always need to have things explained. My Drawing Number Two looked like this: Boa Constrictor in sectional drawing by Antoine de Saint Exup ry The grown-ups' response, this time, was to advise me to lay aside my drawings of boa constrictors, whether from the inside or the outside, and devote myself instead to geography, history, arithmetic, and grammar. That is why, at the age of six, I gave up what might have been a magnificent career as a painter. I had been disheartened by the failure of my Drawing Number One and my Drawing Number Two. Grown-ups never understand anything by themselves, and it is tiresome for children to be always and forever explaining things to them." from The Little Prince by Antoine de Saint Exup ry Outcome of Cases (Recovery or Death) in Germany by Worldometers from the Corona Fun with Statistics department at Worldometers (source, archive.org link)

13 April 2020

Shirish Agarwal: Migrant worker woes and many other stories

I was gonna use this blog post to share about the migrant worker woes as there has been multiple stories doing the rounds. For e.g. a story which caught the idea of few people but most of us, i.e. middle-class people are so much into our own thing that we care a fig leaf about what happens to migrants. This should not be a story coming from a humane society but it seems India is no different than any other country of the world and in not a good way. Allow me to share
Or for those who don t like youtube, here s an alternative link https://www.invidio.us/watch?v=JGEgZq_1jmc Now the above two editorial shares two stories, one of Trump retaliatory threat to India in the Q&A of the journalist. In fact, Trump has upped the ante on visa sanctions as India buckled so easily under pressure. There have been other stories doing the rounds how people who have illnesses who need HCQ in India are either dying or are close to death because of unavailability of HCQ in the medicine shop. There have been reports in Pune as well as South Mumbai (one of the poshest localities in Mumbai/Bombay) that medicine shops are running empty or emptier. There have been so many stories on that, with reporters going to shops and asking owners of the medicine shops and shop-owners being clueless. I think the best article which vividly describes the Government of India (GOI) response to the pandemic is the free-to-read article shared by Arundhati Roy in Financial Times. It has reduced so much of my work or sharing that it s unbelievable. And she has shared it with pictures and all so I can share other aspects of how the pandemic has been affecting India and bringing the worst out in the Government in its our of need. In fact, not surprisingly though, apparently there was also a pro-Israel similar thing which happened in Africa too . As India has too few friends now globally, hence it decided to give a free pass to them.

Government of India, news agencies and paid News One of the attempts the state tried to do, although very late IMHO is that it tried to reach out to the opposition i.e. Congress party and the others. Mrs. Sonia Gandhi, who is the Congress president asked that the Government should not run any of its ads on private television channels for a period of two years. There had been plenty of articles, both by medianama and others who have alleged that at least from the last 6 odd years, Government ads. comprise of almost 50-60% advertising budget of a channel advertising budget. This has been discussed also in medianama s roundtable on online content which happened few months back. While an edited version is out there on YT, this was full two day s event which happened across two different cities.
or the alternative to youtube https://www.invidio.us/watch?v=c1PhWR1-Urs It was as if the roundtable discussions were not enough, Mrs. Gandhi clarion call was answered by News Broadcaster s Association (NBA) and this is what they had to say
News Broadcasters Association reply to Mrs. Gandhi
To put it simply, NBA deplored the suggestion by Mrs. Gandhi and even called the economy in recession and all they had were the Government s own advertising budget to justify their existence. The statements in themselves are highly pregnant and reveal both the relationship that the media, print or mainstream news channels have with the Government of India. Now if you see that, doesn t it make sense that media always slants the story from the Government s perspective rather than remaining neutral. If my bread basket were on the onus of me siding with the Govt. that is what most sane persons would do, otherwise they would resign and leave which many reporters who had a conscience did. Interestingly enough, the NBA statement didn t just end there but also used the word recession , this is the term that Government of India (GOI) hates and has in turn has been maintaining the word, terminology slowdown . While from a layman s perspective the two terms may seem to be similar, if India has indeed been in recession then the tools and the decisions that should have been taken by GOI should have been much different than what they took. Interestingly, enough GOI has refrained from saying anything on the matter which only reveals their own interests in the matter. Also if an association head is making the statement, it is more than likely that he consulted a lawyer or two and used application of mind while drafting the response. In other words, or put more simply, this was a very carefully drafted letter because they know that tomorrow the opposition party may come into power so they don t want to upset the power dynamics too much.

Privacy issues arising due to the Pandemic On the same Financial Times, two stories which dealt with the possible privacy violations due to the Pandemic have been doing the rounds. The first one, by Yuval Noah Harari is more exploratory by nature and makes some very good points without going far too deep into specific instances of recent times but rather goes into history and past instances where Governments have used the pandemics to exert more control over their populace and drive their agenda. I especially liked the last few lines which he shared in his op-ed Even if the current administration eventually changes tack and comes up with a global plan of action, few would follow a leader who never takes responsibility, who never admits mistakes, and who routinely takes all the credit for himself while leaving all the blame to others. Yuval Noah Harari . The whole statement could right fit onto the American President which he was talking about while at the same time, fits right into the current Indian Prime Minister, Boris Johnson of UK and perhaps Jair Bolsanaro of Brazil. All these three-four individuals have in common is that most of them belong to right-wing and hence cater only to the rich industrialist s agenda. While I don t know about Jair Bolsanaro much, at least three out of four had to turn to socialism and had to give some bailout packages to the public at large, even though continuing to undermine their own actions. More on this probably a bit down the line. The second story shared by Nic Fildes and Javier Espinoza who broke the story of various surveillance attempts and the privacy concerns that people have. Even the Indian PMO has asked this data and because there was no protest by the civil society, a token protest was done by COAI (Cellular Operator Association of India) but beyond that nothing, I am guessing because the civil society didn t make much noise as everybody is busy with their own concerns of safety and things going on, it s possible that such data may have gone to the Government. There is not much new here that people who had been working on the privacy issues know, it s just how easy Governments are finding to do it. The part of informed consent is really a misnomer . Governments lie all the time, for e.g. in the UK, did the leave party and people take informed consent, no they pushed their own agenda. This is and will be similar in many countries of the world.

False Socialism by RW parties In at least the three countries I have observing, simply due to available time, that lot of false promises are being made by our leaders and more often than not, the bailouts will be given to already rich industrialists. An op-ed by Vivek Kaul, who initially went by his handle which means somebody who is educated but unemployed. While Vivek has been one-man army in revealing most of the Government s mischiefs especially as fudging numbers are concerned among other things, there have been others too. As far as the US is concerned, an e-zine called free press (literally) has been sharing Trump s hollowness and proclamations for U.S. . Far more interestingly, I found New York times investigated and found a cache of e-mails starting from early January, which they are calling Red Dawn . The cache is undeniable proof that medical personnel in the U.S. were very much concerned since January 2020 but it was only after other countries started lock-down that U.S. had to follow suit. I am sure Indian medical professionals may have done similar mail exchanges but we will never know as the Indian media isn t independent enough.

Domestic violence and Patriarchy There have been numerous reports of domestic violence against women going up, in fact two prominent publications have shared pieces about how domestic violence has gone up in India since the lockdown but the mainstream press is busy with its own tropes, the reasons already stated above. In fact, interestingly enough, most women can t wear loose fitting clothes inside the house because of the near ones being there 24 7 . This was being shared as India is going through summer where heat waves are common and most families do not have access to A/C s and rely on either a fan or just ventilation to help them out. I can t write more about this as simply I m not a woman so I haven t had to face the pressures that they have to every day. Interestingly though, there was a piece shared by arre. Interestingly, also arre whose content I have shared a few times on my blog has gone from light, funny to be much darker and more serious tone. Whether this is due to the times we live in is something that a social scientist or a social anthropologist may look into in the times to come. One of the good things though, there hasn t been any grid failures as no industrial activity is happening (at all). In fact SEB s (State Electricity Boards) has shown a de-growth in electricity uptake as no industrial activity has been taken. While they haven t reduced any prices (which they ideally should have) as everybody is suffering.

Loot and price rise Again, don t think it is an Indian issue but perhaps may be the same globally. Because of broken supply chains, there are both real and artificial shortages happening which is leading to reasonable and unreasonable price hikes in the market. Fresh veggies which were normally between INR 10/- to INR 20/- for 250 gm have reached INR 40/- 50/- and even above. Many of the things that we have to become depend upon are not there anymore. The shortage of plastic bottles being case in point.
Aryan Plastic bottle
This and many others like these pictures have been shared on social media but it seems the Government is busy doing something else. The only thing we know for sure is that the lock-down period is only gonna increase, no word about PPE s (Personal Protection Equipment) or face masks or anything else. While India has ordered some, those orders are being diverted to US or EU. In fact, many doctors who have asked for the same have been arrested, sacked or suspended for asking such inconvenient questions, although whether in BJP ruled states or otherwise. In fact, the Centre has suspended MPLADS funds , members of parliament get funds which they can use to provide relief work or whatever they think the money is best to spend upon.

Conditions of Labor in the Pandemic Another sort of depressing story has been how the Supreme Court CJI Justice SA Bobde has made statements and refrained from playing any role in directing the Center to provide relief to the daily wage laborers. In fact, Mr. Bobde made statements such as why they need salaries if they are getting food. This was shared by barandbench, a site curated by lawyers and reporters alike. Both livelaw as well as barandbench have worked to enhance people s awareness about the legal happenings in our High Courts and Supreme Court. And while sadly, they cannot cover all, they at least do attempt to cover a bit of what s hot atm. The Chief Justice who draws a salary of INR 250,000 per month besides other perks is perhaps unaware or doesn t care about fate of millions of casual workers, 400 460 million workers who will face abject poverty and by extension even if there are 4 members of the family so probably 1.2 billion people will fall below the poverty line. Three, four major sectors are going to be severely impacted, namely Agriculture, Construction and then MSME (Micro, small and medium enterprises) which cover everything from autos, industrial components, FMCG, electronics, you name it, it s done by the MSME sector. We know that the Rabi crop, even though it was gonna be a bumper crop this year will rot away in the fields. Even the Kharif crop whose window for sowing is at the most 2-3 weeks will not be able to get it done in time. In fact, with the extended lockdown of another 21 days, people will probably return home after 2 months by which time they would have nothing to do there as well as here in the cities. Another good report was done by the wire, the mainstream media has already left the station.

Ministry of Public Health There was an article penned by Dr. Edmond Fernandes which he published last year. The low salary along with the complexities that Indian doctors are and may face in the near future are just mind-boggling.

The Loss Losses have already started pouring in. Just today Air Deccan has ceased all its operations. I had loved Mr. Gopinath s airline which was started in the early 2000 s. While I won t bore you with the history, most of it can be seen from simplify Deccan . This I believe is just the start and it s only after the few months after the lock-down has been lifted would we really know the true extent of losses everywhere. And the more lenghthier the lockdown, the more difficult it would be businesses to ramp back. People have already diagnosed at the very least 15-20 sectors of the economy which would be hit and another similar or more number of sectors which will have first and second-order of losses and ramp-downs. While some guesses are being made, many are wildly optimistic and many are wildly pessimistic, as shared we would only know the results when the lockdown is opened up.

Predictions for the future While things are very much in the air, some predictions can be made or rationally deduced. For instance, investments made in automation and IT would remain and perhaps even accelerate a little. Logistics models would need to be re-worked and maybe, just maybe there would be talk and action in making local supply chains a bit more robust. Financing is going to be a huge issue for at least 6 months to a year. Infrastructure projects which require huge amount of cash upfront will either have to be re-worked or delayed, how they will affect projects like Pune Metro and other such projects only time will tell.

Raghuram Rajan Raghuram Rajan was recently asked if he would come back and let bygones be bygones. Raghuram in his own roundabout way said no. He is right now with Chicago Booth doing the work that he always love. Why would he leave that and be right in the middle of the messes other people have made. He probably gets more money, more freedom and probably has a class full of potential future economists. Immigration Control, Conferences and thought experiment There are so many clueless people out there, who don t know why it takes so long for any visa to be processed. From what little I know, it is to verify who you say you are and you have valid reason to enter the country. The people from home ministry verify credentials, as well as probably check with lists of known criminals and their networks world-wide. They probably have programs for such scenarios and are part and parcel of their everyday work. The same applies to immigration control at Airports. there has been a huge gap at immigration counters and the numbers of passengers who were flying internationally to and fro from India. While in India, we call them as Ministry of Home Affairs, in U.S. it s Department of Homeland security, other countries using similar jargons. Now even before this pandemic happened, the number of people who are supposed to do border control and check people was way less and there have been scenes of Air rage especially in Indian airports after people came after a long-distance flight. Now there are couple of thought experiments, just day before yesterday scientists discovered six new coronaviruses in bats and scientists in Iceland found 40 odd mutations of the virus on people. Now are countries going to ban people from Iceland as in time the icelandic people probably would have anti-bodies on all the forty odd mutations. Now if and when they come in contact onto others who have not, what would happen ? And this is not specifically about one space or ethnicity or whatever, microbes and viruses have been longer on earth than we have. In our greed we have made viruses resistant to antibiotics. While Mr. Trump says as he discovered it today, this has been known to the medical fraternity since tht 1950 s. CDC s own chart shows it. We cannot live in fear of a virus, the only way we can beat it is by understanding it and using science. Jon Cohen shared some of the incredible ways science is looking to beat this thing
or as again an alternative to youtube https://www.invidio.us/watch?v=MPVG_n3w_vM One of the most troubling question is how the differently-abled communities which don t have media coverage at the best of times, haven t had any media coverage at all during the pandemic. What are their stories and what they are experiencing ? How are they coping ? Are there anyways we could help each other ? By not having those stories, we perhaps have left them more vulnerable than we intend. And what does that speak about us, as people or as a community or a society ?

Silver Linings While there is not a lot to be positive about, one interesting project I came about is openbreath.tech . This is an idea, venture started by IISER (Indian Institute of Science Education and Research) , IUCAA (Inter-University Centre for Astronomy and Astrophysics). They are collaborating with octogeneraian Capt (Retd) Rustom Barucha from Barucha Instrumentation and Control, besides IndoGenius, New Delhi, and King s College, London. The first two institutes are from my home town, Pune. While I don t know much of the specifics of this idea other than that there is an existing Barucha ventilator which they hope to open-source and make it easier for people to produce their own. While I have more questions than answers at this point, this is something hopefully to watch out for in the coming days and weeks. The other jolly bit of good news has come from Punjab where after several decades, people in Northern Punjab are finally able to see the Himalayas or the Himalayan mountain range.
Dhauladhar range Northern Punjab Copyright CNN.Com
There you have it, What I have covered is barely scratching the surface. As a large section of the media only focuses on one narrative, other stories and narratives are lost. Be safe, till later.

Matthew Garrett: Implementing support for advanced DPTF policy in Linux

Intel's Dynamic Platform and Thermal Framework (DPTF) is a feature that's becoming increasingly common on highly portable Intel-based devices. The adaptive policy it implements is based around the idea that thermal management of a system is becoming increasingly complicated - the appropriate set of cooling constraints to place on a system may differ based on a whole bunch of criteria (eg, if a tablet is being held vertically rather than lying on a table, it's probably going to be able to dissipate heat more effectively, so you should impose different constraints). One way of providing these criteria to the OS is to embed them in the system firmware, allowing an OS-level agent to read that and then incorporate OS-level knowledge into a final policy decision.

Unfortunately, while Intel have released some amount of support for DPTF on Linux, they haven't included support for the adaptive policy. And even more annoyingly, many modern laptops run in a heavily conservative thermal state if the OS doesn't support the adaptive policy, meaning that the CPU throttles down extremely quickly and the laptop runs excessively slowly.

It's been a while since I really got stuck into a laptop reverse engineering project, and I don't have much else to do right now, so I've been working on this. It's been a combination of examining what source Intel have released, reverse engineering the Windows code and staring hard at hex dumps until they made some sort of sense. Here's where I am.

There's two main components to the adaptive policy - the adaptive conditions table (APCT) and the adaptive actions table (APAT). The adaptive conditions table contains a set of condition sets, with up to 10 conditions in each condition set. A condition is something like "is the battery above a certain charge", "is this temperature sensor below a certain value", "is the lid open or closed", "is the machine upright or horizontal" and so on. Each condition set is evaluated in turn - if all the conditions evaluate to true, the condition set's target is implemented. If not, we move onto the next condition set. There will typically be a fallback condition set to catch the case where none of the other condition sets evaluate to true.

The action table contains sets of actions associated with a specific target. Once we've picked a target by evaluating the conditions, we execute the actions that have a corresponding target. Actions are things like "Set the CPU power limit to this value" or "Load a passive policy table". Passive policy tables are simply tables associating sensors with devices and an associated temperature limit. If the limit is exceeded, the associated device should be asked to reduce its heat output until the situation is resolved.

There's a couple of twists. The first is the OEM conditions. These are conditions that refer to values that are exposed by the firmware and are otherwise entirely opaque - the firmware knows what these mean, but we don't, so conditions that rely on these values are magical. They could be temperature, they could be power consumption, they could be SKU variations. We just don't know. The other is that older versions of the APCT table didn't include a reference to a device - ie, if you specified a condition based on a temperature, you had no way to express which temperature sensor to use. So, instead, you specified a condition that's greater than 0x10000, which tells the agent to look at the APPC table to extract the device and the appropriate actual condition.

Intel already have a Linux app called Thermal Daemon that implements a subset of this - you're supposed to run the binary-only dptfxtract against your firmware to parse a few bits of the DPTF tables, and it writes out an XML file that Thermal Daemon makes use of. Unfortunately it doesn't handle most of the more interesting bits of the adaptive performance policy, so I've spent the past couple of days extending it to do so and to remove the proprietary dependency.

My current work is here - it requires a couple of kernel patches (that are in the patches directory), and it only supports a very small subset of the possible conditions. It's also entirely possible that it'll do something inappropriate and cause your computer to melt - none of this is publicly documented, I don't have access to the spec and you're relying on my best guesses in a lot of places. But it seems to behave roughly as expected on the one test machine I have here, so time to get some wider testing?

comment count unavailable comments

12 April 2020

Christian Kastner: hello, world

It slipped my mind that the recent March 17, 2020, marked the 10th anniversary of my first upload to the official Debian archive. How time flies! Although I was never much of a blogging person myself, I do enjoy reading other people's contributions, and on some occasions felt that the Planet Debian feed might have been a more appropriate medium for sharing a particular tidbit, instead of sharing it with a specific mailing list, or not sharing at all. So, to commemorate my anniversary, I venture into this new space. first upload: https://lists.debian.org/debian-devel-changes/2010/03/msg01807.html

Dirk Eddelbuettel: #25: Test, test, test, those R 4.0.0 binaries with Ubuntu 20.04 and Rocker

Welcome to the 25nd post in the randomly recurring R recitations series, or R4 for short. Just yesterday, we posted a short post along with a video and supporting slides. It covered how to test the soon-to-be-released R 4.0.0 on a custom Ubuntu 18.04 Rocker container. A container for Ubuntu 20.04, which is itself in final beta stages, was being built while the video was made. As it is available now, we created a quick follow-up video showing the use under Ubuntu 20.04:

The updated supporting slides from the video are still at this link. What we showed in both videos does of course also work directly on Ubuntu (or Debian, using those source repos) installations; the commands shown in the Rocker use case generally apply equally to a normal installation. If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Dirk Eddelbuettel: #24: Test, test, test, those R 4.0.0 binaries with Ubuntu and Rocker

Welcome to the 24nd post in the relentlessly regular R ravings series, or R4 for short. R 4.0.0 will be released in less than two weeks, and testing is very important. I had uploaded two alpha release builds (at the end of March and a good week ago) as well as a first beta release yesterday, all to the Debian experimental distribution (as you can see here) tracking the release schedule set by Peter Dalgaard. Because R 4.0.0 will require reinstallation of all packages, it makes some sense to use a spare machine. Or a Docker container. So to support that latter mode, I have now complemented the binaries created from the r-base source package with all base and recommended packages, providing a starting point for actually running simple tests. Which is what we do in the video, using again the R on Ubuntu (18.04) Rocker container:

Slides from the video are at this link. This container based on 18.04 is described here on the Docker Hub; a new 20.04 container with the pre-release of the next Ubuntu LTS should be there shortly once it leaves the build queue. What we showed does of course also work on direct Ubuntu (or Debian, using those source repos) installations; the commands shown in the Rocker use case generally apply equally to a normal installation. If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

8 April 2020

David Bremner: Tangling multiple files

I have lately been using org-mode literate programming to generate example code and beamer slides from the same source. I hit a wall trying to re-use functions in multiple files, so I came up with the following hack. Thanks 'ngz' on #emacs and Charles Berry on the org-mode list for suggestions and discussion. 
(defun db-extract-tangle-includes ()
  (goto-char (point-min))
  (let ((case-fold-search t)
        (retval nil))
    (while (re-search-forward "^#[+]TANGLE_INCLUDE:" nil t)
      (let ((element (org-element-at-point)))
        (when (eq (org-element-type element) &aposkeyword)
          (push (org-element-property :value element) retval))))
    retval))
(defun db-ob-tangle-hook ()
  (let ((includes (db-extract-tangle-includes)))
    (mapc #&aposorg-babel-lob-ingest includes)))
(add-hook &aposorg-babel-pre-tangle-hook #&aposdb-ob-tangle-hook t)
Use involves something like the following in your org-file. 
#+SETUPFILE: presentation-settings.org
#+SETUPFILE: tangle-settings.org
#+TANGLE_INCLUDE: lecture21.org
#+TITLE: GC V: Mark & Sweep with free list
For batch export with make, I do something like 
%.tangle-stamp: %.org
    emacs --batch --quick  -l org  -l $ HOME /.emacs.d/org-settings.el --eval "(org-babel-tangle-file \"$<\")"
    touch $@

7 April 2020

Shirish Agarwal: GMRT 2020 and lots of stories

First of all, congratulations to all those who got us 2022 Debconf, so we will finally have a debconf in India. There is of course, lot of work to be done between now and then. For those who would be looking forward to visit India and especially Kochi I would suggest you to hear this enriching tale
I am sorry I used youtube link but it is too good a podcast not to be shared. Those who don t want youtube can use the invidio.us link for the same as shared below. https://www.invidio.us/watch?v=BvjgKuKmnQ4 I am sure there are lot more details, questions, answers etc. but would direct them gently to Praveen, Shruti, Balasankar and the rest who are from Kochi to answer if you have any questions about that history.

National Science Day, GMRT 2020 First, as always, we are and were grateful to both NCRA as well as GMRT for taking such good care of us. Even though Akshat was not around, probably getting engaged, a few of us were there. About 6-7 from the Mozilla Nasik while the rest representing the foss community. Here is a small picture which commentrates the event
National Science Day, GMRT 2020
While there is and was a lot to share about the event. For e.g. Akshay had bought RPI- Zero as well as RPI-2 (Raspberry Pi s ) and showed some things. He had also bought up a Debian stable live drive with persistence although the glare from the sun was too much that we couldn t show it to clearly to students. This was also the case with RPI but still we shared what and how much we could. Maybe next year, we either ask them to have double screens or give us dark room so we can showcase things much better. We did try playing with contrast and all but it didn t have much of an effect . Of course in another stall few students had used RPI s as part of their projects so at times we did tell some of the newbies to go to those stalls and see and ask about those projects so they would have a much wider experience of things. The Mozilla people were pushing VR as well as Mozilla lite the browser for the mobile. We also gossiped quite a bit. I shared about indicatelts , a third-party certificate extension although I dunno if I should file a wnpp about it or not. We didn t have a good experience of when I had put an RFP (Request for Package) which was accepted for an extension which had similar functionality which we later come to know was sharing the sites people were using the extension to call home and share both the URL and the IP Address they were using it from. Sadly, didn t leave a good taste in mouth

Delhi Riots One thing I have been disappointed with is the lack of general awareness about things especially in the youth. We have people who didn t know that for e.g. in the Delhi riots which happened recently the law and order (Police) lies with Home Minister of India, Amit Shah. This is perhaps the only capital in the world which has its own Chief Minister but doesn t have any say on its law and order. And this has been the case for last 70 years i.e. since independance. The closest I know so far is the UK but they too changed their tune in 2012. India and especially Delhi seems to be in a time-capsule which while being dysfunctional somehow is made to work. In many ways, it s three body or a body split into three personalities which often makes governance a messy issue but that probably is a topic for another day. In fact, scroll had written a beautiful editorial that full statehood for Delhi was not only Arvind Kejriwal s call (AAP) but also something that both BJP as well as Congress had asked in the past. In fact, nothing about the policing is in AAP s power. All salaries, postings, transfers of police personnel everything is done by the Home Ministry, so if any blame has to be given it has to be given to the Home Ministry for the same.

American Capitalism and Ventilators America had been having a history of high cost healthcare as can be seen in this edition of USA today from 2017 . The Affordable Care Act was signed as a law by President Obama in 2010 which Mr. Trump curtailed when he came into power couple of years back. An estimated 80,000 people died due to seasonal flu in 2018-19 . Similarly, anywhere between 24-63,000 have supposed to have died from Last October to February-March this year. Now if the richest country can t take care of their population which is 1/3rd of the population of this country while at the same time United States has thrice the area that India has. This I am sharing as seasonal flu also strikes the elderly as well as young children more than adults. So in one senses, the vulnerable groups overlap although from some of the recent stats, for Covid-19 even those who are 20+ are also vulnerable but that s another story altogether. If you see the CDC graph of the seasonal flu it is clear that American health experts knew about it. One another common factor which joins both the seasonal flu and covid is both need ventilators for the most serious cases. So, in 2007 it was decided that the number of ventilators needed to be ramped up, they had approximately 62k ventilators at that point in time all over U.S. The U.S. in 2010, asked for bids and got bid from a small californian company called Newport Medic Instruments. The price of the ventilators was approximately INR 700000 at 2010 prices, while Newport said they would be able to mass-produce at INR 200000 at 2010 prices. The company got the order and they started designing the model which needed to be certified by FDA. By 2011, they got the product ready when a big company called Covidgen bought Newport Medic and shutdown the project. This was shared in a press release in 2012. The whole story was broken by New York Times again, just a few days ago which highlighted how America s capitalism rough shod over public health and put people s life unnecessarily in jeopardy. If those new-age ventilators would have been a reality then not just U.S. but India and many other countries would have bought the ventilators as every county has same/similar needs but are unable to pay the high cost which in many cases would be passed on to their citizens either as price of service, or by raising taxes or a mixture of both with public being none the wiser. Due to dearth of ventilators and specialized people to operate it and space, there is possibility that many countries including India may have to make tough choices like Italian doctors had to make as to who to give ventilator to and have the mental and emotional guilt which would be associated with the choices made.

Some science coverage about diseases in wire and other publications Since Covid coverage broke out, the wire has been bringing various reports of India s handling of various epidemics, mysteries, some solved, some still remaining unsolved due to lack of interest or funding or both. The Nipah virus has been amply discussed in the movie Virus (2019) which I shared in the last blog post and how easily it could have been similar to Italy in Kerala. Thankfully, only 24 people including a nurse succumbed to that outbreak as shared in the movie. I had shared about Kerala nurses professionalism when I was in hospital couple of years back. It s no wonder that their understanding of hygeine and nursing procedures are a cut above the rest hence they are sought after not just in India but world-over including US and UK and the middle-east. Another study on respitory illness was bought to my attention by my friend Pavithran.

Possibility of extended lockdown in India There was talk in the media of extended lockdown or better put an environment is being created so that an extended lockdown can be done. This is probably in part due to a mathematical model and its derivatives shared by two Indian-origin Cambridge scholars who predict that a minimum 49 days lockdown may be necessary to flatten the covid curve about a week back.
Predictions of the outcome of the current 21-day lockdown (Source: Rajesh Singh, R. Adhikari, Cambridge University)
Alternative lockdown strategies suggested by the Cambridge model (Source: Rajesh Singh, R. Adhikari, Cambridge University)
India caving to US pressure on Hydroxychloroquine While there has been lot of speculation in U.S. about Hydroxychloroquine as the wonder cure, last night Mr. Trump threatened India in a response to a reporter that Mr. Modi has to say no for Hydroxychloroquine and there may be retaliations.
As shared before if youtube is not your cup you can see the same on invidio.us https://www.invidio.us/watch?v=YP-ewgoJPLw Now while there have been several instances in the past of U.S. trying to bully India, going all the way back to 1954. In fact, in recent memory, there were sanctions on India by US under Atal Bihari Vajpayee Government (BJP) 1998 but he didn t buckle under the pressure and now we see our current PM taking down our own notification from a day ago and not just sharing Hydroxychloroquine but also Paracetemol to other countries so it would look as if India is sharing with other countries. Keep in mind, that India, Brazil haven t seen eye to eye on trade agreements of late and Paracetemol prices have risen in India. The price rise has been because the API (Active Pharmaceutical Ingredients) for the same come from China where the supply chain will take time to be fixed and we would also have to open up, although should we, should we not is another question altogether. I talk about supply chains as lean supply chains were the talk since late 90 s when the Japanese introduced Just-in-time manufacturing which lead to lean supply chains as well as lot of outsourcing as consequence. Of course, the companies saved money but at the cost of flexibility and how this model was perhaps flawed was shared by a series of articles in Economist as early as 2004 when there were lot of shocks to that model and would be exaberated since then. There have been frequent shocks to these fragile ecosystem more since 2008 after the financial meltdown and this would put more companies out of business than ever before. The MSME sector in India had already been severely impacted first by demonetization and then by the horrendous implementation of GST whose cries can be heard from all sectors. Also the frequent changing of GST taxes has made markets jumpy and investors unsure. With judgements such as retrospective taxes, AGR (Adjusted Gross Revenue) etc. it made not only the international investors scared, but also domestic investors. The flight of the capital has been noticeable. This I had shared before when Indian Government shared about LRS report which it hasn t since then. In fact Outlook Business had an interesting article about it where incidentally it talked about localcircles, a community networking platform where you get to know of lot of things and whom I am also a member of. At the very end I apologize for not sharing the blog post before but then I was feeling down but then I m not the only one.

Reproducible Builds: Reproducible Builds in March 2020

Welcome to the March 2020 report from the Reproducible Builds project. In our reports we outline the most important things that we have been up to over the past month and some plans for the future.
What are reproducible builds? One of the original promises of open source software is that distributed peer review and transparency of process results in enhanced end-user security. However, whilst anyone may inspect the source code of free and open source software for malicious flaws, almost all software today is distributed as pre-compiled binaries. This allows nefarious third-parties to compromise systems by injecting malicious code into ostensibly secure software during the various compilation and distribution processes.

News The report from our recent summit in Marrakesh was published and is now available in both PDF and HTML formats. A sincere thank you to all of the Reproducible Builds community for the input to the event a sincere thank you to Aspiration for preparing and collating this report. Harmut Schorrig published a detailed document on how to compile Java applications in such as way that the .jar build artefact is reproducible across builds. A practical and hands-on guide, it details how to avoid unnecessary differences between builds by explicitly declaring an encoding as the default value differs across Linux and MS Windows systems and ensuring that the generated .jar a variant of a .zip archive does not embed any nondeterministic filesystem metadata, and so on. Janneke gave a quick presentation on GNU Mes and reproducible builds during the lighting talk session at LibrePlanet 2020. [ ] Vagrant Cascadian presented There and Back Again, Reproducibly! video at SCaLE 18x in Pasadena in California which generated some attention on Twitter. Herv Boutemy mentioned on our mailing list in a thread titled Rebuilding and checking Reproducible Builds from Maven Central repository that since the update of a central build script (the parent POM ) every Apache project using the Maven build system should build reproducibly. A follow-up discussion regarding how to perform such rebuilds was also started on the Apache mailing list. The Telegram instant-messaging platform announced that they had updated their iOS and Android OS applications and claim that they are reproducible according to their full instructions, verifying that its original source code is exactly the same code that is used to build the versions available on the Apple App Store and Google Play distribution platforms respectfully. Herv Boutemy also reported about a new project called reproducible-central which aims to allow anyone to rebuild a component from the Maven Central Repository that is expected to be reproducible and check that the result is as expected. In last month s report we detailed Omar Navarro Leija s work in and around an academic paper titled Reproducible Containers which describes in detail the workings of a user-space container tool called dettrace (PDF). Since then, the PhD student from the University Of Pennsylvania presented on this tool at the ASPLOS 2020 conference in Lausanne, Switzerland. Furthermore, there were contributions to dettrace from the Reproducible Builds community itself. [ ][ ]

Distribution work

openSUSE In openSUSE, Bernhard M. Wiedemann published his monthly Reproducible Builds status update as well as made the following changes within the distribution itself:

Debian Chris Lamb further refined his merge request for the debian-installer component to allow all arguments from sources.list files (such as [check-valid-until=no] ) in order that we can test the reproducibility of the installer images on the Reproducible Builds own testing infrastructure. (#13) Holger Levsen filed a number of bug reports against the debrebuild tool that attempts to rebuild a Debian package given a .buildinfo file as input, including: 48 reviews of Debian packages were added, 17 were updated and 34 were removed this month adding to our knowledge about identified issues. Many issue types were noticed, categorised and updated by Chris Lamb, including: Finally, Holger opened a bug report against the software running tracker.debian.org, a service for Debian Developers to follow the evolution of packages via web and email interfaces to request that they integrate information from buildinfos.debian.net (#955434) and Chris Lamb kept isdebianreproducibleyet.com up to date. [ ]

Software development

diffoscope Chris Lamb made the following changes to diffoscope, the Reproducible Builds project s in-depth and content-aware diff utility that can locate and diagnose reproducibility issues, including preparing and uploading version 138 to Debian:
  • Improvements:
    • Don t allow errors with R script deserialisation cause the entire operation to fail, for example if an external library cannot be loaded. (#91)
    • Experiment with memoising output from expensive external commands, eg. readelf. (#93)
    • Use dumppdf from the python3-pdfminer if we do not see any other differences from pdftext, etc. (#92)
    • Prevent a traceback when comparing two R .rdx files directly as the get_member method will return a file even if the file is missing. [ ]
  • Reporting:
    • Display the supported file formats into the package long description. (#90)
    • Print a potentially-helpful message if the PyPDF2 module is not installed. [ ]
    • Remove any duplicate comparator descriptions when formatting in the --help output or in the package long description. [ ]
    • Weaken Install the X package to get a better output message to may produce a better output as the former is not actually guaranteed. [ ]
  • Misc:
    • Ensure we only parse the recommended packages from --list-debian-substvars when we want them for debian/tests/control generation. [ ]
    • Add upstream metadata file [ ] and add a Lintian override for upstream-metadata-in-native-source as we are upstream. [ ]
    • Inline the RequiredToolNotFound.get_package method s functionality as it is only used once. [ ]
    • Drop the deprecated py36 = [..] argument in the pyproject.toml file. [ ]
In addition, Vagrant Cascadian updated diffoscope in GNU Guix to version 138 [ ], as well as updating reprotest our end-user tool to build same source code twice in widely differing environments and then checks the binaries produced by each build for any differences to version 0.7.14 [ ].

Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month we wrote a large number of such patches, including:

Project documentation There was further work performed on our documentation and website this month including Alex Wilson adding a section regarding using Gradle for reproducible builds in JVM projects [ ] and Holger Levsen added the report from our recent summit in Marrakesh [ ][ ]. In addition, Chris Lamb made a number of changes, including correcting the syntax of some CSS class formatting [ ], improved some filed against copy a little better [ ] and corrected a reference to calendar.monthrange Python method in a utility function. [ ]

Testing framework We operate a large and many-featured Jenkins-based testing framework that powers tests.reproducible-builds.org that, amongst many other tasks, tracks the status of our reproducibility efforts as well as identifies any regressions that have been introduced. This month, Chris Lamb reworked the web-based package rescheduling tool to:
  • Require a HTTP POST method in the web-based scheduler as not only should HTTP GET requests be idempotent but this will allow many future improvements in the user interface. [ ][ ][ ]
  • Improve the authentication error message in said rescheduler to suggest that the developer s SSL certificate may have expired. [ ]
In addition, Holger Levsen made the following changes:
  • Add a new ath97 subtarget for the OpenWrt distribution.
  • Revisit ordering of Debian suites; sort the experimental distribution last and reverse the ordering of suites to prioritise the suites in development. [ ][ ][ ]
  • Schedule Debian buster and bullseye a little less in order to allow unstable to catch up on the i386 architecture. [ ][ ]
  • Various cosmetic changes to the web-based scheduler. [ ][ ][ ][ ]
  • Improve wordings in the node health maintenance output. [ ]
Lastly, Vagrant Cascadian updated a link to the (formerly) weekly news to our reports page [ ] and kpcyrd fixed the escaping in an Alpine Linux inline patch [ ]. The usual build nodes maintenance was performed by Holger Levsen [ ][ ], Mattia Rizzolo [ ] and Vagrant Cascadian [ ][ ].
If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

This month s report was written by Bernhard M. Wiedemann, Chris Lamb, Holger Levsen and Vagrant Cascadian. It was subsequently reviewed by a bunch of Reproducible Builds folks on IRC and the mailing list.

6 April 2020

Joachim Breitner: A Telegram bot in Haskell on Amazon Lambda

I just had a weekend full of very successful serious geekery. On a whim I thought: Wouldn't it be nice if people could interact with my game Kaleidogen also via a telegram bot? This led me to learn about how I write a Telegram bot in Haskell and how I can deploy such a Haskell program to Amazon Lambda. In particular the latter bit might be interesting to some of my readers, so here is how went about it.

Kaleidogen Kaleidogen is a little contemplative game (or toy game where, starting from just unicolored disks, you combine abstract circular patterns to breed more interesting patterns. See my FARM 2019 talk for more details, or check out the source repository. BTW, I am looking for help turning it into an Android app!
KaleidogenBot in action

KaleidogenBot in action

Amazon Lambda Amazon Lambda is the Function as a service offering of Amazon Web Services. The idea is that you don t rent a server, where you have to deal with managing the whole system and that you are paying for constantly, but you just upload the code that responds to outside requests, and AWS takes care of the rest: Starting and stopping instances, providing a secure base system etc. When nobody is using the service, no cost occurs. This sounds ideal for hosting a toy Telegram bot: Most of the time nobody will be using it, and I really don't want to have to baby sit yet another service on my server. On Amazon Lambda, I can probably just forget about it. But Haskell is not one of the officially supported languages on Amazon Lambda. So to run Haskell on Lambda, one has to solve two problems:
  • how to invoke the Haskell code on the server, and
  • how to build Haskell so that it runs on the Amazon Linux distribution

A Haskell runtime for Lambda For the first we need a custom runtime. While this sounds complicated, it is actually a pretty simple concept: A runtime is an executable called bootstrap that queries the Lambda Runtime Interface for the next request to handle. The Lambda documentation is phrased as if this runtime has to be a dispatcher that calls the separate function s handler. But it could just do all the things directly. I found the Haskell package aws-lambda-haskell-runtime which provides precisely that: A function 
runLambda :: (LambdaOptions -> IO (Either String LambdaResult)) -> IO ()
that talks to the Lambda Runtime API and invokes its argument on each message. The package also provides Template Haskell magic to collect handlers of any JSON-able type and generates a dispatcher, like you might expect from other, more dynamic languages. But that was too much magic for me, so I ignored that and just wrote the handler manually: 
main :: IO ()
main = runLambda run
  where
   run ::  LambdaOptions -> IO (Either String LambdaResult)
   run opts = do
    result <- handler (decodeObj (eventObject opts)) (decodeObj (contextObject opts))
    either (pure . Left . encodeObj) (pure . Right . LambdaResult . encodeObj) result
data Event = Event
    path :: T.Text
  , body :: Maybe T.Text
    deriving (Generic, FromJSON)
data Response = Response
    statusCode :: Int
  , headers :: Value
  , body :: T.Text
  , isBase64Encoded :: Bool
    deriving (Generic, ToJSON)
handler :: Event -> Context -> IO (Either String Response)
handler Event body, path  context =
I expose my Lambda function to the world via Amazon s API Gateway, configured to just proxy the HTTP requests. This means that my code receives a JSON data structure describing the HTTP request (here called Event, listing only the fields I care about), and it will respond with a Response, again as JSON. The handler can then simply pattern-match on the path to decide what to do. For example this code handles URLs like /img/CAFFEEFACE.png, and responds with an image. 
handler :: TC -> Event -> Context -> IO (Either String Response)
handler Event body, path  context
      Just bytes <- isImgPath path >>= T.decodeHex = do
        let pngData = genPurePNG bytes
        pure $ Right Response
              statusCode = 200
            , headers = object [ "Content-Type" .= ("image/png" :: String) ]
            , isBase64Encoded = True
            , body = T.decodeUtf8 $ LBS.toStrict $ Base64.encode pngData
             
     
isImgPath :: T.Text -> Maybe T.Text
isImgPath  = T.stripPrefix "/img/" >=> T.stripSuffix ".png"
If this program would grow more, then one should probably use something more structured for routing here; maybe servant, or bridging towards wai apps (amost like wai-lamda, but that still assumes an existing runtime, instead of simply being the runtime). But for my purposes, no extra layers of indirection or abstraction are needed!

Deploying Haskell to Lambda Building Haskell locally and deploying to different machiens is notoriously tricky; you often end up depending on a shared library that is not available on the other platform. The aws-lambda-haskell-runtime package, and similar projects like serverless-haskell, solve this using stack and Docker two technologies that are probably great, but I never warmed up to them. So instead adding layers and complexities, can I solve this instead my making things simpler? If I compiler my bootstrap into a static Linux binary, it should run on any Linux, including Amazon Linux. Unfortunately, building Haskell programs statically is also notoriously tricky. But it is made much simpler by the work of Niklas Hamb chen and others in the context of the Nix package manager, coordinated in the static-haskell-nix project. The promise here is that once you have set up building your project with Nix, then getting a static version is just one flag away. The support is not completely upstreamed into nixpkgs proper yet, but their repository has a nix file that contains a nixpkgs set with their patches: 
let pkgs = (import (sources.nixpkgs-static + "/survey/default.nix")  ).pkgs; in
This, plus a fairly standard nix setup to build the package, yields what I was hoping for: 
$ nix-build -A kaleidogen
/nix/store/ppwyq4d964ahd6k56wsklh93vzw07ln0-kaleidogen-0.1.0.0
$ file result/bin/kaleidogen-amazon-lambda
result/bin/kaleidogen-amazon-lambda: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, stripped
$ ls -sh result/bin/kaleidogen-amazon-lambda
6,7M result/bin/kaleidogen-amazon-lambda
If we put this file, named bootstrap, into a zip file and upload it to Amazon Lambda, then it just works! Creating the zip file is easily scripted using nix: 
  function-zip = pkgs.runCommandNoCC "kaleidogen-lambda"  
    buildInputs = [ pkgs.zip ];
    ''
    mkdir -p $out
    cp $ kaleidogen /bin/kaleidogen-amazon-lambda bootstrap
    zip $out/function.zip bootstrap
  '';
So to upload this, I use this one-liner (line-wrapped for your convenience): 
nix-build -A function-zip &&
aws lambda update-function-code --function-name kaleidogen \
  --zip-file fileb://result/function.zip
Thanks to how Nix pins all dependencies, I am fairly confident that I can return to this project in 4 months and still be able to build it. Of course, I want continuous integration and deployment. So I build the project with GitHub Actions, using a cachix nix cache to significantly speed up the build, and auto-deploy to Lambda using aws-lambda-deploy; see my workflow file for details.

The Telegram part The above allows me to run basically any stateless service, and a Telegram bot is nothing else: When configured to act as a WebHook, Telegram will send a request with a message to our Lambda function, where we can react on it. The telegram-api package provides bindigs for the Telegram Bot API (although I had to use the repository version, as the version on Hackage has some bitrot). Slightly simplified I can write a handler for an Update: 
handleUpdate :: Update -> TelegramClient ()
handleUpdate Update  message = Just m   = do
  let c = ChatId (chat_id (chat m))
  liftIO $ printf "message from %s: %s\n" (maybe "?" user_first_name (from m)) (maybe "" T.unpack (text m))
  if "/start"  T.isPrefixOf  fromMaybe "" (text m)
  then do
    rm <- sendMessageM $ sendMessageRequest c "Hi! I am @KaleidogenBot.  "
    return ()
  else do
    m1 <- sendMessageM $ sendMessageRequest c "One moment "
    withPNGFile  $ \pngFN -> do
      m2 <- uploadPhotoM $ uploadPhotoRequest c
        (FileUpload (Just "image/png") (FileUploadFile pngFN))
      return ()
handleUpdate _ u =
  liftIO $ putStrLn $ "Unhandled message: " ++ show u
and call this from the handler that I wrote above: 
     
      path == "/telegram" =
      case eitherDecode (LBS.fromStrict (T.encodeUtf8 (fromMaybe "" body))) of
        Left err ->  
        Right update -> do
          runTelegramClient token manager $ handleUpdate Nothing update
          pure $ Right Response
              statusCode = 200
            , headers = object [ "Content-Type" .= ("text/plain" :: String) ]
            , isBase64Encoded = False
            , body = "Done"
Note that the Lambda code receives the request as JSON data structure with a body that contains the original HTTP request body. Which, in this case, is itself JSON, so we have to decode that. All that is left to do is to tell Telegram where this code lives: 
curl --request POST \
  --url https://api.telegram.org/bot<token>/setWebhook
  --header 'content-type: application/json'
  --data ' "url": "https://api.kaleidogen.nomeata.de/telegram" '
As a little add on, I also created a Telegram game for Kaleidogen. A Telegram game is nothing but a webpage that runs inside Telegram, so it wasn t much work to wrap the Web version of Kaleidogen that way, but the resulting Telegram game (which you can access via https://core.telegram.org/bots/games) still looks pretty neat.

No /dev/dri/renderD128 I am mostly happy with this setup: My game is now available to more people in more ways. I don t have to maintain any infrastructure. When nobody is using this bot no resources are wasted, and the costs of the service are neglectible -- this is unlikely to go beyond the free tier, and even if it would, the cost per generated image is roughly USD 0.000021. There is one slight disappointment, though. What I find most intersting about Kaleidogen from a technical point of view is that when you play it in the browser, the images are not generated by my code. Instead, my code creates a WebGL shader program on the fly, and that program generates the image on your graphics card. I even managed to make the GL rendering code work headlessly, i.e. from a command line program, using EGL and libgbm and a helper written in C. But it needs access to a graphics card via /dev/dri/renderD128. Amazon does not provide that to Lambda code, and neither do the other big Function-as-a-service providers. So I had to swallow my pride and reimplement the rendering in pure Haskell. So if you think the bot is kinda slow, then that s why. Despite properly optimizing the pure implementation (the inner loop does not do allocations and deals only with unboxed Double# values), the GL shader version is still three times as fast. Maybe in a few years GPU access will be so ubiquitous that it s even on Amazon Lambda; then I can easily use that.

Russ Allbery: Review: Thick

Review: Thick, by Tressie McMillan Cottom
Publisher: The New Press
Copyright: 2019
ISBN: 1-62097-437-1
Format: Kindle
Pages: 247
Tressie McMillan Cottom is an associate professor of sociology at Virginia Commonwealth University. I first became aware of her via retweets and recommendations from other people I follow on Twitter, and she is indeed one of the best writers on that site. Thick: And Other Essays is an essay collection focused primarily on how American culture treats black women. I will be honest here, in part because I think much of the regular audience for my book reviews is similar to me (white, well-off from working in tech, and leftist but privileged) and therefore may identify with my experience. This is the sort of book that I always want to read and then struggle to start because I find it intimidating. It received a huge amount of praise on release, including being named as a finalist for the National Book Award, and that praise focused on its incisiveness, its truth-telling, and its depth and complexity. Complex and incisive books about racism are often hard for me to read; they're painful, depressing, and infuriating, and I have to fight my tendency to come away from them feeling more cynical and despairing. (Despite loving his essays, I'm still procrastinating reading Ta-Nehisi Coates's books.) I want to learn and understand but am not good at doing anything with the information, so this reading can feel like homework. If that's also your reaction, read this book. I regret having waited as long as I did. Thick is still, at times, painful, depressing, and infuriating. It's also brilliantly written in a way that makes the knowledge being conveyed easier to absorb. Rather than a relentless onslaught of bearing witness (for which, I should stress, there is an important place), it is a scalpel. Each essay lays open the heart of a subject in a few deft strokes, points out important features that the reader has previously missed, and then steps aside, leaving you alone with your thoughts to come to terms with what you've just learned. I needed this book to be an essay collection, with each thought just long enough to have an impact and not so long that I became numb. It's the type of collection that demands a pause at the end of each essay, a moment of mental readjustment, and perhaps a paging back through the essay again to remember the sharpest points. The essays often start with seeds of the personal, drawing directly on McMillan Cottom's own life to wrap context around their point. In the first essay, "Thick," she uses advice given her younger self against writing too many first-person essays to talk about the writing form, its critics, and how the backlash against it has become part of systematic discrimination because black women are not allowed to write any other sort of authoritative essay. She then draws a distinction between her own writing and personal essays, not because she thinks less of that genre but because that genre does not work for her as a writer. The essays in Thick do this repeatedly. They appear to head in one direction, then deepen and shift with the added context of precise sociological analysis, defying predictability and reaching a more interesting conclusion than the reader had expected. And, despite those shifts, McMillan Cottom never lost me in a turn. This is a book that is not only comfortable with complexity and nuance, but helps the reader become comfortable with that complexity as well. The second essay, "In the Name of Beauty," is perhaps my favorite of the book. Its spark was backlash against an essay McMillan Cottom wrote about Miley Cyrus, but the topic of the essay wasn't what sparked the backlash.
What many black women were angry about was how I located myself in what I'd written. I said, blithely as a matter of observable fact, that I am unattractive. Because I am unattractive, the argument went, I have a particular kind of experience of beauty, race, racism, and interacting with what we might call the white gaze. I thought nothing of it at the time I was writing it, which is unusual. I can usually pinpoint what I have said, written, or done that will piss people off and which people will be pissed off. I missed this one entirely.
What follows is one of the best essays on the social construction of beauty I've ever read. It barely pauses at the typical discussion of unrealistic beauty standards as a feminist issue, instead diving directly into beauty as whiteness, distinguishing between beauty standards that change with generations and the more lasting rules that instead police the bounds between white and not white. McMillan Cottom then goes on to explain how beauty is a form of capital, a poor and problematic one but nonetheless one of the few forms of capital women have access to, and therefore why black women have fought to be included in beauty despite all of the problems with judging people by beauty standards. And the essay deepens from there into a trenchant critique of both capitalism and white feminism that is both precise and illuminating.
When I say that I am unattractive or ugly, I am not internalizing the dominant culture's assessment of me. I am naming what has been done to me. And signaling who did it. I am glad that doing so unsettles folks, including the many white women who wrote to me with impassioned cases for how beautiful I am. They offered me neoliberal self-help nonsense that borders on the religious. They need me to believe beauty is both achievable and individual, because the alternative makes them vulnerable.
I could go on. Every essay in this book deserves similar attention. I want to quote from all of them. These essays are about racism, feminism, capitalism, and economics, all at the same time. They're about power, and how it functions in society, and what it does to people. There is an essay about Obama that contains the most concise explanation for his appeal to white voters that I've read. There is a fascinating essay about the difference between ethnic black and black-black in U.S. culture. There is so much more.
We do not share much in the U.S. culture of individualism except our delusions about meritocracy. God help my people, but I can talk to hundreds of black folks who have been systematically separated from their money, citizenship, and personhood and hear at least eighty stories about how no one is to blame but themselves. That is not about black people being black but about people being American. That is what we do. If my work is about anything it is about making plain precisely how prestige, money, and power structure our so-called democratic institutions so that most of us will always fail.
I, like many other people in my profession, was always more comfortable with the technical and scientific classes in college. I liked math and equations and rules, dreaded essay courses, and struggled to engage with the mandatory humanities courses. Something that I'm still learning, two decades later, is the extent to which this was because the humanities are harder work than the sciences and I wasn't yet up to the challenge of learning them properly. The problems are messier and more fluid. The context required is broader. It's harder to be clear and precise. And disciplines like sociology deal with our everyday lived experience, which means that we all think we're entitled to an opinion. Books like this, which can offer me a hand up and a grounding in the intellectual rigor while simultaneously being engaging and easy to read, are a treasure. They help me fill in the gaps in my education and help me recognize and appreciate the depth of thought in disciplines that don't come as naturally to me. This book was homework, but the good kind, the kind that exposes gaps in my understanding, introduces topics I hadn't considered, and makes the time fly until I come up for air, awed and thinking hard. Highly recommended. Rating: 9 out of 10

3 April 2020

Dirk Eddelbuettel: RcppSimdJson 0.0.4: Even Faster Upstream!

A new (upstream) simdjson release was announced by Daniel Lemire earlier this week, and my Twitter mentions have been running red-hot ever since as he was kind enough to tag me. Do look at that blog post, there is some impressive work in there. We wrapped up the (still very simple) rcppsimdjson around it last night and shipped it this morning. RcppSimdJson wraps the fantastic and genuinely impressive simdjson library by Daniel Lemire. Via some very clever algorithmic engineering to obtain largely branch-free code, coupled with modern C++ and newer compiler instructions, it results in parsing gigabytes of JSON parsed per second which is quite mindboggling. For illustration, I highly recommend the video of the recent talk by Daniel Lemire at QCon (which was also voted best talk). The best-case performance is faster than CPU speed as use of parallel SIMD instructions and careful branch avoidance can lead to less than one cpu cycle use per byte parsed. This release brings upstream 0.3 (and 0.3.1) plus a minor tweak (also shipped back upstream). Our full NEWS entry follows.

Changes in version 0.0.4 (2020-04-03)
  • Upgraded to new upstream releases 0.3 and 0.3.1 (Dirk in #9 closing #8)
  • Updated example validateJSON to API changes.

But because Daniel is such a fantastic upstream developer to collaborate with, he even filed a full feature-request maybe you can consider upgrading as issue #8 at our repo containing the fully detailed list of changes. As it is so impressive I will simple quote the upper half of just the major changes:

Highlights
  • Multi-Document Parsing: Read a bundle of JSON documents (ndjson) 2-4x faster than doing it individually. API docs / Design Details
  • Simplified API: The API has been completely revamped for ease of use, including a new JSON navigation API and fluent support for error code and exception styles of error handling with a single API. Docs
  • Exact Float Parsing: Now simdjson parses floats flawlessly without any performance loss (https://github.com/simdjson/simdjson/pull/558). Blog Post
  • Even Faster: The fastest parser got faster! With a shiny new UTF-8 validator and meticulously refactored SIMD core, simdjson 0.3 is 15% faster than before, running at 2.5 GB/s (where 0.2 ran at 2.2 GB/s).

For questions, suggestions, or issues please use the issue tracker at the GitHub repo. Courtesy of CRANberries, there is also a diffstat report for this release. If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

2 April 2020

Sven Hoexter: New TLDs and Automatic link detection was a bad idea

Update: Seems this is a Firefox specific bug in the Slack Webapplication, it works in Chrome and the Slack Electron Application as it should. Tested with Firefox ESR on Debian/buster and Firefox 74 on OS X. Ah I like it that we now have so many TLDs and matching on those seems to go bad more often now. Last occassion is Slack (which I think is a pile of shit written by morons, but that is a different story) which somehow does not properly match on .co domains. Leading to this auto linking: nsswitch.conf link Now I'm not sure if someone enocountered the same issue, or people just registered random domains just because they could. I found registrations for I've a few more .conf files in /etc which could be interesting in an IT environment, but for the sake of playing with it I registered nsswitch.co at godaddy. I do not want to endorse them in anyway, but for the first year it's only 13.08EUR right now, which is okay to pay for a stupid demo. So if you feel like it, you can probably register something stupid for yourself to play around with. I do not intent to renew this domain next year, so be aware of what happens then with the next owner.

Mike Gabriel: Q: RoamingProfiles under GNU/Linux? What's your Best Practice?

This post is an open question to the wide range of GNU/Linux site admins out there. Possibly some of you have the joy of maintaining GNU/Linux also on user endpoint devices (i.e. user workstations, user notebooks, etc.), not only on corporate servers. TL;DR; In the context of a customer project, I am researching ways of mimicking (or inventing anew) a feature well known (and sometimes also well hated) from the MS Windows world: Roaming User Profiles. If anyone does have any input on that, please contact me (OFTC/Freenode IRC, Telegram, email). I am curious what your solution may be. The Use Case Scenario In my use case, all user machines shall be mobile (notebooks, convertibles, etc). The machines maybe on-site most of the time, but they need offline capabilities so that the users can transparently move off-site and continue their work. At the same time, a copy of the home directory (or the home directory itself) shall be stored on some backend fileservers (for central backups as well as for providing the possibility to the user to login to another machine and be up-and-running +/- out-of-the-box). The Vision Initial Login Ideally, I'd like to have a low level file system feature for this that handles it all. On corporate user logon (which must take place on-site and uses some LDAP database as backend), the user credentials get cached locally (and get re-mapped and re-cached with every on-site login later on), and the home directory gets mounted from a remote server at first. Shortly after having logged in everything in the user's home gets sync'ed to a local cache in the background without the user noticing. At the end of the sync a GUI user notification would be nice, e.g. like "All user data has been cached locally, you are good to go and leave off-site now with this machine." Moving Off-Site A day later, the user may be travelling or such, the user logs into the machine again, the machine senses being offline or on some alien (not corporate) network, but the user can just continue their work, all in local cache. Several days later, the same user with the same machine returns back to office, logs into the machine again, and immediately after login, all cached data gets synced back to the user's server filespace. Possible Conflict Policies Now there might be cases where the user has been working locally for a while and all the profile data received slight changes. The user might have had the possibility to log into other corporate servers from the alien network he*she is on and with that login, some user profile files probably will have gotten changed. Regarding client-server sync policies, one could now enforce a client-always-wins policy that leads to changes being dropped server-side once the user's mobile workstation returns back on-site. One could also set up a bi-directional sync policy for normal data files, but a client-always-wins policy for configuration files (.files and .folders). Etc.pp. Request for Feedback and Comments I could go on further and further with making up edges and corner cases of all this. We had a little discussion on this some days ago on the #debian-devel IRC channel already. Thanks to all contributors to that discussion. And again, if you have solved the above riddle on your site and are corporate-wise allowed to share the concept, I'd be happy about your feedback. Plese get in touch! light+love
Mike (aka sunweaver on the Fediverse and in Debian)

1 April 2020

Joey Hess: DIN distractions

My offgrid house has an industrial automation panel. A row of electrical devices, mounted on a metal rail. Many wires neatly extend from it above and below, disappearing into wire gutters. I started building this in February, before covid-19 was impacting us here, when lots of mail orders were no big problem, and getting an unusual 3D-printed DIN rail bracket for a SSD was just a couple clicks. I finished a month later, deep into social isolation and quarentine, scrounging around the house for scrap wire, scavenging screws from unused stuff and cutting them to size, and hoping I would not end up in a "need just one more part that I can't get" situation. It got rather elaborate, and working on it was often a welcome distraction from the news when I couldn't concentrate on my usual work. I'm posting this now because people sometimes tell me they like hearing about my offfgrid stuff, and perhaps you could use a distraction too. The panel has my house's computer on it, as well as both AC and DC power distribution, breakers, and switching. Since the house is offgrid, the panel is designed to let every non-essential power drain be turned off, from my offgrid fridge to the 20 terabytes of offline storage to the inverter and satellite dish, the spring pump for my gravity flow water system, and even the power outlet by the kitchen sink. Saving power is part of why I'm using old-school relays and stuff and not IOT devices, the other reason is of course: IOT devices are horrible dystopian e-waste. I'm taking the utopian Star Trek approach, where I can command "full power to the vacuum cleaner!" Two circuit boards, connected by numerous ribbon cables, and clearly hand-soldered. The smaller board is suspended above the larger. An electrical schematic, of moderate complexity. At the core of the panel, next to the cubietruck arm board, is a custom IO daughterboard. Designed and built by hand to fit into a DIN mount case, it uses every GPIO pin on the cubietruck's main GPIO header. Making this board took 40+ hours, and was about half the project. It got pretty tight in there. This was my first foray into DIN rail mount, and it really is industrial lego -- a whole universe of parts that all fit together and are immensely flexible. Often priced more than seems reasonable for a little bit of plastic and metal, until you look at the spec sheets and the ratings. (Total cost for my panel was $400.) It's odd that it's not more used outside its niche -- I came of age in the Bay Area, surrounded by rack mount equipment, but no DIN mount equipment. Hacking the hardware in a rack is unusual, but DIN invites hacking. Admittedly, this is a second system kind of project, replacing some unsightly shelves full of gear and wires everywhere with something kind of overdone. But should be worth it in the long run as new gear gets clipped into place and it evolves for changing needs. Also, wire gutters, where have you been all my life? A cramped utility room with an entire wall covered with electronic gear, including the DIN rail, which is surrounded by wire gutters Detail of a wire gutter with the cover removed. Numerous large and small wires run along it and exit here and there. Finally, if you'd like to know what everything on the DIN rail is, from left to right: Ground block, 24v DC disconnect, fridge GFI, spare GFI, USB hub switch, computer switch, +24v block, -24v block, IO daughterboard, 1tb SSD, arm board, modem, 3 USB hubs, 5 relays, AC hot block, AC neutral block, DC-DC power converters, humidity sensor. Full width of DIN rail.

Next.

Previous.