I had a twitter fight few days back with a gentleman and the article is a result of that fight. Sadly, I do not know the name of the gentleman as he goes via a psuedo name and then again I ve not taken permission from him to quote him in either way. So I will just state the observations I was able to make from the conversations we had. As people who read this blog regularly would know, I am and have been against Railway Privatization which is happening in India. And will be sharing some of the case studies from other countries as to how it panned out for them.
UK Railways
How Privatization Fails : Railways
The Above video is by a gentleman called Shaun who basically shared that privatization as far as UK is concerned is nothing but monopolies and while there are complex reasons for the same, the design of the Railways is such that it will always be a monopoly structure. At the most what you can do is have several monopolies but that is all that can happen. The idea of competition just cannot happen. Even the idea that subsidies will be less or/and trains will run on time is far from fact. Both of thesefacts have been checked and found to be truthful by fullfact.org. It is and argued that UK is small and perhaps it doesn t have the right conditions. It is probably true but still we do deserve to have a glance at the UK railway map.
UK railway map with operators
The above map is copyrighted to Map Marketing where you could see it today . As can be seen above most companies had their own specified areas. Now if you had looked at the facts then you would have seen that UK fares have been higher. In fact, an oldish article from Metro (a UK publication) shares the same. In fact, UK nationalized its railways effectively as many large rail operators were running in red. Even Scotland is set to nationalised back in March 2022. Remember this is a country which hasn t seen inflation go upwards of 5% in nearly a decade. The only outlier was 2011 where they indeed breached the 5% mark. So from this, what we see is Private Gains and Private Gains Public Losses perhaps seem fit. But then maybe we didn t use the right example. Perhaps Japan would be better. They have bullet trains while UK is still thinking about it. (HS2).
Japanese Railway
Below is the map of Japanese Railway
Railway map of Japan with private ownership courtesy Wikimedia commons
Japan started privatizing its railway in 1987 and to date it has not been fully privatized. And on top of it, amount as much as 24 trillion of the long-term JNR debt was shouldered by the government at the expense of taxpayers of Japan while also reducing almost 1/4th of it employees. To add to it, while some parts of Japanese Railways did make profits, many of them made profits by doing large-scale non-railway business mostly real estate of land adjacent to railway stations. In many cases, it seems this went all the way up to 60% of the revenue. The most profitable has been the Shinkansen though. And while it has been profitable, it has not been without safety scandals over the years, the biggest in recent years was the 2005 Amagasaki derailment. What was interesting to me was the Aftermath, while the Wikipedia page doesn t share much, I had read at the time and probably could be found how a lot of ordinary people stood up to the companies in a country where it is a known fact that most companies are owned by the Yakuza. And this is a country where people are loyal to their corporation or company no matter what. It is a strange culture to west and also here in India where people change jobs on drop of hat, although nowadays we have record unemployment. So perhaps Japan too does not meet our standard as it doesn t do competition with each other but each is a set monopoly in those regions. Also how much subsidy is there or not is not really transparent.
U.S. Railways
Last, but not the least I share the U.S. Railway map. This is provided by A Mr. Tom Alison on reddit on channel maporn. As the thread itself is archived and I do not know the gentleman concerned, nor have taken permission for the map, hence sharing the compressed version
U.S. Railway lines with the different owners
Now the U.S. Railways is and has always been peculiar as unlike the above two the U.S. has always been more of a freight network. Probably, much of it has to do that in the 1960 s when oil was cheap, the U.S. made zillions of roadways and romanticized the road trip and has been doing it ever since. Also the creation of low-cost airlines definitely didn t help the railways to have more passenger services, in fact the opposite.
There are and have been smaller services and attempts of privatization in both New Zealand and Australia and both have been failures. Please see papers in that regard. My simple point is this, as can be seen above, there have been various attempts at privatization of railways and most of them have been a mixed bag. The only one which comes close to what we think as good is Japanese but that also used a lot of public debt which we don t know what will happen on next. Also for higher-speed train services like a bullet train or whatever, you need to direct, no hair pen bends. In fact, a good talk on the topic is the TBD podcast which while it talks about hyperloop, the same questions is and would be asked if were to do in India. Another thing to be kept in mind is that the Japanese have been exceptional builders and this is because they have been forced to. They live in a seismically active zone which made Fukushima disaster a reality but at the same time, their buildings are earthquake-resistant.
Standard Disclaimer The above is a simplified version of things. I could have added in financial accounts but that again has no set pattern. For e.g. some Railways use accrual, some use cash and some use hybrid. I could have also shared in either the guage or electrification but all have slightly different standards, although uniguage is something that all Railways aspire for and electrification is again something that all Railways want although in many cases it just isn t economically feasible.
Indian Railways
Indian Railways itself recently made the move from Cash to Accrual couple of years back. In-between for a couple of years, it was hybrid. The sad part is and was you can now never measure against past performance in the old way because it is so different. Hence, whether the Railways will be making a loss or a profit, we would come to know only much later. Also, most accountants don t know the new system well, so it is gonna take more time, how much unknown. Sadly, what GOI did a few years back is merge the Railway budget into the Union Budget. Of course, the excuse they gave is too many pressures of new trains, while the truth is, by doing this, they decreased transparency about the whole thing. For e.g. for the last few years, the only state which had significant work being done is in U.P. (Uttar Pradesh) and a bit in Goa, although that is has been protested time and again. I being from the neighborly state of Maharashtra , and have been there several times. Now it does feels all like a dream, going to Goa :(.
Covid news
Now before I jump on the news, I should share the movie Virus (2019) which was made by the talented Aashiq Abu. Even though, am not a Malayalee, I still have enjoyed many of his movies simply because he is a terrific director and Malayalam movies, at least most of them have English subtitles and lot of original content.. Interestingly, unlike the first couple of times when I saw it a couple of years back. The first time I saw it, I couldn t sleep a wink for a week. Even the next time, it was heavy. I had shared the movie with mum, and even she couldn t see it in one go. It is and was that powerful Now maybe because we are headlong in the pandemic, and the madness is all around us. There are two terms that helped me though understand a great deal of what is happening in the movie, the first term was altered sensorium which has been defined here. The other is saturation or to be more precise oxygen saturation . This term has also entered the Indian twitter lexicon quite a bit as India has started running out of oxygen. Just today Delhi High Court did an emergency hearing on the subject late at night. Although there is much to share about the mismanagement of the center, the best piece on the subject has been by Miss Priya Ramani. Yup, the same lady who has won against M.J. Akbar and this is when Mr. Akbar had 100 lawyers for this specific case. It would be interesting to see what happens ahead.
There are however few things even she forgot in her piece, For e.g. reverse migration i.e. from urban to rural migration started again. Twoarticles from different entities sharing a similar outlook.Sadly, the right have no empathy or feeling for either the poor or the sick. Even the labor minister Santosh Gangwar s statement that around 1.04 crores were the only people who walked back home. While there is not much data, however some work/research has been done on migration to cites that the number could be easily 10 times as much. And this was in the lockdown of last year. This year, again the same issue has re-surfaced and migrants learning lessons started leaving cities. And I m ashamed to say I think they are doing the right thing. Most State Governments have not learned lessons nor have they done any work to earn the trust of migrants. This is true of almost all state Governments. Last year, just before the lockdown was announced, me and my friend spent almost 30k getting a cab all the way from Chennai to Pune, how much we paid for the cab, how much we bribed the various people just so we could cross the state borders to return home to our anxious families. Thankfully, unlike the migrants, we were better off although we did make a loss. I probably wouldn t be alive if I were in their situation as many didn t. That number is still in the air undocumented deaths Vaccine issues
Currently, though the issue has been the Vaccine and the pricing of the same. A good article to get a summation of the issues outlined has been shared on Economist. Another article that goes to the heart of the issue is at scroll. To buttress the argument, the SII chairman had shared this few weeks back
Adar Poonawala talking to Vishnu Som on Left, right center, 7th April 2021.
So, a licensee manufacturer wants to make super-profits during the pandemic. And now, as shared above they can very easily do it. Even the quotes given to nearby countries is smaller than the quotes given to Indian states
Prices of AstraZeneca among various states and countries.
The situation around beds, vaccines, oxygen, anything is so dire that people could go to any lengths to save their loved ones. Even if they know if a certain medicine doesn t work. For e.g. Remdesivir, 5 WHO trials have concluded that it doesn t increase mortality. Heck, even AIIMS chief said the same. But both doctors and relatives desperation to cling on hope has made Remdesivir as a black market drug with unoffical prices hovering anywhere between INR 14k/- to INR30k/- per vial. One of the executives of a top firm was also arrested in Gujarat. In Maharashtra, the opposition M.P. came to the rescue of the officials of Bruick pharms in Mumbai.
Sadly, this strange affliction to the party in the center is also there in my extended family. At one end, they will heap praise on Mr. Modi, at the same time they can t get wait to get fast out of India. Many of them have settled in horrors of horror Dubai, as it is the best place to do business, get international schools for the young ones at decent prices, cheaper or maybe a tad more than what they paid in Delhi or elsewhere. Being an Agarwal or a Gupta makes it easier to compartmentalize both things. Ease of doing business, 5 days flat to get a business registered, up and running. And the paranoia is still there. They won t talk on the phone about him because they are afraid they may say something which comes back to bite them. As far as their decision to migrate, can t really blame them. If I were 20-25 yeas younger and my mum were in a better shape than she is, we probably would have migrated as well, although would have preferred Europe than anywhere else.
Internet Freedom and Aarogya Setu App.
Internet Freedom had shared the chilling effects of the Aarogya Setu App. This had also been shared by FSCI in the past, and recently had their handle being banned on Twitter. This was also apparent in a legal bail order which the high court judge gave. While I won t go into the merits and demerits of the bail order, it is astounding for the judge to say that the accused, even though he would be on bail install an app. so he can be surveilled. And this is a high court judge, such a sad state of affairs. We seem to be putting up new lows every day when it comes to judicial jurisprudence. One interesting aspect of the whole case was shared by Aishwarya Iyer. She shared a story that she and her team worked on quint which raises questions on the quality of the work done by Delhi Police. This is of course, up to Delhi Police to ascertain the truth of the matter because unless and until they are able to tie in the PMO s office in for a leak or POTUS s office it hardly seems possible. For e.g. the dates when two heads of state can meet each other would be decided by the secretaries of the two. Once the date is known, it would be shared with the press while at the same time some sort of security apparatus would kick in place. It is incumbent, especially on the host to take as much care as he can of the guest. We all remember that World War 1 (the war to end all wars) started due to the murder of Archduke Ferdinand.
As nobody wants that, the best way is to make sure that a political murder doesn t happen on your watch. Now while I won t comment on what it would be, it would be safe to assume that it would be z+ security along with higher readiness. Especially if it as somebody as important as POTUS. Now, it would be quite a reach for Delhi Police to connect the two dates. They either will have to get creative with the dates or some other way. Otherwise, with practically no knowledge in the public domain, they can t work in limbo. In either case, I do hope the case comes up for hearing soon and we see what the Delhi Police says and contends in the High Court about the same. At the very least, it would be irritating for them to talk of the dates unless they can contend some mass conspiracy which involves the PMO (and would bring into question the constant vetting done by the Intelligence dept. of all those who work in PMO). And this whole case is to kind of shelter to the Delhi riots which happened in which majorly the Muslims died but their deaths lay unaccounted till date
Conclusion
In Conclusion, I would like to share a bit of humor because right now the atmosphere is humorless, both with authoritarian tendencies of the Central Govt. and the mass mismanagement of public health which they now have left to the state to do as they fit. The peice I am sharing is from arre, one of my goto sites whenever I feel low.
The dovecot version which will be released with bullseye seems to require some
subtle config adjustment if you
use ssl (ok that should be almost everyone)
and you would like to execute doveadm as a user, who can not read the ssl cert and
keys (quite likely).
I guess one of the common cases is executing doveadm pw e.g. if you use
postfixadmin. For myself
that manifested in the nginx error log, which I use in combination with php-fpm, as.
2021/04/19 20:22:59 [error] 307467#307467: *13 FastCGI sent in stderr: "PHP message:
Failed to read password from /usr/bin/doveadm pw ... stderr: doveconf: Fatal:
Error in configuration file /etc/dovecot/conf.d/10-ssl.conf line 12: ssl_cert:
Can't open file /etc/dovecot/private/dovecot.pem: Permission denied
You easily see the same error message if you just execute something like doveadm pw -p test123.
The workaround is to move your ssl configuration to a new file which is only readable by root,
and create a dummy one which disables ssl, and has a !include_try on the real one. Maybe
best explained by showing the modification:
I ve mostly had the preference of controlling my data rather than depend on someone else. That s one reason why I still believe email to be my most reliable medium for data storage, one that is not plagued/locked by a single entity. If I had the resources, I d prefer all digital data to be broken down to its simplest form for storage, like email format, and empower the user with it i.e. their data.
Yes, there are free services that are indirectly forced upon common users, and many of us get attracted to it. Many of us do not think that the information, which is shared for the free service in return, is of much importance. Which may be fair, depending on the individual, given that they get certain services without paying any direct dime.
New age communication
So first, we had email and usenet. As I mentioned above, email was designed with fine intentions. Intentions that make it stand even today, independently.
But not everything, back then, was that great either. For example, instant messaging was very closed and centralised then too. Things like: ICQ, MSN, Yahoo Messenger; all were centralized. I wonder if people still have access to their ICQ logs.
Not much has chagned in the current day either. We now have domination by: Facebook Messenger, Google Whatever the new marketing term they introdcue, WhatsApp, Telegram, Signal etc. To my knowledge, they are all centralized.
Over all this time, I m yet to see a product come up with good (business) intentions, to really empower the end user. In this information age, the most invaluable data is user activity. That s one data everyone is after. If you decline to share that bit of free data in exchange for the free services, mind you, that that free service like Facebook, Google, Instagram, WhatsApp, Truecaller, Twitter; none of it would come to you at all. Try it out.
So the reality is that while you may not be valuating the data you offer in exchange correctly, there s a lot that is reaped from it. But still, I think each user has (and should have) the freedom to opt in for these tech giants and give them their personal bit, for free services in return. That is a decent barter deal. And it is a choice that one if free to choose
Retaining my data
I m fond of keeping an archive folder in my mailbox. A folder that holds significant events in the form of an email usually, if documented. Over the years, I chose to resort to the email format because I felt it was more reliable in the longer term than any other formats.
The next best would be plain text.
In my lifetime, I have learnt a lot from the internet; so it is natural that my preference has been with it. Mailing Lists, IRCs, HOWTOs, Guides, Blog posts; all have helped. And over the years, I ve come across hundreds of such content that I d always like to preserve.
Now there are multiple ways to preserving data. Like, for example, big tech giants. In most usual cases, your data for your lifetime, should be fine with a tech giant. In some odd scenarios, you may be unlucky if you relied on a service provider that went bankrupt. But seriously, I think users should be fine if they host their data with Microsoft, Google etc; as long as they abide by their policies.
There s also the catch of alignment. As the user, you should ensure to align (and transition) with the product offerings of your service provider. Otherwise, what may look constant and always reliable, will vanish in the blink of an eye. I guess Google Plus would be a good example. There was some Google Feed service too. Maybe Google Photos in the near decade future, just like Google Picasa in the previous (or current) decade.
History what is
On the topic of retaining information, lets take a small drift. I still admire our ancestors. I don t know what went in their mind when they were documenting events in the form of scriptures, carvings, temples, churches, mosques etc; but one things for sure, they were able to leave a fine means of communication. They are all gone but a large number of those events are evident through the creations that they left. Some of those events have been strong enough that further rulers/invaders have had tough times trying to wipe it out from history. Remember, history is usually not the truth, but the statement to be believed by the teller. And the teller is usually the survivor, or the winner you may call.
But still, the information retention techniques were better.
I haven t visited, but admire whosoever built the Kailasa Temple, Ellora, without which, we d be made to believe what not by all invaders and rulers of the region. But the majestic standing of the temple, is a fine example of the history and the events that have occured in the past.
Ellora Temple - The majectic carving believed to be carved out of a single stone
Dominance has the power to rewrite history and unfortunately that s true and it has done its part. It is just that in a mere human s defined lifetime, it is not possible to witness the transtion from current to history, and say that I was there then and I m here now, and this is not the reality.
And if not dominance, there s always the other bit, hearsay. With it, you can always put anything up for dispute. Because there s no way one can go back in time and produce a fine evidence.
There s also a part about religion. Religion can be highly sentimental. And religion can be a solid way to get an agenda going. For example, in India - a country which today consitutionally is a secular country, there have been multiple attempts to discard the belief, that never ever did the thing called Ramayana exist. That the Rama Setu, nicely reworded as Adam s Bridge by who so ever, is a mere result of science. Now Rama, or Hanumana, or Ravana, or Valmiki, aren t going to come over and prove that that is true or false. So such subjects serve as a solid base to get an agenda going. And probably we ve even succeeded in proving and believing that there was never an event like Ramayana or the Mahabharata. Nor was there ever any empire other than the Moghul or the British Empire.
But yes, whosoever made the Ellora Temple or the manymanymore of such creations, did a fine job of making a dent for the future, to know of what the history possibly could also be.
Enough of the drift
So, in my opinion, having events documented is important. It d be nice to have skills documented too so that it can be passed over generations but that s a debatable topic. But events, I believe should be documented. And documented in the best possible ways so that its existence is not diminished.
A documentation in the form of certain carvings on a rock is far more better than links and posts shared on Facebook, Twitter, Reddit etc. For one, these are all corporate entities with vested interests and can pretext excuse in the name of compliance and conformance.
So, for the helpless state and generation I am in, I felt email was the best possible independent form of data retention in today s age. If I really had the resources, I d not rely on digital age. This age has no guarantee of retaining and recording information in any reliable manner. Instead, it is just mostly junk, which is manipulative and changeable, conditionally.
Email and RSS
So for my communication, I like to prefer emails over any other means. That doesn t mean I don t use the current trends. I do. But this blog is mostly about penning my desires. And desire be to have communication over email format.
Such is the case that for information useful over the internet, I crave to have it formatted in email for archival.
RSS feeds is my most common mode of keeping track of information I care about. Not all that I care for is available in RSS feeds but hey such is life. And adaptability is okay.
But my preference is still RSS.
So I use RSS feeds through a fine software called feed2imap. A software that fits my bill fairly well.
feed2imap is:
An rss feed news aggregator
Pulls and extracts news feeds in the form of an email
Can push the converted email over pop/imap
Can convert all image content to email mime attachment
In a gist, it makes the online content available to me offline in the most admired email format
In my mail box, in today s day, my preferred email client is Evolution. It does a good job of dealing with such emails (rss feed items). An image example of accessing the rrs feed item through it is below
RSS News Item through Evolution
The good part is that my actual data is always independent of such MUAs. Tomorrow, as technology - trends - economics evolve, something new would come as a replacement but my data would still be mine.
Mobile trends
Trends have shown. Data mobility is a commodity expectation now. As such, I wanted to have something fill in that gap for me. So that I could access my information - which I ve preferred in a format - easily in today s trendy options.
I tried multiple options on my current preferred platform of choice for mobiles, i.e. Android. Finally I came across Aqua Mail, which fits in most of my requirements.
Aqua Mail does
Connect to my laptop over imap
Can sync the requested folders
Can sync requested data for offlince accessibility
Can present the synchronized data in a quite flexible and customizable manner, to match my taste
Has a very extensible User Interface, allowing me to customize it to my taste
Pictures can do a better job of describing my english words.
Aqua Mail Message List Interface
Aqua Mail Message View Normal
Aqua Mail Message View Zoomed - Fit to Screen
All of this done with no dependence on network connectivity, post the sync. And all information is stored is best possible simple format.
Previously in George Orwell travel posts: Sutton Courtenay, Marrakesh, Hampstead, Paris, Southwold & The River Orwell.
Wallington is a small village in Hertfordshire, approximately fifty miles north of London and twenty-five miles from the outskirts of Cambridge. George Orwell lived at No. 2 Kits Lane, better known as 'The Stores', on a mostly-permanent basis from 1936 to 1940, but he would continue to journey up from London on occasional weekends until 1947.
His first reference to The Stores can be found in early 1936, where Orwell wrote from Lancashire during research for The Road to Wigan Pier to lament that he would very much like "to do some work again impossible, of course, in the [current] surroundings":
I am arranging to take a cottage at Wallington near Baldock in Herts, rather a pig in a poke because I have never seen it, but I am trusting the friends who have chosen it for me, and it is very cheap, only 7s. 6d. a week [ 20 in 2021].
For those not steeped in English colloquialisms, "a pig in a poke" is an item bought without seeing it in advance. In fact, one general insight that may be drawn from reading Orwell's extant correspondence is just how much he relied on a close network of friends, belying the lazy and hagiographical picture of an independent and solitary figure. (Still, even Orwell cultivated this image at times, such as in a patently autobiographical essay he wrote in 1946. But note the off-hand reference to varicose veins here, for they would shortly re-appear as a symbol of Winston's repressed humanity in Nineteen Eighty-Four.)
Nevertheless, the porcine reference in Orwell's idiom is particularly apt, given that he wrote the bulk of Animal Farm at The Stores his 1945 novella, of course, portraying a revolution betrayed by allegorical pigs. Orwell even drew inspiration for his 'fairy story' from Wallington itself, principally by naming the novel's farm 'Manor Farm', just as it is in the village. But the allusion to the purchase of goods is just as appropriate, as Orwell returned The Stores to its former status as the village shop, even going so far as to drill peepholes in a door to keep an Orwellian eye on the jars of sweets. (Unfortunately, we cannot complete a tidy circle of references, as whilst it is certainly Napoleon Animal Farm's substitute for Stalin who is quoted as describing Britain as "a nation of shopkeepers", it was actually the maraisardBertrand Bar re who first used the phrase).
"It isn't what you might call luxurious", he wrote in typical British understatement, but Orwell did warmly emote on his animals. He kept hens in Wallington (perhaps even inspiring the opening line of Animal Farm: "Mr Jones, of the Manor Farm, had locked the hen-houses for the night, but was too drunk to remember to shut the pop-holes.") and a photograph even survives of Orwell feeding his pet goat, Muriel. Orwell's goat was the eponymous inspiration for the white goat in Animal Farm, a decidedly under-analysed character who, to me, serves to represent an intelligentsia that is highly perceptive of the declining political climate but, seemingly content with merely observing it, does not offer any meaningful opposition. Muriel's aesthetic of resistance, particularly in her reporting on the changes made to the Seven Commandments of the farm, thus rehearses the well-meaning (yet functionally ineffective) affinity for 'fact checking' which proliferates today. But I digress.
There is a tendency to "read Orwell backwards", so I must point out that Orwell wrote several other works whilst at The Stores as well. This includes his Homage to Catalonia, his aforementioned The Road to Wigan Pier, not to mention countless indispensable reviews and essays as well. Indeed, another result of focusing exclusively on Orwell's last works is that we only encounter his ideas in their highly-refined forms, whilst in reality, it often took many years for concepts to fully mature we first see, for instance, the now-infamous idea of "2 + 2 = 5" in an essay written in 1939.
This is important to understand for two reasons. Although the ostentatiously austere Barnhill might have housed the physical labour of its writing, it is refreshing to reflect that the philosophical heavy-lifting of Nineteen Eighty-Four may have been performed in a relatively undistinguished North Hertfordshire village. But perhaps more importantly, it emphasises that Orwell was just a man, and that any of us is fully capable of equally significant insight, with to quote Christopher Hitchens "little except a battered typewriter and a certain resilience."
The red commemorative plaque not only limits Orwell's tenure to the time he was permanently in the village, it omits all reference to his first wife, Eileen O'Shaughnessy, whom he married in the village church in 1936.Wallington's Manor Farm, the inspiration for the farm in Animal Farm. The lower sign enjoins the public to inform the police "if you see anyone on the [church] roof acting suspiciously". Non-UK-residents may be surprised to learn about the systematic theft of lead.
First up, I am alive and well. I have been receiving calls from friends for quite sometime but now that I have become deaf, it is a pain and the hearing aids aren t all that useful. But moreover, where we have been finding ourselves each and every day sinking lower and lower feels absurd as to what to write and not write about India. Thankfully, I ran across this piece which does tell in far more detail than I ever could. The only interesting and somewhat positive news I had is from south of India otherwise sad days, especially for the poor. The saddest story is that this time Covid has reached alarming proportions in India and surprise, surprise this time the villain for many is my state of Maharashtra even though it hasn t received its share of GST proceeds for last two years and this was Kerala s perspective, different state, different party, different political ideology altogether.
Kerala Finance Minister Thomas Issac views on GST, October 22, 2020 Indian Express.
I briefly also share the death of somewhat liberal Film censorship in India unlike Italy which abolished film censorship altogether. I don t really want spend too much on how we have become No. 2 in Covid cases in the world and perhaps death also. Many people still believe in herd immunity but don t really know what it means. So without taking too much time and effort, bid adieu. May post when I m hopefully emotionally feeling better, stronger
I ve just setup the Yama LSM module on some of my Linux systems. Yama controls ptrace which is the debugging and tracing API for Unix systems. The aim is to prevent a compromised process from using ptrace to compromise other processes and cause more damage. In most cases a process which can ptrace another process which usually means having capability SYS_PTRACE (IE being root) or having the same UID as the target process can interfere with that process in other ways such as modifying it s configuration and data files. But even so I think it has the potential for making things more difficult for attackers without making the system more difficult to use.
If you put kernel.yama.ptrace_scope = 1 in sysctl.conf (or write 1 to /proc/sys/kernel/yama/ptrace_scope) then a user process can only trace it s child processes. This means that strace -p and gdb -p will fail when run as non-root but apart from that everything else will work. Generally strace -p (tracing the system calls of another process) is of most use to the sysadmin who can do it as root. The command gdb -p and variants of it are commonly used by developers so yama wouldn t be a good thing on a system that is primarily used for software development.
Another option is kernel.yama.ptrace_scope = 3 which means no-one can trace and it can t be disabled without a reboot. This could be a good option for production servers that have no need for software development. It wouldn t work well for a small server where the sysadmin needs to debug everything, but when dozens or hundreds of servers have their configuration rolled out via a provisioning tool this would be a good setting to include.
See Documentation/admin-guide/LSM/Yama.rst in the kernel source for the details.
When running with capability SYS_PTRACE (IE root shell) you can ptrace anything else and if necessary disable Yama by writing 0 to /proc/sys/kernel/yama/ptrace_scope .
I am enabling mode 1 on all my systems because I think it will make things harder for attackers while not making things more difficult for me.
Also note that SE Linux restricts SYS_PTRACE and also restricts cross-domain ptrace access, so the combination with Yama makes things extra difficult for an attacker.
Yama is enabled in the Debian kernels by default so it s very easy to setup for Debian users, just edit /etc/sysctl.d/whatever.conf and it will be enabled on boot.
I ve been watching the show Riverdale on Netflix recently. It s an interesting modern take on the Archie comics. Having watched Josie and the Pussycats in Outer Space when I was younger I was anticipating something aimed towards a similar audience. As solving mysteries and crimes was apparently a major theme of the show I anticipated something along similar lines to Scooby Doo, some suspense and some spooky things, but then a happy ending where criminals get arrested and no-one gets hurt or killed while the vast majority of people are nice. Instead the first episode has a teen being murdered and Ms Grundy being obsessed with 15yo boys and sleeping with Archie (who s supposed to be 15 but played by a 20yo actor).
Everyone in the show has some dark secret. The filming has a dark theme, the sky is usually overcast and it s generally gloomy. This is a significant contrast to Veronica Mars which has some similarities in having a young cast, a sassy female sleuth, and some similar plot elements. Veronica Mars has a bright theme and a significant comedy element in spite of dealing with some dark issues (murder, rape, child sex abuse, and more). But Riverdale is just dark. Anyone who watches this with their kids expecting something like Scooby Doo is in for a big surprise.
There are lots of interesting stylistic elements in the show. Lots of clothing and uniform designs that seem to date from the 1940 s. It seems like some alternate universe where kids have smartphones and laptops while dressing in the style of the 1940s. One thing that annoyed me was construction workers using tools like sledge-hammers instead of excavators. A society that has smart phones but no earth-moving equipment isn t plausible.
On the upside there is a racial mix in the show that more accurately reflects American society than the original Archie comics and homophobia is much less common than in most parts of our society. For both race issues and gay/lesbian issues the show treats them in an accurate way (portraying some bigotry) while the main characters aren t racist or homophobic.
I think it s generally an OK show and recommend it to people who want a dark show. It s a good show to watch while doing something on a laptop so you can check Wikipedia for the references to 1940s stuff (like when Bikinis were invented). I m half way through season 3 which isn t as good as the first 2, I don t know if it will get better later in the season or whether I should have stopped after season 2.
I don t usually review fiction, but the interesting aesthetics of the show made it deserve a review.
The Viability of Small Disks
Less than a year ago I wrote a blog post about storage trends [1]. My main point in that post was that disks smaller than 2TB weren t viable then and 2TB disks wouldn t be economically viable in the near future.
Now MSY has 2TB disks for $72 and 2TB SSD for $245, saving $173 if you get a hard drive (compared to saving $240 10 months ago). Given the difference in performance and noise 2TB hard drives won t be worth using for most applications nowadays.
NVMe vs SSD
Last year NVMe prices were very comparable for SSD prices, I was hoping that trend would continue and SSDs would go away. Now for sizes 1TB and smaller NVMe and SSD prices are very similar, but for 2TB the NVMe prices are twice that of SSD presumably partly due to poor demand for 2TB NVMe. There are also no NVMe devices larger than 2TB on sale at MSY (a store which caters to home stuff not special server equipment) but SSDs go up to 8TB.
It seems that NVMe is only really suitable for workstation storage and for cache etc on a server. So SATA SSDs will be around for a while.
Small Servers
There are a range of low end servers which support a limited number of disks. Dell has 2 disk servers and 4 disk servers. If one of those had 8TB SSDs you could have 8TB of RAID-1 or 24TB of RAID-Z storage in a low end server. That covers the vast majority of servers (small business or workgroup servers tend to have less than 8TB of storage).
Larger Servers
Anandtech has an article on Seagates roadmap to 120TB disks [2]. They currently sell 20TB disks using HAMR technology
Currently the biggest disks that MSY sells are 10TB for $395, which was also the biggest disk they were selling last year. Last year MSY only sold SSDs up to 2TB in size (larger ones were available from other companies at much higher prices), now they sell 8TB SSDs for $949 (4* capacity increase in less than a year). Seagate is planning 30TB disks for 2023, if SSDs continue to increase in capacity by 4* per year we could have 128TB SSDs in 2023. If you needed a server with 100TB of storage then having 2 or 3 SSDs in a RAID array would be much easier to manage and faster than 4*30TB disks in an array.
When you have a server with many disks you can expect to have more disk failures due to vibration. One time I built a server with 18 disks and took disks from 2 smaller servers that had 4 and 5 disks. The 9 disks which had been working reliably for years started having problems within weeks of running in the bigger server. This is one of the many reasons for paying extra for SSD storage.
Seagate is apparently planning 50TB disks for 2026 and 100TB disks for 2030. If that s the best they can do then SSD vendors should be able to sell larger products sooner at prices that are competitive. Matching hard drive prices is not required, getting to less than 4* the price should be enough for most customers.
The Anandtech article is worth reading, it mentions some interesting features that Seagate are developing such as having 2 actuators (which they call Mach.2) so the drive can access 2 different tracks at the same time. That can double the performance of a disk, but that doesn t change things much when SSDs are more than 100* faster. Presumably the Mach.2 disks will be SAS and incredibly expensive while providing significantly less performance than affordable SATA SSDs.
Computer Cases
In my last post I speculated on the appearance of smaller cases designed to not have DVD drives or 3.5 hard drives. Such cases still haven t appeared apart from special purpose machines like the NUC that were available last year.
It would be nice if we could get a new industry standard for smaller power supplies. Currently power supplies are expected to be almost 5 inches wide (due to the expectation of a 5.25 DVD drive mounted horizontally). We need some industry standards for smaller PCs that aren t like the NUC, the NUC is very nice, but most people who build their own PC need more space than that. I still think that planning on USB DVD drives is the right way to go. I ve got 4PCs in my home that are regularly used and CDs and DVDs are used so rarely that sharing a single DVD drive among all 4 wouldn t be a problem.
Conclusion
I m tempted to get a couple of 4TB SSDs for my home server which cost $487 each, it currently has 2*500G SSDs and 3*4TB disks. I would have to remove some unused files but that s probably not too hard to do as I have lots of old backups etc on there. Another possibility is to use 2*4TB SSDs for most stuff and 2*4TB disks for backups.
I m recommending that all my clients only use SSDs for their storage. I only have one client with enough storage that disks are the only option (100TB of storage) but they moved all the functions of that server to AWS and use S3 for the storage. Now I don t have any clients doing anything with storage that can t be done in a better way on SSD for a price difference that s easy for them to afford.
Affordable SSD also makes RAID-1 in workstations more viable. 2 disks in a PC is noisy if you have an office full of them and produces enough waste heat to be a reliability issue (most people don t cool their offices adequately on weekends). 2 SSDs in a PC is no problem at all. As 500G SSDs are available for $73 it s not a significant cost to install 2 of them in every PC in the office (more cost for my time than hardware). I generally won t recommend that hard drives be replaced with SSDs in systems that are working well. But if a machine runs out of space then replacing it with SSDs in a RAID-1 is a good choice.
Moore s law might cover SSDs, but it definitely doesn t cover hard drives. Hard drives have fallen way behind developments of most other parts of computers over the last 30 years, hopefully they will go away soon.
Host to Kanchenjunga, the world s third-highest mountain peak and the endangered Red Panda, Sikkim is a state in northeastern India. Nestled between Nepal, Tibet (China), Bhutan and West Bengal (India), the state offers a smorgasbord of cultures and cuisines. That said, it s hardly surprising that the old spice route meanders through western Sikkim, connecting Lhasa with the ports of Bengal. Although the latter could also be attributed to cardamom (kali elaichi), a perennial herb native to Sikkim, which the state is the second-largest producer of, globally. Lastly, having been to and lived in India, all my life, I can confidently say Sikkim is one of the cleanest & safest regions in India, making it ideal for first-time backpackers.
Brief History
17th century: The Kingdom of Sikkim is founded by the Namgyal dynasty and ruled by Buddhist priest-kings known as the Chogyal.
1890: Sikkim becomes a princely state of British India.
1947: Sikkim continues its protectorate status with the Union of India, post-Indian-independence.
1973: Anti-royalist riots take place in front of the Chogyal's palace, by Nepalis seeking greater representation.
1975: Referendum leads to the deposition of the monarchy and Sikkim joins India as its 22nd state.
Languages
Official: English, Nepali, Sikkimese/Bhotia and Lepcha
Though Hindi and Nepali share the same script (Devanagari), they are not mutually intelligible. Yet, most people in Sikkim can understand and speak Hindi.
Ethnicity
Nepalis: Migrated in large numbers (from Nepal) and soon became the dominant community
Bhutias: People of Tibetan origin. Major inhabitants in Northern Sikkim.
Lepchas: Original inhabitants of Sikkim
Food
Tibetan/Nepali dishes (mostly consumed during winter)
Thukpa: Noodle soup, rich in spices and vegetables. Usually contains some form of meat. Common variations: Thenthuk and Gyathuk
Momos: Steamed or fried dumplings, usually with a meat filling.
Saadheko: Spicy marinated chicken salad.
Gundruk Soup: A soup made from Gundruk, a fermented leafy green vegetable.
Sinki : A fermented radish tap-root product, traditionally consumed as a base for soup and as a pickle. Eerily similar to Kimchi.
While pork and beef are pretty common, finding vegetarian dishes is equally easy.
Staple: Dal-Bhat with Subzi. Rice is a lot more common than wheat (rice) possibly due to greater carb content and proximity to West Bengal, India s largest producer of Rice.
Good places to eat in Gangtok
Hamro Bhansa Ghar, Nimtho (Nepali)
Taste of Tibet
Dragon Wok (Chinese & Japanese)
Buddhism in Sikkim
Bayul Demojong (Sikkim), is the most sacred Land in the Himalayas as per the belief of the Northern Buddhists and various religious texts.
Sikkim was blessed by Guru Padmasambhava, the great Buddhist saint who visited Sikkim in the 8th century and consecrated the land.
However, Buddhism is said to have reached Sikkim only in the 17th century with the arrival of three Tibetan monks viz. Rigdzin Goedki Demthruchen, Mon Kathok Sonam Gyaltshen & Rigdzin Legden Je at Yuksom. Together, they established a Buddhist monastery.
In 1642 they crowned Phuntsog Namgyal as the first monarch of Sikkim and gave him the title of Chogyal, or Dharma Raja.
The faith became popular through its royal patronage and soon many villages had their own monastery.
Today Sikkim has over 200 monasteries.
Major monasteries
Rumtek Monastery, 20Km from Gangtok
Lingdum/Ranka Monastery, 17Km from Gangtok
Phodong Monastery, 28Km from Gangtok
Ralang Monastery, 10Km from Ravangla
Tsuklakhang Monastery, Royal Palace, Gangtok
Enchey Monastery, Gangtok
Tashiding Monastery, 35Km from Ravangla
Reaching Sikkim
Gangtok, being the capital, is easiest to reach amongst other regions, by public transport and shared cabs.
About 20 minutes from Siliguri and 4 hours from Gangtok.
Reserved cabs cost about INR 3000. Shared cabs from INR 350.
By Road:
NH10 connects Siliguri to Gangtok
If you can t find buses plying to Gangtok directly, reach Siliguri and then take a cab to Gangtok.
Sikkim Nationalised Transport Div. also runs hourly buses between Siliguri and Gangtok and daily buses on other common routes. They re cheaper than shared cabs.
Wizzride also operates shared cabs between Siliguri/Bagdogra/NJP, Gangtok and Darjeeling. They cost about the same as shared cabs but pack in half as many people in luxury cars (Innova, Xylo, etc.) and are hence more comfortable.
The easiest & most economical way to explore North Sikkim is the 3D/2N package offered by shared-cab drivers.
This includes food, permits, cab rides and accommodation (1N in Lachen and 1N in Lachung)
The accommodation on both nights are at homestays with bare necessities, so keep your hopes low.
In the spirit of sustainable tourism, you ll be asked to discard single-use plastic bottles, so please carry a bottle that you can refill along the way.
Zero Point and Gurdongmer Lake are snow-capped throughout the year
3D/2N Shared-cab Package Itinerary
Day 1
Gangtok (10am) - Chungthang - Lachung (stay)
Day 2
Pre-lunch : Lachung (6am) - Yumthang Valley [12,139ft] - Zero Point - Lachung [15,300ft]
Post-lunch : Lachung - Chungthang - Lachen (stay)
Day 3
Pre-lunch : Lachen (5am) - Kala Patthar - Gurdongmer Lake [16,910ft] - Lachen
Post-lunch : Lachen - Chungthang - Gangtok (7pm)
This itinerary is idealistic and depends on the level of snowfall.
Some drivers might switch up Day 2 and 3 itineraries by visiting Lachen and then Lachung, depending upon the weather.
Areas beyond Lachen & Lachung are heavily militarized since the Indo-China border is only a few miles away.
East Sikkim
Zuluk and Silk Route
Time needed: 2D/1N
Zuluk [9,400ft] is a small hamlet with an excellent view of the eastern Himalayan range including the Kanchenjunga.
Was once a transit point to the historic Silk Route from Tibet (Lhasa) to India (West Bengal).
The drive from Gangtok to Zuluk takes at least four hours. Hence, it makes sense to spend the night at a homestay and space out your trip to Zuluk
Tsomgo Lake and Nathula
Time Needed : 1D
A Protected Area Permit is required to visit these places, due to their proximity to the Chinese border
Located on the Indo-Tibetan border crossing of the Old Silk Route, it is one of the three open trading posts between India and China.
Plays a key role in the Sino-Indian Trade and also serves as an official Border Personnel Meeting(BPM) Point.
May get cordoned off by the Indian Army in event of heavy snowfall or for other security reasons.
West Sikkim
Time needed: 3N/1N
Hostels at Pelling : Mochilerro Ostillo
Itinerary
Day 1: Gangtok - Ravangla - Pelling
Leave Gangtok early, for Ravangla through the Temi Tea Estate route.
Spend some time at the tea garden and then visit Buddha Park at Ravangla
Head to Pelling from Ravangla
Day 2: Pelling sightseeing
Hire a cab and visit Skywalk, Pemayangtse Monastery, Rabdentse Ruins, Kecheopalri Lake, Kanchenjunga Falls.
Day 3: Pelling - Gangtok/Siliguri
Wake up early to catch a glimpse of Kanchenjunga at the Pelling Helipad around sunrise
Head back to Gangtok on a shared-cab
You could take a bus/taxi back to Siliguri if Pelling is your last stop.
Darjeeling
In my opinion, Darjeeling is lovely for a two-day detour on your way back to Bagdogra/Siliguri and not any longer (unless you re a Bengali couple on a honeymoon)
Once a part of Sikkim, Darjeeling was ceded to the East India Company after a series of wars, with Sikkim briefly receiving a grant from EIC for gifting Darjeeling to the latter
Post-independence, Darjeeling was merged with the state of West Bengal.
Reach Darjeeling by noon and check in to your Hostel. I stayed at Hideout.
Spend the evening visiting either a monastery (or the Batasia Loop), Nehru Road and Mall Road.
Grab dinner at Glenary whilst listening to live music.
Day 2:
Wake up early to catch the sunrise and a glimpse of Kanchenjunga at Tiger Hill. Since Tiger Hill is 10km from Darjeeling and requires a permit, book your taxi in advance.
Alternatively, if you don t want to get up at 4am or shell out INR1500 on the cab to Tiger Hill, walk to the Kanchenjunga View Point down Mall Road
Next, queue up outside Keventers for breakfast with a view in a century-old cafe
Get a cab at Gandhi Road and visit a tea garden (Happy Valley is the closest) and the Ropeway. I was lucky to meet 6 other backpackers at my hostel and we ended up pooling the cab at INR 200 per person, with INR 1400 being on the expensive side, but you could bargain.
Get lunch, buy some tea at Golden Tips, pack your bags and hop on a shared-cab back to Siliguri. It took us about 4hrs to reach Siliguri, with an hour to spare before my train.
If you ve still got time on your hands, then check out the Peace Pagoda and the Darjeeling Himalayan Railway (Toy Train). At INR 1500, I found the latter to be too expensive and skipped it.
Tips and hacks
Download offline maps, especially when you re exploring Northern Sikkim.
Food and booze are the cheapest in Gangtok. Stash up before heading to other regions.
In rural areas and some cafes, you may get to try Rhododendron Wine, made from Rhododendron arboreum a.k.a Gurans. Its production is a little hush-hush since the flower is considered holy and is also the National Flower of Nepal.
If you don t want to invest in a new jacket, boots or a pair of gloves, you can always rent them at nominal rates from your hotel or little stores around tourist sites.
Check the weather of a region before heading there. Low visibility and precipitation can quite literally dampen your experience.
Keep your itinerary flexible to accommodate for rest and impromptu plans.
Shops and restaurants close by 8pm in Sikkim and Darjeeling. Plan for the same.
Carry
a couple of extra pairs of socks (woollen, if possible)
a pair of slippers to wear indoors
a reusable water bottle
an umbrella
a power bank
a couple of tablets of Diamox. Helps deal with altitude sickness
extra clothes and wet bags since you may not get a chance to wash/dry your clothes
a few passport size photographs
Shared-cab hacks
Intercity rides can be exhausting. If you can afford it, pay for an additional seat.
Call shotgun on the drives beyond Lachen and Lachung. The views are breathtaking.
Return cabs tend to be cheaper (WB cabs travelling from SK and vice-versa)
Cost
My median daily expenditure (back when I went to Sikkim in early March 2021) was INR 1350.
This includes stay (bunk bed), food, wine and transit (shared cabs)
In my defence, I splurged on food, wine and extra seats in shared cabs, but if you re on a budget, you could easily get by on INR 1 - 1.2k per day.
For a 9-day trip, I ended up shelling out nearly INR 15k, including 2AC trains to & from Kolkata
Note : Summer (March to May) and Autumn (October to December) are peak seasons, and thereby more expensive to travel around.
Souvenirs and things you should buy
Buddhist souvenirs :
Colourful Prayer Flags (great for tying on bikes or behind car windshields)
Miniature Prayer/Mani Wheels
Lucky Charms, Pendants and Key Chains
Cham Dance masks and robes
Singing Bowls
Common symbols: Om mani padme hum, Ashtamangala, Zodiac signs
Following on from 2020 in Fiction: In 2020 I read a couple of collections
of short fiction from some of my favourite authors.
I started the year with Christopher Priest's Episodes. The stories within are
collected from throughout his long career, and vary in style and tone. Priest
wrote new little prologues and epilogues for each of the stories, explaining
the context in which they were written. I really enjoyed this additional view
into their construction.
By contrast, Adam Robert's Adam Robots presents the stories on their own
terms. Each of the stories is written in a different mode: one as golden-age
SF, another as a kind of Cyberpunk, for example, although they all blend or
confound sub-genres to some degree. I'm not clever enough to have decoded all
their secrets on a first read, and I would have appreciated some "Cliff's Notes on
any deeper meaning or intent.
Ted Chiang's Exhalation was up to the fantastic standard of his earlier
collection and had some extremely thoughtful explorations of philosophical
ideas. All the stories are strong but one stuck in my mind the longest:
Omphalos)
With my daughter I finished three of Terry Pratchett's short story collections
aimed at children: Dragon at Crumbling Castle; The Witch's Vacuum Cleaner
and The Time-Travelling Caveman. If you are a Pratchett fan and you've
overlooked these because they're aimed at children, take another look. The
quality varies, but there are some true gems in these. Several stories take
place in common settings, either the town of Blackbury, in Gritshire (and the
adjacent Even Moor), or the Welsh border-town of Llandanffwnfafegettupagogo.
The sad thing was knowing that once I'd finished them (and the fourth, Father
Christmas's Fake Beard) that was it: there will be no more.
8/31 of the "books" I read in 2020 were issues of Interzone. Counting them as
"books" for my annual reading goal has encouraged me to read full issues,
whereas before I would likely have only read a couple of stories from each
issue. Reading full issues has rekindled the enjoyment I got out of it when I
first discovered the magazine at the turn of the Century. I am starting to
recognise stories by authors that have written stories in other issues, as well
as common themes from the current era weaving their way into the work (Trump,
Brexit, etc.) No doubt the Pandemic will leave its mark on 2021's stories.
It all started with the big bang! We nearly lost 33 of 36 disks on a Proxmox/Ceph Cluster; this is the story of how we recovered them.
At the end of 2020, we eventually had a long outstanding maintenance window for taking care of system upgrades at a customer. During this maintenance window, which involved reboots of server systems, the involved Ceph cluster unexpectedly went into a critical state. What was planned to be a few hours of checklist work in the early evening turned out to be an emergency case; let s call it a nightmare (not only because it included a big part of the night). Since we have learned a few things from our post mortem and RCA, it s worth sharing those with others. But first things first, let s step back and clarify what we had to deal with.
The system and its upgrade
One part of the upgrade included 3 Debian servers (we re calling them server1, server2 and server3 here), running on Proxmox v5 + Debian/stretch with 12 Ceph OSDs each (65.45TB in total), a so-called Proxmox Hyper-Converged Ceph Cluster.
First, we went for upgrading the Proxmox v5/stretch system to Proxmox v6/buster, before updating Ceph Luminous v12.2.13 to the latest v14.2 release, supported by Proxmox v6/buster. The Proxmox upgrade included updating corosync from v2 to v3. As part of this upgrade, we had to apply some configuration changes, like adjust ring0 + ring1 address settings and add a mon_host configuration to the Ceph configuration.
During the first two servers reboots, we noticed configuration glitches. After fixing those, we went for a reboot of the third server as well. Then we noticed that several Ceph OSDs were unexpectedly down. The NTP service wasn t working as expected after the upgrade. The underlying issue is a race condition of ntp with systemd-timesyncd (see #889290). As a result, we had clock skew problems with Ceph, indicating that the Ceph monitors clocks aren t running in sync (which is essential for proper Ceph operation). We initially assumed that our Ceph OSD failure derived from this clock skew problem, so we took care of it. After yet another round of reboots, to ensure the systems are running all with identical and sane configurations and services, we noticed lots of failing OSDs. This time all but three OSDs (19, 21 and 22) were down:
% sudo ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 65.44138 root default
-2 21.81310 host server1
0 hdd 1.08989 osd.0 down 1.00000 1.00000
1 hdd 1.08989 osd.1 down 1.00000 1.00000
2 hdd 1.63539 osd.2 down 1.00000 1.00000
3 hdd 1.63539 osd.3 down 1.00000 1.00000
4 hdd 1.63539 osd.4 down 1.00000 1.00000
5 hdd 1.63539 osd.5 down 1.00000 1.00000
18 hdd 2.18279 osd.18 down 1.00000 1.00000
20 hdd 2.18179 osd.20 down 1.00000 1.00000
28 hdd 2.18179 osd.28 down 1.00000 1.00000
29 hdd 2.18179 osd.29 down 1.00000 1.00000
30 hdd 2.18179 osd.30 down 1.00000 1.00000
31 hdd 2.18179 osd.31 down 1.00000 1.00000
-4 21.81409 host server2
6 hdd 1.08989 osd.6 down 1.00000 1.00000
7 hdd 1.08989 osd.7 down 1.00000 1.00000
8 hdd 1.63539 osd.8 down 1.00000 1.00000
9 hdd 1.63539 osd.9 down 1.00000 1.00000
10 hdd 1.63539 osd.10 down 1.00000 1.00000
11 hdd 1.63539 osd.11 down 1.00000 1.00000
19 hdd 2.18179 osd.19 up 1.00000 1.00000
21 hdd 2.18279 osd.21 up 1.00000 1.00000
22 hdd 2.18279 osd.22 up 1.00000 1.00000
32 hdd 2.18179 osd.32 down 1.00000 1.00000
33 hdd 2.18179 osd.33 down 1.00000 1.00000
34 hdd 2.18179 osd.34 down 1.00000 1.00000
-3 21.81419 host server3
12 hdd 1.08989 osd.12 down 1.00000 1.00000
13 hdd 1.08989 osd.13 down 1.00000 1.00000
14 hdd 1.63539 osd.14 down 1.00000 1.00000
15 hdd 1.63539 osd.15 down 1.00000 1.00000
16 hdd 1.63539 osd.16 down 1.00000 1.00000
17 hdd 1.63539 osd.17 down 1.00000 1.00000
23 hdd 2.18190 osd.23 down 1.00000 1.00000
24 hdd 2.18279 osd.24 down 1.00000 1.00000
25 hdd 2.18279 osd.25 down 1.00000 1.00000
35 hdd 2.18179 osd.35 down 1.00000 1.00000
36 hdd 2.18179 osd.36 down 1.00000 1.00000
37 hdd 2.18179 osd.37 down 1.00000 1.00000
Our blood pressure increased slightly! Did we just lose all of our cluster? What happened, and how can we get all the other OSDs back?
We stumbled upon this beauty in our logs:
The same issue was present for the other failing OSDs. We hoped, that the data itself was still there, and only the mounting of the XFS partitions failed. The Ceph cluster was initially installed in 2017 with Ceph jewel/10.2 with the OSDs on filestore (nowadays being a legacy approach to storing objects in Ceph). However, we migrated the disks to bluestore since then (with ceph-disk and not yet via ceph-volume what s being used nowadays). Using ceph-disk introduces these 100MB XFS partitions containing basic metadata for the OSD.
Given that we had three working OSDs left, we decided to investigate how to rebuild the failing ones. Some folks on #ceph (thanks T1, ormandj + peetaur!) were kind enough to share how working XFS partitions looked like for them. After creating a backup (via dd), we tried to re-create such an XFS partition on server1. We noticed that even mounting a freshly created XFS partition failed:
synpromika@server1 ~ % sudo mkfs.xfs -f -i size=2048 -m uuid="4568c300-ad83-4288-963e-badcd99bf54f" /dev/sdc1
meta-data=/dev/sdc1 isize=2048 agcount=4, agsize=6272 blks
= sectsz=4096 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=0
= reflink=0
data = bsize=4096 blocks=25088, imaxpct=25
= sunit=128 swidth=64 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=1608, version=2
= sectsz=4096 sunit=1 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
synpromika@server1 ~ % sudo mount /dev/sdc1 /mnt/ceph-recovery
SB stripe unit sanity check failed
Metadata corruption detected at 0x433840, xfs_sb block 0x0/0x1000
libxfs_writebufr: write verifer failed on xfs_sb bno 0x0/0x1000
cache_node_purge: refcount was 1, not zero (node=0x1d3c400)
SB stripe unit sanity check failed
Metadata corruption detected at 0x433840, xfs_sb block 0x18800/0x1000
libxfs_writebufr: write verifer failed on xfs_sb bno 0x18800/0x1000
SB stripe unit sanity check failed
Metadata corruption detected at 0x433840, xfs_sb block 0x0/0x1000
libxfs_writebufr: write verifer failed on xfs_sb bno 0x0/0x1000
SB stripe unit sanity check failed
Metadata corruption detected at 0x433840, xfs_sb block 0x24c00/0x1000
libxfs_writebufr: write verifer failed on xfs_sb bno 0x24c00/0x1000
SB stripe unit sanity check failed
Metadata corruption detected at 0x433840, xfs_sb block 0xc400/0x1000
libxfs_writebufr: write verifer failed on xfs_sb bno 0xc400/0x1000
releasing dirty buffer (bulk) to free list!releasing dirty buffer (bulk) to free list!releasing dirty buffer (bulk) to free list!releasing dirty buffer (bulk) to free list!found dirty buffer (bulk) on free list!bad magic number
bad magic number
Metadata corruption detected at 0x433840, xfs_sb block 0x0/0x1000
libxfs_writebufr: write verifer failed on xfs_sb bno 0x0/0x1000
releasing dirty buffer (bulk) to free list!mount: /mnt/ceph-recovery: wrong fs type, bad option, bad superblock on /dev/sdc1, missing codepage or helper program, or other error.
Ouch. This very much looked related to the actual issue we re seeing. So we tried to execute mkfs.xfs with a bunch of different sunit/swidth settings. Using -d sunit=512 -d swidth=512 at least worked then, so we decided to force its usage in the creation of our OSD XFS partition. This brought us a working XFS partition. Please note, sunit must not be larger than swidth (more on that later!).
Then we reconstructed how to restore all the metadata for the OSD (activate.monmap, active, block_uuid, bluefs, ceph_fsid, fsid, keyring, kv_backend, magic, mkfs_done, ready, require_osd_release, systemd, type, whoami). To identify the UUID, we can read the data from ceph --format json osd dump , like this for all our OSDs (Zsh syntax ftw!):
With all of this, we reconstructed the keyring, fsid, whoami, block + block_uuid files. All the other files inside the XFS metadata partition are identical on each OSD. So after placing and adjusting the corresponding metadata on the XFS partition for Ceph usage, we got a working OSD hurray! Since we had to fix yet another 32 OSDs, we decided to automate this XFS partitioning and metadata recovery procedure.
We had a network share available on /srv/backup for storing backups of existing partition data. On each server, we tested the procedure with one single OSD before iterating over the list of remaining failing OSDs. We started with a shell script on server1, then adjusted the script for server2 and server3. This is the script, as we executed it on the 3rd server.
Thanks to this, we managed to get the Ceph cluster up and running again. We didn t want to continue with the Ceph upgrade itself during the night though, as we wanted to know exactly what was going on and why the system behaved like that. Time for RCA!
Root Cause Analysis
So all but three OSDs on server2 failed, and the problem seems to be related to XFS. Therefore, our starting point for the RCA was, to identify what was different on server2, as compared to server1 + server3. My initial assumption was that this was related to some firmware issues with the involved controller (and as it turned out later, I was right!). The disks were attached as JBOD devices to a ServeRAID M5210 controller (with a stripe size of 512). Firmware state:
synpromika@server1 ~ % sudo storcli64 /c0 show all grep '^Firmware'
Firmware Package Build = 24.16.0-0092
Firmware Version = 4.660.00-8156
synpromika@server2 ~ % sudo storcli64 /c0 show all grep '^Firmware'
Firmware Package Build = 24.21.0-0112
Firmware Version = 4.680.00-8489
synpromika@server3 ~ % sudo storcli64 /c0 show all grep '^Firmware'
Firmware Package Build = 24.16.0-0092
Firmware Version = 4.660.00-8156
This looked very promising, as server2 indeed runs with a different firmware version on the controller. But how so? Well, the motherboard of server2 got replaced by a Lenovo/IBM technician in January 2020, as we had a failing memory slot during a memory upgrade. As part of this procedure, the Lenovo/IBM technician installed the latest firmware versions. According to our documentation, some OSDs were rebuilt (due to the filestore->bluestore migration) in March and April 2020. It turned out that precisely those OSDs were the ones that survived the upgrade. So the surviving drives were created with a different firmware version running on the involved controller. All the other OSDs were created with an older controller firmware. But what difference does this make?
Now let s check firmware changelogs. For the 24.21.0-0097 release we found this:
- Cannot create or mount xfs filesystem using xfsprogs 4.19.x kernel 4.20(SCGCQ02027889)
- xfs_info command run on an XFS file system created on a VD of strip size 1M shows sunit and swidth as 0(SCGCQ02056038)
Our XFS problem certainly was related to the controller s firmware. We also recalled that our monitoring system reported different sunit settings for the OSDs that were rebuilt in March and April. For example, OSD 21 was recreated and got different sunit settings:
WARN server2.example.org Mount options of /var/lib/ceph/osd/ceph-21 WARN - Missing: sunit=1024, Exceeding: sunit=512
We compared the new OSD 21 with an existing one (OSD 25 on server3):
synpromika@server2 ~ % systemctl show var-lib-ceph-osd-ceph\\x2d21.mount grep sunit
Options=rw,noatime,attr2,inode64,sunit=512,swidth=512,noquota
synpromika@server3 ~ % systemctl show var-lib-ceph-osd-ceph\\x2d25.mount grep sunit
Options=rw,noatime,attr2,inode64,sunit=1024,swidth=512,noquota
Thanks to our documentation, we could compare execution logs of their creation:
So back then, we even tried to track this down but couldn t make sense of it yet. But now this sounds very much like it is related to the problem we saw with this Ceph/XFS failure. We follow Occam s razor, assuming the simplest explanation is usually the right one, so let s check the disk properties and see what differs:
It doesn t make sense that the minimum I/O size (iomin, AKA BLKIOMIN) is bigger than the optimal I/O size (ioopt, AKA BLKIOOPT). This leads us to Bug 202127 cannot mount or create xfs on a 597T device, which matches our findings here. But why did this XFS partition work in the past and fails now with the newer kernel version?
The XFS behaviour change
Now given that we have backups of all the XFS partition, we wanted to track down, a) when this XFS behaviour was introduced, and b) whether, and if so how it would be possible to reuse the XFS partition without having to rebuild it from scratch (e.g. if you would have no working Ceph OSD or backups left).
Let s look at such a failing XFS partition with the Grml live system:
root@grml ~ # grml-version
grml64-full 2020.06 Release Codename Ausgehfuahangl [2020-06-24]
root@grml ~ # uname -a
Linux grml 5.6.0-2-amd64 #1 SMP Debian 5.6.14-2 (2020-06-09) x86_64 GNU/Linux
root@grml ~ # grml-hostname grml-2020-06
Setting hostname to grml-2020-06: done
root@grml ~ # exec zsh
root@grml-2020-06 ~ # dpkg -l xfsprogs util-linux
Desired=Unknown/Install/Remove/Purge/Hold
Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
/ Name Version Architecture Description
+++-==============-============-============-=========================================
ii util-linux 2.35.2-4 amd64 miscellaneous system utilities
ii xfsprogs 5.6.0-1+b2 amd64 Utilities for managing the XFS filesystem
There it s failing, no matter which mount option we try:
Now let s try with an older kernel version (4.9.0), using old Grml 2017.05 release:
root@grml ~ # grml-version
grml64-small 2017.05 Release Codename Freedatensuppe [2017-05-31]
root@grml ~ # uname -a
Linux grml 4.9.0-1-grml-amd64 #1 SMP Debian 4.9.29-1+grml.1 (2017-05-24) x86_64 GNU/Linux
root@grml ~ # lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 9.0 (stretch)
Release: 9.0
Codename: stretch
root@grml ~ # grml-hostname grml-2017-05
Setting hostname to grml-2017-05: done
root@grml ~ # exec zsh
root@grml-2017-05 ~ #
root@grml-2017-05 ~ # xfs_info ./sdd1.dd
xfs_info: ./sdd1.dd is not a mounted XFS filesystem
1 root@grml-2017-05 ~ # xfs_repair ./sdd1.dd
Phase 1 - find and verify superblock...
bad primary superblock - bad stripe width in superblock !!!
attempting to find secondary superblock...
..............................................................................................Sorry, could not find valid secondary superblock
Exiting now.
1 root@grml-2017-05 ~ # mount ./sdd1.dd /mnt
root@grml-2017-05 ~ # mount -t xfs
/root/sdd1.dd on /mnt type xfs (rw,relatime,attr2,inode64,sunit=1024,swidth=512,noquota)
root@grml-2017-05 ~ # ls /mnt
activate.monmap active block block_uuid bluefs ceph_fsid fsid keyring kv_backend magic mkfs_done ready require_osd_release systemd type whoami
root@grml-2017-05 ~ # xfs_info /mnt
meta-data=/dev/loop1 isize=2048 agcount=4, agsize=6272 blks
= sectsz=4096 attr=2, projid32bit=1
= crc=1 finobt=1 spinodes=0 rmapbt=0
= reflink=0
data = bsize=4096 blocks=25088, imaxpct=25
= sunit=128 swidth=64 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal bsize=4096 blocks=1608, version=2
= sectsz=4096 sunit=1 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
Mounting there indeed works! Now, if we mount the filesystem with new and proper sunit/swidth settings using the older kernel, it should rewrite them on disk:
FTR: The sunit=512,swidth=512 from the xfs mount option is identical to xfs_info s output sunit=64,swidth=64 (because mount.xfs s sunit value is given in 512-byte block units, see man 5 xfs, and the xfs_info output reported here is in blocks with a block size (bsize) of 4096, so sunit = 512*512 := 64*4096 ).
mkfs uses minimum and optimal sizes for stripe unit and stripe width; you can check this e.g. via (note that server2 with fixed firmware version reports proper values, whereas server3 with broken controller firmware reports non-sense):
This is the underlying reason why the initially created XFS partitions were created with incorrect sunit/swidth settings. The broken firmware of server1 and server3 was the cause of the incorrect settings they were ignored by old(er) xfs/kernel versions, but treated as an error by new ones.
Make sure to also read the XFS FAQ regarding How to calculate the correct sunit,swidth values for optimal performance . We also stumbled upon two interesting reads in RedHat s knowledge base: 5075561 + 2150101 (requires an active subscription, though) and #1835947.
Am I affected? How to work around it?
To check whether your XFS mount points are affected by this issue, the following command line should be useful:
If you run into the above situation, the only known solution to get your original XFS partition working again, is to boot into an older kernel version again (4.17 or older), mount the XFS partition with correct sunit/swidth settings and then boot back into your new system (kernel version wise).
Lessons learned
document everything and ensure to have all relevant information available (including actual times of changes, used kernel/package/firmware/ versions. The thorough documentation was our most significant asset in this case, because we had all the data and information we needed during the emergency handling as well as for the post mortem/RCA)
if something changes unexpectedly, dig deeper
know who to ask, a network of experts pays off
including timestamps in your shell makes reconstruction easier (the more people and documentation involved, the harder it gets to wade through it)
keep an eye on changelogs/release notes
apply regular updates and don t forget invisible layers (e.g. BIOS, controller/disk firmware, IPMI/OOB (ILO/RAC/IMM/ ) firmware)
apply regular reboots, to avoid a possible delta becoming bigger (which makes debugging harder)
Thanks: Darshaka Pathirana, Chris Hofstaedtler and Michael Hanscho.
Looking for help with your IT infrastructure? Let us know!
One of my goals for Consfigurator is to make it
capable of installing Debian to my laptop, so that I can stop booting to GRML
and manually partitioning and debootstrapping a basic system, only to then
turn to configuration management to set everything else up. My configuration
management should be able to handle the partitioning and debootstrapping, too.
The first stage was to make Consfigurator capable of debootstrapping a basic
system, chrooting into it, and applying other arbitrary configuration, such as
installing packages. That s been in place for some weeks now. It s
sophisticated enough to avoid starting up newly installed services, but I
still need to add some bind mounting.
Another significant piece is teaching Consfigurator how to partition block
devices. That s quite tricky to do in a sufficiently general way I want to
cleanly support various combinations of LUKS, LVM and regular partitions,
including populating /etc/crypttab and /etc/fstab. I have some ideas about
how to do it, but it ll probably take a few tries to get the abstractions
right.
Let s imagine that code is all in place, such that Consfigurator can be
pointed at a block device and it will install a bootable Debian system to it.
Then to install Debian to my laptop I d just need to take my laptop s disk
drive out and plug it into another system, and run Consfigurator on that
system, as root, pointed at the block device representing my laptop s disk
drive. For virtual machines, it would be easy to write code which loop-mounts
an empty disk image, and then Consfigurator could be pointed at the
loop-mounted block device, thereby making the disk image file bootable.
This is adequate for virtual machines, or small single-board computers with
tiny storage devices (not that I actually use any of those, but I want
Consfigurator to be able to make disk images for them!). But it s not much
good for my laptop. I casually referred to taking out my laptop s disk drive
and connecting it to another computer, but this would void my laptop s
warranty. And Consfigurator would not be able to update my laptop s NVRAM, as
is needed on UEFI systems.
What s wanted here is a live system which can run Consfigurator directly on
the laptop, pointed at the block device representing its physical disk drive.
Ideally this live system comes with a chroot with the root filesystem for the
new Debian install already built, so that network access is not required, and
all Consfigurator has to do is partition the drive and copy in the contents of
the chroot. The live system could be set up to automatically start doing that
upon boot, but another option is to just make Consfigurator itself available
to be used interactively. The user boots the live system, starts up Emacs,
starts up Lisp, and executes a Consfigurator deployment, supplying the block
device representing the laptop s disk drive as an argument to the deployment.
Consfigurator goes off and partitions that drive, copies in the contents of
the chroot, and executes grub-install to make the laptop bootable. This is
also much easier to debug than a live system which tries to start partitioning
upon boot. It would look something like this:
;; melete.silentflame.com is a Consfigurator host object representing the
;; laptop, including information about the partitions it should have
(deploy-these :local ...
(chroot:partitioned-and-installed
melete.silentflame.com "/srv/chroot/melete" "/dev/nvme0n1"))
Now, building live systems is a fair bit more involved than installing Debian
to a disk drive and making it bootable, it turns out. While I want
Consfigurator to be able to completely replace the Debian Installer, I decided
that it is not worth trying to reimplement the relevant parts of the Debian
Live tool suite, because I do not need to make arbitrary customisations to any
live systems. I just need to have some packages installed and some files in
place. Nevertheless, it is worth teaching Consfigurator how to invoke Debian
Live, so that the customisation of the chroot which isn t just a matter of
passing options to lb_config(1) can be done with Consfigurator. This is what
I ve ended up with in Consfigurator s source code:
(defpropspec image-built :lisp (config dir properties)
"Build an image under DIR using live-build(7), where the resulting live
system has PROPERTIES, which should contain, at a minimum, a property from
CONSFIGURATOR.PROPERTY.OS setting the Debian suite and architecture. CONFIG
is a list of arguments to pass to lb_config(1), not including the '-a' and
'-d' options, which Consfigurator will supply based on PROPERTIES.
This property runs the lb_config(1), lb_bootstrap(1), lb_chroot(1) and
lb_binary(1) commands to build or rebuild the image. Rebuilding occurs only
when changes to CONFIG or PROPERTIES mean that the image is potentially
out-of-date; e.g. if you just add some new items to PROPERTIES then in most
cases only lb_chroot(1) and lb_binary(1) will be re-run.
Note that lb_chroot(1) and lb_binary(1) both run after applying PROPERTIES,
and might undo some of their effects. For example, to configure
/etc/apt/sources.list, you will need to use CONFIG not PROPERTIES."
(:desc (declare (ignore config properties))
#?"Debian Live image built in $ dir ")
(let* (...)
;; ...
(eseqprops
;; ...
(on-change
(eseqprops
(on-change
(file:has-content ,auto/config ,(auto/config config) :mode #o755)
(file:does-not-exist ,@clean)
(%lbconfig ,dir)
(%lbbootstrap t ,dir))
(%lbbootstrap nil ,dir)
(deploys ((:chroot :into ,chroot)) ,host))
(%lbchroot ,dir)
(%lbbinary ,dir)))))
Here, %lbconfig is a property running lb_config(1), %lbbootstrap one which
runs lb_bootstrap(1), etc. Those properties all just change directory to the
right place and run the command, essentially, with a little extra code to
handle failed debootstraps and the like.
The ON-CHANGE and ESEQPROPS combinators work together to sequence the
interaction of the Debian Live suite and Consfigurator.
In the innermost ON-CHANGE expression: create the file auto/config and
populate it with the call to lb_config(1) that we need to make, as described
in the Debian Live manual, chapter 6.
If doing so resulted in a change to the auto/config file e.g. the user
added some more options ensure that lb_config(1) and lb_bootstrap(1)
both get rerun.
Now in the inner ESEQPROPS expression, use DEPLOYS to configure the chroot,
essentially by forking into the chroot and recursively reinvoking
Consfigurator.
Finally, if any of the above resulted in a change being made, call
lb_chroot(1) and lb_binary(1).
This way, we only rebuild the chroot if the configuration changed, and we only
rebuild the image if the chroot changed.
Now over in my personal consfig:
(try-register-data-source
:git-snapshot :name "consfig" :repo #P"src/cl/consfig/" ...)
(defproplist hybrid-live-iso-built :lisp ()
"Build a Debian Live system in /srv/live/spw.
Typically this property is not applied in a DEFHOST form, but rather run as
needed at the REPL. The reason for this is that otherwise the whole image will
get rebuilt each time a commit is made to my dotfiles repo or to my consfig."
(:desc "Sean's Debian Live system image built")
(live-build:image-built.
'("--archive-areas" "main contrib non-free" ...)
"/srv/live/spw"
(os:debian-stable "buster" :amd64)
(basic-props)
(apt:installed "whatever" "you" "want")
(git:snapshot-extracted "/etc/skel/src" "dotfiles")
(file:is-copy-of "/etc/skel/.bashrc" "/etc/skel/src/dotfiles/.bashrc")
(git:snapshot-extracted "/root/src/cl" "consfig")))
The first argument to LIVE-BUILD:IMAGE-BUILT. is additional arguments to
lb_config(1). The third argument onwards are the properties for the live
system. The cool thing is GIT:SNAPSHOT-EXTRACTED the calls to this ensure
that a copy of my Emacs configuration and my consfig end up in the live image,
ready to be used interactively to install Debian, as described above. I ll
need to add something like (chroot:host-chroot-bootstrapped
melete.silentflame.com "/srv/chroot/melete") too.
As with everything Consfigurator-related, Joey Hess s
Propellor
is the giant upon whose shoulders I m standing.
The Debian Janitor is an automated
system that commits fixes for (minor) issues in Debian packages that can be
fixed by software. It gradually started proposing merges in early
December. The first set of changes sent out ran lintian-brush on sid packages maintained in
Git. This post is part of a series about the progress of the Janitor.
In my last blogpost, I introduced the buildlog
consultant - a tool that can
identify many reasons why a Debian build failed.
For example, here s a fragment of a build log where the Build-Depends lack
python3-setuptools:
Automatically acting on buildlog problems
A common reason why Debian builds fail is missing dependencies or incorrect versions of dependencies declared in the package build depends.
Based on the output of the buildlog consultant, it is possible in many cases to determine what dependency needs to be added to Build-Depends. In the example given above, we can use apt-file to look for the package that contains the path /usr/lib/python3/dist-packages/setuptools/__init__.py - and voila, we find python3-setuptools:
It builds the package using sbuild; if the package successfully builds then it just exits successfully
It tries to identify the problem by looking through the build log; if it can t or if it s a problem it has seen before (but apparently failed to resolve), then it exits with a non-zero exit code
It tries to find a dependency that can address the problem
It updates Build-Depends in debian/control or Depends in debian/tests/control
Go to step 1
This takes away the tedious manual process of building a package, discovering that a dependency is missing, updating Build-Depends and trying again.
For example, when I ran deb-fix-build while packaging saneyaml, the output looks something like this:
% deb-fix-build
Using output directory /tmp/tmpyz0nkgqq Using sbuild chroot unstable-amd64-sbuild Using fixers: Building debian packages, running 'sbuild --no-clean-source -A -s -v'. Attempting to use fixer upstream requirement fixer(apt) to address MissingPythonDistribution('setuptools_scm', python_version=3, minimum_version='4') Using apt-file to search apt contents Adding build dependency: python3-setuptools-scm (>= 4) Building debian packages, running 'sbuild --no-clean-source -A -s -v'. Attempting to use fixer upstream requirement fixer(apt) to address MissingPythonDistribution('toml', python_version=3, minimum_version=None) Adding build dependency: python3-toml Building debian packages, running 'sbuild --no-clean-source -A -s -v'. Built 0.5.2-1- changes files at [ saneyaml_0.5.2-1_amd64.changes ].
And in our Git repository, we see these changes as well:
Using deb-fix-build
You can run deb-fix-build by installing the ognibuild package from unstable. The only requirements
for using it are that:
The package is maintained in Git
A sbuild schroot is available for use
Caveats
deb-fix-build is fairly easy to understand, and if it doesn t work then you re no worse off than you were without it - you ll have to add your own Build-Depends.
That said, there are a couple of things to keep in mind:
At the moment, it doesn t distinguish between general, Arch or Indep Build-Depends.
It can only add dependencies for things that are actually in the archive
Sometimes there are multiple packages that can provide a file, command or python package - it tries to find the right one with heuristics but doesn t always get it right
Previously: v5.8
Linux v5.9 was released in October, 2020. Here s my summary of various security things that I found interesting:
seccomp user_notif file descriptor injection
Sargun Dhillon added the ability for SECCOMP_RET_USER_NOTIF filters to inject file descriptors into the target process using SECCOMP_IOCTL_NOTIF_ADDFD. This lets container managers fully emulate syscalls like open() and connect(), where an actual file descriptor is expected to be available after a successful syscall. In the process I fixed a couple bugs and refactored the file descriptor receiving code.
zero-initialize stack variables with Clang
When Alexander Potapenko landed support for Clang s automatic variable initialization, it did so with a byte pattern designed to really stand out in kernel crashes. Now he s added support for doing zero initialization via CONFIG_INIT_STACK_ALL_ZERO, which besides actually being faster, has a few behavior benefits as well. Unlike pattern initialization, which has a higher chance of triggering existing bugs, zero initialization provides safe defaults for strings, pointers, indexes, and sizes. Like the pattern initialization, this feature stops entire classes of uninitialized stack variable flaws.
common syscall entry/exit routines
Thomas Gleixner created architecture-independent code to do syscall entry/exit, since much of the kernel s work during a syscall entry and exit is the same. There was no need to repeat this in each architecture, and having it implemented separately meant bugs (or features) might only get fixed (or implemented) in a handful of architectures. It means that features like seccomp become much easier to build since it wouldn t need per-architecture implementations any more. Presently only x86 has switched over to the common routines.
SLAB kfree() hardening
To reach CONFIG_SLAB_FREELIST_HARDENED feature-parity with the SLUB heap allocator, I added naive double-free detection and the ability to detect cross-cache freeing in the SLAB allocator. This should keep a class of type-confusion bugs from biting kernels using SLAB. (Most distro kernels use SLUB, but some smaller devices prefer the slightly more compact SLAB, so this hardening is mostly aimed at those systems.)
new CAP_CHECKPOINT_RESTORE capability
Adrian Reber added the new CAP_CHECKPOINT_RESTORE capability, splitting this functionality off of CAP_SYS_ADMIN. The needs for the kernel to correctly checkpoint and restore a process (e.g. used to move processes between containers) continues to grow, and it became clear that the security implications were lower than those of CAP_SYS_ADMIN yet distinct from other capabilities. Using this capability is now the preferred method for doing things like changing /proc/self/exe.
debugfs boot-time visibility restriction
Peter Enderborg added the debugfs boot parameter to control the visibility of the kernel s debug filesystem. The contents of debugfs continue to be a common area of sensitive information being exposed to attackers. While this was effectively possible by unsetting CONFIG_DEBUG_FS, that wasn t a great approach for system builders needing a single set of kernel configs (e.g. a distro kernel), so now it can be disabled at boot time.
more seccomp architecture support
Michael Karcher implemented the SuperH seccomp hooks, Guo Ren implemented the C-SKY seccomp hooks, and Max Filippov implemented the xtensa seccomp hooks. Each of these included the ever-important updates to the seccomp regression testing suite in the kernel selftests.
stack protector support for RISC-V
Guo Ren implemented -fstack-protector (and -fstack-protector-strong) support for RISC-V. This is the initial global-canary support while the patches to GCC to support per-task canaries is getting finished (similar to the per-task canaries done for arm64). This will mean nearly all stack frame write overflows are no longer useful to attackers on this architecture. It s nice to see this finally land for RISC-V, which is quickly approaching architecture feature parity with the other major architectures in the kernel.
new tasklet API
Romain Perier and Allen Pais introduced a new tasklet API to make their use safer. Much like the timer_list refactoring work done earlier, the tasklet API is also a potential source of simple function-pointer-and-first-argument controlled exploits via linear heap overwrites. It s a smaller attack surface since it s used much less in the kernel, but it is the same weak design, making it a sensible thing to replace. While the use of the tasklet API is considered deprecated (replaced by threaded IRQs), it s not always a simple mechanical refactoring, so the old API still needs refactoring (since that CAN be done mechanically is most cases).
x86 FSGSBASE implementation
Sasha Levin, Andy Lutomirski, Chang S. Bae, Andi Kleen, Tony Luck, Thomas Gleixner, and others landed the long-awaited FSGSBASE series. This provides task switching performance improvements while keeping the kernel safe from modules accidentally (or maliciously) trying to use the features directly (which exposed an unprivileged direct kernel access hole).
filter x86 MSR writes
While it s been long understood that writing to CPU Model-Specific Registers (MSRs) from userspace was a bad idea, it has been left enabled for things like MSR_IA32_ENERGY_PERF_BIAS. Boris Petkov has decided enough is enough and has now enabled logging and kernel tainting (TAINT_CPU_OUT_OF_SPEC) by default and a way to disable MSR writes at runtime. (However, since this is controlled by a normal module parameter and the root user can just turn writes back on, I continue to recommend that people build with CONFIG_X86_MSR=n.) The expectation is that userspace MSR writes will be entirely removed in future kernels.
uninitialized_var() macro removed
I made treewide changes to remove the uninitialized_var() macro, which had been used to silence compiler warnings. The rationale for this macro was weak to begin with ( the compiler is reporting an uninitialized variable that is clearly initialized ) since it was mainly papering over compiler bugs. However, it creates a much more fragile situation in the kernel since now such uses can actually disable automatic stack variable initialization, as well as mask legitimate unused variable warnings. The proper solution is to just initialize variables the compiler warns about.
function pointer cast removals
Oscar Carter has started removing function pointer casts from the kernel, in an effort to allow the kernel to build with -Wcast-function-type. The future use of Control Flow Integrity checking (which does validation of function prototypes matching between the caller and the target) tends not to work well with function casts, so it d be nice to get rid of these before CFI lands.
flexible array conversions
As part of Gustavo A. R. Silva s on-going work to replace zero-length and one-element arrays with flexible arrays, he has documented the details of the flexible array conversions, and the various helpers to be used in kernel code. Every commit gets the kernel closer to building with -Warray-bounds, which catches a lot of potential buffer overflows at compile time.
That s it for now! Please let me know if you think anything else needs some attention. Next up is Linux v5.10.
Reading build logs
Build logs for Debian packages can be quite long and difficult
for a human to read. Anybody who has looked at these logs
trying to figure out why a build failed will have spent
time scrolling through them and skimming for certain phrases
(lines starting with error: for example). In many cases,
you can spot the problem in the last 10 or 20 lines of
output but it s also quite common that the error is somewhere
at the beginning of many pages of error output.
The buildlog consultant
The buildlog consultant project attempts to aid in this
process by parsing sbuild and non-Debian (e.g. the output
of make ) build logs and trying to identify the key line
that explains why a build failed. It can then either display
this specific line, or a fragment of the log around surrounding
the key line.
Classification
In addition to finding the key line explaining the failure,
it can also classify and parse the error in many cases
and return a result code and some metadata.
For example, in a failed build
of gnss-sdr that has produced 2119 lines of output,
the reason for the failure is that
log4cpp is missing which is on line 641:
-- Required GNU Radio Component: ANALOG missing! -- Could NOT find GNURADIO (missing: GNURADIO_RUNTIME_FOUND) -- Could NOT find PkgConfig (missing: PKG_CONFIG_EXECUTABLE) -- Could NOT find LOG4CPP (missing: LOG4CPP_INCLUDE_DIRS LOG4CPP_LIBRARIES) CMake Error at CMakeLists.txt:593 (message): *** Log4cpp is required to build gnss-sdr -- Configuring incomplete, errors occurred! See also "/<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/CMakeFiles/ CMakeOutput.log". See also "/<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/CMakeFiles/ CMakeError.log".
In this case, the buildlog consultant can both figure out
line was problematic and what the problem was:
How it works
The consultant does some structured parsing (most notably
it can parse the sections from a sbuild log), but otherwise
is a large set of carefully crafted regular expressions
and heuristics. It doesn t always find the problem, but
has proven to be fairly accurate. It is constantly improved
as part of the Debian Janitor project, and
that exposes it to a wide variety of different errors.
You can see the classification and error detection in action
on the result codes
page of the Janitor.
Using the buildlog consultant
You can get the buildlog consultant from either pip or Debian
unstable (package: python3-buildlog-consultant ).
The buildlog consultant comes with two scripts analyse-build-log and analyse-sbuild-log, for analysing
build logs and sbuild logs respectively.
Prince Caspian is the second book of the Chronicles of Narnia in
the original publication order (the fourth in the new publication order)
and a direct sequel to The Lion, the Witch
and the Wardrobe. As much as I would like to say you could start here
if you wanted less of Lewis's exploration of secondary-world Christianity
and more children's adventure, I'm not sure it would be a good reading
experience. Prince Caspian rests heavily on the events of
The Lion, the Witch and the Wardrobe.
If you haven't already, you may also want to read
my review of that book for some introductory
material about my past relationship with the series and why I follow the
original publication order.
Prince Caspian always feels like the real beginning of a re-read.
Re-reading The Lion, the Witch and the Wardrobe is okay but a bit
of a chore: it's very random, the business with Edmund drags on, and it's
very concerned with hitting the mandatory theological notes. Prince
Caspian is more similar to the following books and feels like Narnia
proper. That said, I have always found the ending of Prince
Caspian oddly forgettable. This re-read helped me see why: one of the
worst bits of the series is in the middle of this book, and then the
dramatic shape of the ending is very strange.
MAJOR SPOILERS BELOW for both this book and The Lion, the
Witch and the Wardrobe.
Prince Caspian opens with the Pevensie kids heading to school by
rail at the end of the summer holidays. They're saying their goodbyes to
each other at a train station when they are first pulled and then dumped
into the middle of a wood. After a bit of exploration and the discovery
of a seashore, they find an overgrown and partly ruined castle.
They have, of course, been pulled back into Narnia, and the castle is Cair
Paravel, their great capital when they ruled as kings and queens. The
twist is that it's over a thousand years later, long enough that Cair
Paravel is now on an island and has been abandoned to the forest. They
discover parts of how that happened when they rescue a dwarf named
Trumpkin from two soldiers who are trying to drown him near the supposedly
haunted woods.
Most of the books in this series have good hooks, but Prince
Caspian has one of the best. I adored everything about the start of this
book as a kid: the initial delight at being by the sea when they were on
their way to boarding school, the realization that getting food was not
going to be easy, the abandoned castle, the dawning understanding of where
they are, the treasure room, and the extended story about Prince Caspian,
his discovery of the Old Narnia, and his flight from his usurper uncle.
It becomes clear from Trumpkin's story that the children were pulled back
into Narnia by Susan's horn (the best artifact in these books), but
Caspian's forces were expecting the great kings and queens of legend from
Narnia's Golden Age. Trumpkin is delightfully nonplussed at four
school-age kids who are determined to join up with Prince Caspian and
help.
That's the first half of Prince Caspian, and it's a solid magical
adventure story with lots of potential. The ending, alas, doesn't
entirely work. And between that, we get the business with Aslan and Lucy
in the woods, or as I thought of it even as a kid, the bit where Aslan is
awful to everyone for no reason.
For those who have forgotten, or who don't care about spoilers, the kids
plus Trumpkin are trying to make their way to Aslan's How (formerly the
Stone Table) where Prince Caspian and his forces were gathered, when they
hit an unexpected deep gorge. Lucy sees Aslan and thinks he's calling for
them to go up the gorge, but none of the other kids or Trumpkin can see
him and only Edmund believes her. They go down instead, which almost gets
them killed by archers. Then, that night, Lucy wakes up and finds Aslan
again, who tells her to wake the others and follow him, but warns she may
have to follow him alone if she can't convince the others to go along.
She wakes them up (which does not go over well), Aslan continues to be
invisible to everyone else despite being right there, Susan is
particularly upset at Lucy, and everything is awful. But this time they
do follow her (with lots of grumbling and over Susan's objections). This,
of course, is the right decision: Aslan leads them to a hidden path that
takes them over the river they're trying to cross, and becomes visible to
everyone when they reach the other side.
This is a mess. It made me angry as a kid, and it still makes me angry
now. No one has ever had trouble seeing Aslan before, so the kids are
rightfully skeptical. By intentionally deceiving them, Aslan puts the
other kids in an awful position: they either have to believe Lucy is
telling the truth and Aslan is being weirdly malicious, or Lucy is
mistaken even though she's certain. It not only leads directly to
conflict among the kids, it makes Lucy (the one who does all the right
things all along) utterly miserable. It's just cruel and mean, for no
purpose.
It seems clear to me that this is C.S. Lewis trying to make a theological
point about faith, and in a way that makes it even worse because I think
he's making a different point than he intended to make. Why is religious
faith necessary; why doesn't God simply make himself apparent to everyone
and remove the doubt? This is one of the major problems in Christian
apologetics, Lewis chooses to raise it here, and the answer he gives is
that God only shows himself to his special favorites and hides from
everyone else as a test. It's clearly not even a question of intention to
have faith; Edmund has way more faith here than Lucy does (since Lucy
doesn't need it) and still doesn't get to see Aslan properly until
everyone else does. Pah.
The worst part of this is that it's effectively the last we see of Susan.
Prince Caspian is otherwise the book in which Susan comes into her
own. The sibling relationship between the kids is great here in general,
but Susan is particularly good. She is the one who takes bold action to
rescue Trumpkin, risking herself by firing an arrow into the helmet of one
of the soldiers despite being the most cautious of the kids. (And then
gets a little defensive about her shot because she doesn't want anyone to
think she would miss that badly at short range, a detail I just love.) I
identified so much with her not wanting to beat Trumpkin at an archery
contest because she felt bad for him (but then doing it anyway). She is,
in short, awesome.
I was fine with her being the most grumpy and frustrated with the argument
over picking a direction. They're all kids, and sometimes one gets grumpy
and frustrated and awful to the people around you. Once everyone sees
Aslan again, Susan offers a truly excellent apology to Lucy, so it seemed
like Lewis was setting up a redemption arc for her the way that he did for
Edmund in The Lion, the Witch and the Wardrobe (although I maintain
that nearly all of this mess was Aslan's fault). But then we never see
Susan's conversation with Aslan, Peter later says he and Susan are now too
old to return to Narnia, and that's it for Susan. Argh.
I'll have more to say about this later (and it's not an original opinion),
but the way Lewis treats Susan is the worst part of this series, and it
adds insult to injury that it happens immediately after she has a chance
to shine.
The rest of the book suffers from the same problem that The Lion,
the Witch and the Wardrobe did, namely that Aslan fixes everything in a
somewhat surreal wild party and it's unclear why the kids needed to be
there. (This is the book where Bacchus and Silenus show up, there is a
staggering quantity of wine for a children's book, and Aslan turns a bunch
of obnoxious school kids into pigs.) The kids do have more of a role to
play this time: Peter and Edmund help save Caspian, and there's a
(somewhat poorly motivated) duel that sends up the ending. But other than
the brief battle in the How, the battle is won by Aslan waking the trees,
and it's not clear why he didn't do that earlier. The ending is, at best,
rushed and not worthy of its excellent setup. I was also disappointed
that the "wait, why are you all kids?" moment was hand-waved away by
Narnia giving the kids magical gravitas.
Lewis never felt in control of either The Lion, the Witch and the
Wardrobe or Prince Caspian. In both cases, he had a great hook
and some ideas of what he wanted to hit along the way, but the endings are
more sense of wonder and random Aslan set pieces than anything that
follows naturally from the setup. This is part of why I'm not commenting
too much on the sour notes, such as the red dwarves being the good and
loyal ones but the black dwarves being suspicious and only out for
themselves. If I thought bits like that were deliberate, I'd complain
more, but instead it feels like Lewis threw random things he liked about
children's books and animal stories into the book and gave it a good stir,
and some of his subconscious prejudices fell into the story along the way.
That said, resolving your civil war children's book by gathering all the
people who hate talking animals (but who have lived in Narnia for
generations) and exiling them through a magical gateway to a conveniently
uninhabited country is certainly a choice, particularly when you wrote the
book only two years after the
Partition of
India. Good lord.
Prince Caspian is a much better book than The Lion, the Witch
and the Wardrobe for the first half, and then it mostly falls apart. The
first half is so good, though. I want to read the book that this could
have become, but I'm not sure anyone else writes quite like Lewis at his
best.
Followed by The Voyage of the Dawn Treader, which is my absolute
favorite of the series.
Rating: 7 out of 10
Here is my monthly update covering what I have been doing in the free software world during March 2021 (previous month):
Reviewed and merged a number of contributions from Dan Palmer for my django-autologin library aimed at applications that use the Django web-development framework that wish to include automatic "login" links in emails (etc.). Changes made include allowing callers to override max-age [...], ensuring we always retrieve the same user account [...] and to improve the usability of the public API [...].
Opened a pull request to make the build process for Scrapy (a framework for extracting data from websites) reproducible. [...]
As part of my role of being the assistant Secretary of the Open Source Initiative and a board director of Software in the Public Interest, I attended their respective monthly meetings and other miscellaneous discussions occurring on the internet as well as the usual internal discussions, etc. As it happens, my term on the OSI board has been temporarily extended due to the discovery of a vulnerability in OSI's recent election as a result, the 2021 election will be re-run to ensure complete transparency of process.
Reproducible Builds
One of the original promises of open source software is that distributed peer review and transparency of process results in enhanced end-user security. However, whilst anyone may inspect the source code of free and open source software for malicious flaws, almost all software today is distributed as pre-compiled binaries. This allows nefarious third-parties to compromise systems by injecting malicious code into ostensibly secure software during the various compilation and distribution processes.
The motivation behind the Reproducible Builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised.
The project is proud to be a member project of the Software Freedom Conservancy. Conservancy acts as a corporate umbrella allowing projects to operate as non-profit initiatives without managing their own corporate structure. If you like the work of the Conservancy or the Reproducible Builds project, please consider becoming an official supporter.
This month, I:
Categorised a huge number of packages and issues in the Reproducible Builds "notes" repository.
Whilst looking into reproducible builds issues, I noticed that the heudiconv package could not be built reproducibly. This was because the call to help2man fails, so the manual page includes a Python traceback instead of the actual manpage. I filed a bug with a patch as Debian bug #984778.
I uploaded flask-peewee (0.6.7-3) to Debian to make the build reproducible (#885326) and did the same for pyvows (3.0.0-3) (#977487) too, refreshing the packaging at the same time.
I also made the following changes to diffoscope, including uploading versions 169, 170 and 171 to Debian:
New features:
If zipinfo(1) shows a difference but we cannot uncover a difference within the underlying .zip or .apk file, add a comment to the output and actually show the binary comparison. (#246)
Ensure all our temporary directories have useful names. [...]
Ignore --debug and similar arguments when creating a (hopefully-useful) temporary directory suffix. [...]
Optimisations:
Avoid frequent long lines in RPM header outputs that cause extremely slow HTML output generation. (#245)
Use larger read buffer block sizes when extracting files from archives. [...]
Use a much-shorter HTML class name instead of diffponct to optimise HTML output. [...]
Output improvements:
Make error extracting X, falling back to binary comparison 'Y' error message in diffoscope's output nicer. [...]
Don't emit "Unable to stat file" debug messages at all. We have entirely-artificial directory "entries" such as ELF sections which, of course, will never exist as files. [...]
Logging improvements:
Add the target directory when logging which directory we are extracting containers to. [...]
Format report size messages when generating HTML reports. [...]
Don't emit a Returning a FooContainer logging message too, as we already emit Instantiating a FooContainer log message. [...]
Reduce "Unable to stat file" warnings to debug messages as these are sometimes by design. [...]
Misc improvements:
Clarify a comment regarding not extracting excluded files. [...]
Remove trailing newline from updated test file (re: #243). [...]
Fix test_libmix_differences failure on openSUSE Tumbleweed. (#244)
Move test_rpm to use the assert_diff utility helper.
Frontdesk duties, responding to user/developer questions, attending monthly meeting, reviewing others' packages, participating in internal mailing list discussions, etc.
Proposed an stable update for python-django in buster. (#983526)
Prepared and uploaded a stable update for redis in buster. (#983527)
Issued DLA 2595-1 and ELA 380-1 for velocity, a Java-based template engine for writing web applications. Velocity could be exploited to run arbitrary code by applications that allowed untrusted users to upload/modify templates.
Issued DLA 2597-1 and ELA 381-1 to address a cross-site scripting (XSS) vulnerability in velocity-tools, a collection of useful tools for the Velocity template engine. The default error page could be exploited to steal session cookies, perform requests in the name of the victim, used for phishing attacks and many other similar attacks.
Issued DLA 2600-1 and ELA 384-1 for pygments as it was discovered that there was a series of denial of service vulnerabilities in this syntax highlighting library for Python. A number of regular expressions had cubic (or even exponential) worst-case complexity which could cause a remote denial of service (DoS) when provided with malicious input.
Issued DLA 2603-1 to address a number of vulnerabilities in libmediainfo, a library reading metadata such as track names, lengths, etc. from media files.
You can find out more about the Debian LTS via the following video:
Note: This is the continuation of part 1 where I presented Freexian and its purpose.
Going forward: growing Freexian
Part 1: From Debian LTS to Debian for the Enterprise
Freexian s Debian LTS service has so far been entirely successful, with a steady growth over the years. Thanks to this, and even if there are always new challenges, it is fair to say that the Debian LTS team has met its goal in the last few years.
While this started from the desire to make LTS a reality, many sponsors are only looking for a way to give back to Debian through their company, and to make sure that Debian fits their needs.
But if you look at the bigger picture outside of this small LTS area, you will easily find many issues that need to be addressed if we want Debian to meet the needs of corporate users. Those issues can have widely different types and complexity. They can be as simple as missing the latest upstream version for an important package because the maintainer disappeared and nobody noticed before it was too late (i.e. the release was frozen); or a somewhat basic piece of software not yet packaged at all; or a release critical bug that was left unattended. On the other end of the spectrum, some corporate requirements will prove tougher to solve, for instance for large software suites that are complex to package, or could potentially have an impact elsewhere in Debian.
Bringing those facts together, we would like to have Freexian s Debian LTS/ELTS offering evolve into a more general Debian Software Assurance offering, where you commit to a yearly budget for Debian sponsorship in the larger sense. That budget would fund different projects and the allocation between those projects would vary over time depending on the desires and needs of the sponsors/customers:
Technical support: the budget would always ensure that you have a few spare hours of technical support available in case you need them
Debian LTS: we want this to continue!
Debian ELTS: when the customer has not managed to migrate their Debian servers in time, they should be able to reallocate their budget towards ELTS and ensure their servers are secure until the migration has taken place.
Debian for the enterprise
Make sure that the packages used by sponsors are in good shape in Debian Testing/Unstable so that they are in the best shape for the next stable release.
Package new software that are relevant for corporate users. Offer to pool the maintenance work.
Fix bugs that customers are hitting.
Etc.
Debian project funding: that s the variable part of the budget (and would have a minimum of 10% like we do for Debian LTS right now). When the other projects do not consume the whole budget, we invest the remaining money into generic Debian improvements.
This major shift in our offering would also be an ideal opportunity to build a professional, free-software based infrastructure aimed at sustaining this business, making it easier to administer the various aspects of this work, and easily allowing many more sponsors to join (individuals included!).
On a more pragmatic/operational note, this shift will bring a lot of challenges to the table, and those can hardly be handled with the current resources of Freexian: if we hope to properly implement this new strategy, we ll need some additional help.
This article is to be continued in an upcoming post. Stay tuned!
I'm a big fan of Makefiles. Almost all my side projects are
using them, and I've been advocating their usage at work too.
Makefiles give your contributors an entry point on how to do certain things
like, building, testing, deploying. And if done correctly, they can massively
simplify your CI/CD pipeline scripts as they can often just stupidly call the
respective make targets. Most importantly, they are a very convenient
shortcut for you as a developer as well.
For Python projects, where I'm almost always using virtual
environments, I've been using two different strategies for Makefiles:
assuming that make is executed inside the virtual environment
wrapping all virtual environment calls inside make
Both strategies have their pros and cons.
Assuming make is executed inside the venv
Let's have a look at a very simple Makefile that allows for building,
testing and releasing a Python project:
This is straightforward and a potential contributor immediately knows the
entry points to your project.
Assuming there is a venv installed already, you have to activate it first and
run the make commands afterwards:
$ . venv/bin/activate
$ make test
The downside is of course, that you have to activate the venv for every new
shell. Which can get a bit annoying when you spawn a new terminal in tmux or
put vim into the background to run make.
Activating the venv inside make will not work, as each recipe runs in its own
shell, moreover each command in each recipe runs in its own shell too. There
are workarounds for the latter, i.e. using the .ONESHELL flag, but that does
not solve the issue of a new shell in each recipe.
Wrapping the venv calls inside make
The second approach mitigates for the issue of activating the venv altogether.
I've borrowed this idea mostly from makefile.venv and simplified it a lot
for my needs.
# system python interpreter. used only to create virtual environmentPY= python3
VENV= venv
BIN=$(VENV)/bin
# make it work on windows tooifeq ($(OS), Windows_NT)BIN=$(VENV)/Scripts
PY=python
endifall:linttest$(VENV):requirements.txtrequirements-dev.txtsetup.py$(PY) -m venv $(VENV)$(BIN)/pip install --upgrade -r requirements.txt
$(BIN)/pip install --upgrade -r requirements-dev.txt
$(BIN)/pip install -e .
touch $(VENV).PHONY:testtest:$(VENV)$(BIN)/pytest
.PHONY:lintlint:$(VENV)$(BIN)/flake8
.PHONY:releaserelease:$(VENV)$(BIN)/python setup.py sdist bdist_wheel upload
clean:
rm -rf $(VENV)
find . -type f -name *.pyc -delete
find . -type d -name __pycache__ -delete
The equivalent Makefile now looks immediately more complicated. So let's
break it down.
Instead of calling just calling pytest, flake8 and python assuming the
venv is already activated or all dependencies are installed on the system
directly, we explicitly call the ones from the venv by prefixing the command
with the path to the venv's bin directory. To ensure the venv exists, each
of the recipes is depending on the $(VENV) target, which ensures we always
have an up-to-date venv installed.
This works, because the . venv/bin/activate-script basically just does the
same: it puts the venv before anything else in your PATH, therefore each
call to python, etc. will find the one installed in the venv first.
While the Makefile is a bit more complicated, we now can just call
$ make test
and don't deal with venvs directly any more (well, for those simple cases at
least...). If you don't need to support Windows, you can remove the
appropriate block and the Makefile looks relatively tame, even for people
that don't use make very often.
Which one is better?
I think the second approach is more convenient. I've used the first approach
happily for years and only learned quite recently about the second one. I
haven't really noticed any downsides yet, but I do realize that almost all
Python projects with Makefiles I've checked out, seem to prefer the first
approach. I wonder why that is?
A new release 1.1.39-3 of x13binary, of the X-13ARIMA-SEATS program by the US Census Bureau (with upstream release 1.1.39) is now on CRAN.
The x13binary package takes the pain out of installing X-13ARIMA-SEATS by making it a fully resolved CRAN dependency. For example, when installing the excellent seasonal package by Christoph, then X-13ARIMA-SEATS will get pulled in via the x13binary package and things just work. Just depend on x13binary and on all major OSs supported by R you should have an X-13ARIMA-SEATS binary installed which will be called seamlessly by the higher-level packages such as seasonal or gunsales. With this the full power of the what is likely the world s most sophisticated deseasonalization and forecasting package is now at your fingertips and the R prompt, just like any other of the 17350+ CRAN packages. You can read more about this (and the seasonal package) in the Journal of Statistical Software paper by Christoph and myself.
This release was needed because the recent M1mac build was reporting leftover detritus in the temporary directory, which we addressed with an explicit removal at end. We also addressed another CRAN Policy change since the last release, namely a conversion of the configure script from bash to sh.
Now, sadly, that second aspect blew up on Solaris, and the detritus issue appears to be persist. By now Christoph and a colleague have installed R(-devel) on such an M1 machine, but still cannot reproduce. We will reach out to CRAN to learn more. A follow-up release 1.1.39-4 is likely.
The good news is that the standard macOS binary works on M1 as do other binaries thanks to the translation layer. We do however lack a genuine binary for Solaris so if any of the esteemed readers of this post happens to have access to R on Solaris along with a basic Fortran compiler, we would love to hear from you. Building X-13ARIMA-SEATS from source on Solaris should be straightforward, it is on the other OSs.
Courtesy of my CRANberries, there is also a diffstat report for this release showing changes to the previous release.
If you like this or other open-source work I do, you can sponsor me at GitHub.