A new version 0.0.6 of RcppSpdlog is now on CRAN. It contains releases 1.9.0 of spdlog which in turn contains an updated version of fmt.
RcppSpdlog bundles spdlog, a wonderful header-only C++ logging library with all the bells and whistles you would want that was written by Gabi Melman, and also includes fmt by Victor Zverovich. No R package-side changes were needed or made.
The (minimal) NEWS entry for this release follows.
Quick hardware review
This is a neat little device. It's very similar to the Glo HD, which
is a bit disappointing: you'd think they would have improved on the
design in the 5+ years since the Glo HD has come out.. It does have an
"amber" night light which is nice, but the bezel is still not level
with the display, and the device is still kind of on the thick side. A
USB-C (instead of micro-USB) port would have been nice too.
But otherwise, it's pretty slick, and just works. And because the
hardware design didn't change, I can still hack at it like a madman,
which is really why I bought this thing in the first place.
Hopefully it will last longer than 5 years. Ebook readers should
really last for decades, not years, but I guess that's too much to
expect from our consumerist, suicidal, extinctionist society.
Configuration hacks
Here are the hacks I done on the device. I had done many more hacks on
the Kobo Glo HD, but I decided to take a more streamlined, minimalist
and, hopefully, easier for new users than the pile of hacks I was
doing before (which I expand on at the end of the article).
SD card replacement
I replaced the SD card. The original card shipped with the Clara
HD was 8GB which meant all my books actually fitted on the original,
but just barely. The new card is 16GB.
Unfortunately, I did this procedure almost at the end of this guide
(right before writing the syncthing scripts, below). Next time, that
should be the first thing done so the original SD card acts as a
pristine copy of the upstream firmware. So even though this seems like
an invasive and difficult procedure, I actually do recommend you do it
first.
The process is basically to:
crack open the Kobo case (don't worry, it sounds awful but I've
done it often)
take the SD card out
copy it over to a new, larger card (say on your computer)
Install koreader
My e-reader of choise is Koreader. It's just that great. I still don't
find the general user interface (ie. the "file browswer") as intuitive
as the builtin one, but the book reading just feels better. And
anyways it's the easier way to get a shell on the device.
Follow those instructions, particularly the NickelMenu
instructions (see also the NickelMenu home page). Yes, you need
to install some other thing to start koreader, which doesn't start on
its own. NickelMenu is the simplest and better integrated I have
found.
You might also want to install some dictionnaries and configure
SSH:
mount USB
drop your SSH public key in
.../KOBOeReader/.adds/koreader/settings/SSH/authorized_keys
Note that ed25519 keys do not work: try an RSA key. This might be
because koreader ships with dropbear (or an older version), but I
haven't verified this.
Install syncthing
I use Syncthing to copy all my books into the device now. I was
previously using Koreader's OPDS support with Calibre's web interface,
but that was clunky and annoying, and I'd constantly have to copy
books around. Now the entire collection is synchronized.
As a bonus, I can actually synchronise (and backup!) the koreader
metadata, since it's stored next to the files. So in theory, this
means I could use koreader from multiple devices and have my reading
progress sync'd, but I haven't tested that feature just yet.
I chose Syncthing because it's simple, lightweight, supported on
Linux and Android, and statically compiles by default which means it's
easy to deploy on the Kobo.
Here is how I installed and started Syncthing at first:
copy a valid ca-certificates.crt file (say from your Linux
desktop) into /etc/ssl/certs/ on the Kobo (otherwise syncthing
cannot bootstrap discovery servers)
launch syncthing over SSH: /mnt/onboard/.adds/syncthing
You should now be able to connect to the syncthing GUI through your
web browser.
Immediately change the GUI admin user and password on the
Settings: GUI tab.
Then, figure out how to start it. Here are your options:
on boot (inittab or whatever). downside: power usage.
on wifi (udev hacks). downside: unreliable (see wallabako).
on demand (e.g. nickel menu, koreader terminal
shortcuts). downside: kind of clunky in koreader, did not work in
nickel menu.
manually, through shell. downside: requires a shell, but then
again we already have one through koreader?
What I have done is to write trivial shell scripts (in
.../KOBOeReader/scripts) to start syncthing. The first is
syncthing-start.sh:
#!/bin/sh
/mnt/onboard/.adds/syncthing serve &
Then syncthing-stop.sh:
#!/bin/sh
/usr/bin/pkill syncthing
This makes those scripts usable from the koreader file browser. Then
the folder can be added to the folder shortcuts and a long-hold on the
script will allow you to execute it.
Still have to figure out why the Nickel Menu script is not working,
but it could simply reuse the above to simplify debugging. This is the
script I ended up with, in .../KOBOeReader/.adds/nm/syncthing:
It's unclear why this doesn't work: I only get "Error starting
Syncthing server" for the toggle, and no output for the (start)
action. In either case, syncthing doesn't actually start.
Avoided tasks
This list wouldn't be complete without listing more explicitly the
stuff I have done before on the Kobo Glo HD and which I have deliberately
decided not to do here because my time is precious:
plato install: beautiful project, but koreader is good enough
wallabako setup: too much work to maintain, Wallabag articles
are too distracting and available on my phone anyways
using calibre to transfer books: not working half the time,
different file layout than the source, one less Calibre
dependency
using calibre to generate e-books based on RSS feeds (yes, I did
that, and yes, it was pretty bad and almost useless)
A new release 0.2.2 of pkgKitten is now on CRAN, and will be uploaded to Debian. pkgKitten makes it simple to create new R packages via a simple function invocation. A wrapper kitten.r exists in the littler package to make it even easier.
This release simply corrects on minor aspect in the optional roxygen2 use, and updates the DESCRIPTION file.
If you have been involved in Debian packaging at all in the last few years, you
are probably aware that autopkgtest is now an important piece of the Debian
release process. Back in 2018, the automated testing migration process started
considering autopkgtest test results as part of its decision
making.
Since them, this process has received several improvements. For example, during
the bullseye freeze, non-key packages with a non-trivial
autopkgtest test suite could migrate automatically to testing without their
maintainers needing to open unblock requests, provided there was no regression
in theirs autopkgtest (or those from their reverse dependencies).
Since 2014 when ci.debian.net was first introduced, we have seen an
amazing increase in the number of packages in Debian that can be automatically
tested. We went from around 100 to 15,000 today. This means not only happier
maintainers because their packages get to testing faster, but also improved
quality assurance for Debian as a whole.
However, the growth rate seems to be decreasing. Maybe the low hanging fruit
have all been picked, or maybe we just need to help more people jump in the
automated testing bandwagon.
With that said, we would like to encourage and help more maintainers to add
autopkgtest to their packages. To that effect, I just created the
autopkgtest-help repository on salsa, where we will take
help requests from maintainers working on autopkgtest for their packages.
If you want help, please go ahead and create an issue in there.
To quote the repository README:
Valid requests:
"I want to add autopkgtest to package X. X is a tool that [...] and it works
by [...]. How should I approach testing it?"
It's OK if you have no idea where to start. But at least try to describe your
package, what it does and how it works so we can try to help you.
"I started writing autopkgtest for X, here is my current work in progress
[link]. But I encountered problem Y. How to I move forward?"
If you already have an autopkgtest but is having trouble making it work as
you think it should, you can also ask here.
Invalid requests:
"Please write autopkgtest for my package X for me".
As with anything else in free software, please show appreciation for other
people's time, and do your own research first. If you pose your question with
enough details (see above) and make it interesting, it may be that whoever
answers will write at least a basic structure for you, but as the maintainer
you are still the expert in the package and what tests are relevant.
If you ask your question soon, you might get your answer recorded in video: we
are going to have a DebConf21 talk next month, where we I and
Paul Gevers (elbrus) will answer a few autopkgtest questions in video for
posterity.
Now, if you have experience enabling autopkgtest for you own packages, please
consider watching that repository there to help us help our fellow maintainers.
BBI Kenya and live Supreme Court streaming on YT
The last few weeks have been unrelenting as all sorts of news have been coming in, mostly about the downturn in the Economy, Islamophobia in India on the rise, Covid, and electioneering. However, in the last few days, Kenya surpassed India in live-streaming proceeds in a Court of Appeals about BBI or Building Bridges Initiative. A background filler article on the topic can be found in BBC. The live-streaming was done via YT and if wants to they can start from
https://www.youtube.com/watch?v=JIQzpmVKvro
One can also subscribe to K24TV which took the initiative of sharing the proceedings with people worldwide. If K24TV continues to share SC proceedings of Kenya, that would add to the soft power of Kenya. I will not go into the details of the case as Gautam Bhatia who has been following the goings-on in Kenya is a far better authority on the subject. In fact, just recently he shared about another Kenyan judgment from a trial which can be seen here. He has shared the proceedings and some hot takes on the Twitter thread started by him. Probably after a couple of weeks or more when he has processed what all has happened there, he may also share some nuances although many of his thoughts would probably go to his book on Comparative Constitutional Law which he hopes to publish maybe in 2021/2022 or whenever he can. Such televised proceedings are sure to alleviate the standing of Kenya internationally. There has been a proposal to do similar broadcasts by India but with surveillance built-in, so they know who is watching. The problems with the architecture and the surveillance built-in have been shared by Srinivas Kodali or DigitalDutta quite a few times, but that probably is a story for another day.
Uttar Pradesh Population Control BillHindus comprise 83% of Indian couples with more than two child children
The U.P. Population Bill came and it came with lot of prejudices. One of the prejudices is the idea that Muslims create or procreate to have the most children. Even with data is presented as shared above from NFHS National Family Health Survey which is supposed to carry our surveys every few years did the last one around 4 years back. The analysis from it has been instrumental not only in preparing graphs as above but also sharing about what sort of death toll must have been in rural India. And as somebody who have had the opportunity in the past, can vouch that you need to be extremely lucky if something happens to you when you are in a rural area.
Even in places like Bodh Gaya (have been there) where millions of tourists come as it is one of the places not to be missed on the Buddhism tourist circuit, the medical facilities are pretty underwhelming. I am not citing it simply because there are too many such newspaper reports from even before the pandemic, and both the State and the Central Govt. response has been dismal. Just a few months back, they were recalled. There were reports of votes being bought at INR 1000/- (around $14) and a bottle or two of liquor. There used to be a time when election monitoring whether national or state used to be a thing, and you had LTO s (Long-time Observers) and STO s (Short-Term Observers) to make sure that the election has been neutral. This has been on the decline in this regime, but that probably is for another time altogether. Although, have to point out the article which I had shared a few months ago on the private healthcare model is flawed especially for rural areas. Instead of going for cheap, telemedicine centers that run some version of a Linux distro. And can provide a variety of services, I know Kerala and Tamil Nadu from South India have experimented in past but such engagements need to be scaled up. This probably will come to know when the next time I visit those places (sadly due to the virus, not anytime soonish.:( ) .
Going back to the original topic, though, I had shared Hans Rosling s famous Ted talk on population growth which shows that even countries which we would not normally associate with family planning for e.g. the middle-east and Africa have also been falling quite rapidly. Of course, when people have deeply held prejudices, then it is difficult. Even when sharing China as to how they had to let go of their old policy in 2016 as they had the thing for leftover men . I also shared the powerful movie So Long my Son. I even shared how in Haryana women were and are trafficked and have been an issue for centuries but as neither suits the RW propaganda, they simply refuse to engage. They are more repulsed by people who publish this news rather than those who are actually practicing it, as that is culture . There is also teenage pregnancy, female infanticide, sex-selective abortion, etc., etc. It is just all too horrible to contemplate.
Personal anecdote I know a couple, or they used to be a couple, where the gentleman wanted to have a male child. It was only after they got an autistic child, they got their DNA tested and came to know that the gentleman had a genetic problem. He again forced and had another child, and that too turned out to be autistic. Finally, he left the wife and the children, divorced them and lived with another woman. Almost a decade of the wife s life was ruined. The wife before marriage was a gifted programmer employed at IBM. This was an arranged marriage. After this, if you are thinking of marrying, apart from doing astrology charts, also look up DNA compatibility charts. Far better than ruining yours or the women s life. Both the children whom I loved are now in heaven, god bless them
If one wants to, one can read a bit more about the Uttar Pradesh Population bill here. The sad part is that the systems which need fixing, nobody wants to fix. The reason being simple. If you get good health service by public sector, who will go to the private sector. In Europe, AFAIK they have the best medical bang for the money. Even the U.S. looks at Europe and hopes it had the systems that Europe has but that again is probably for another day.
South Africa and India long-lost brothers.
As had shared before, after the 2016 South African Debconf convention, I had been following South Africa. I was happy when FeesMustFall worked and in 2017 the then ANC president Zuma declared it in late 2017. I am sure that people who have been regular visitors to this blog know how my position is on student loans. They also must be knowing that even in U.S. till the 1970s it had free education all the way to be a lawyer and getting a lawyer license. It is only when people like Thurgood Marshall, Martin Luther King Jr., and others from the civil rights movement came out as a major force that the capitalists started imposing fees. They wanted people who could be sold to corporate slavery, and they won. Just last week, Biden took some steps and canceled student loans and is working on steps towards broad debt forgiveness.
Interestingly, NASA has an affirmative diversity program for people from diverse backgrounds, where a couple of UC (Upper Caste) women got the job. While they got the job, the RW (Right-Wing) was overjoyed as they got jobs on merit . Later, it was found that both the women were the third or fourth generation of immigrants in U.S.
NASA Federal Equal Opportunity Policy Directive NPD 3713 2H
Going back to the original question and topic, while there has been a concerning spate of violence, some calling it the worst sort of violence not witnessed since 1994. The problem, as ascertained in that article, is the same as here in India or elsewhere.
Those, again, who have been on my blog know that merit 90% of the time is a function of privilege and there is a vast amount of academic literature which supports that.
If, for a moment, you look at the data that is shared in the graph above which shows that 83% of Hindus and 13% of Muslims have more than 2 children, what does it show, it shows that 83+13 = 96% of the population is living in insecurity. The 5% are the ones who have actually consolidated more power during this regime rule in India. Similarly, from what I understood living in Cape Town for about a month, it is the Dutch Afrikaans as they like to call themselves and the immigrants who come from abroad who have enjoyed the fruits of tourism and money and power while the rest of the country is dying due to poverty. It is the same there, it is the same here. Corruption is also rampant in both countries, and the judiciary is virtually absent from both communities in India and SA. Interestingly, South Africa and India have been at loggerheads, but I suspect that is more due to the money and lobbying power by the Dutch. Usually, those who have money power, do get laws and even press on their side, and it is usually the ruling party in power. I cannot help but share about the Gupta brothers and their corruption as I came to know about it in 2016. And as have shared that I m related to Gupta s on my mother s side, not those specific ones but Gupta as a clan. The history of the Gupta dynasty does go back to the 3rd-4th century.
Equally interesting have been Sonali Ranade s series of articles which she wrote in National Herald, the latest on exports which is actually the key to taking India out of poverty rather than anything else. While in other countries Exporters are given all sort of subsidies, here it is being worked as how to give them less. This was in Economic times hardly a week back Export incentive schemes being reduced
I can t imagine the incredible stupidity done by the Finance Minister. And then in an attempt to prove that, they will attempt to present a rosy picture with numbers that have nothing to do with reality.
Interestingly enough, India at one time was a major exporter of apples, especially from Kashmir. Now instead of exporting, we are importing them from Afghanistan as well as Belgium and now even from the UK. Those who might not want to use the Twitter link could use this article. Of course, what India got out of this trade deal is not known. One can see that the UK got the better deal from this. Instead of investing in our own capacity expansion, we are investing in increasing the capacity of others. This is at the time when due to fuel price hike (Central taxes 66%) demand is completely flat. And this is when our own CEA (Chief Economic Adviser) tells us that growth will be at the most 6-7% and that too in 2023-2024 while currently, the inflation rate is around 12%. Is it then any wonder that almost 70% are living on Govt. ration and people in the streets of Kolkata, Assam, and other places have to sell kidneys to make sure they have some money for their kids for tomorrow. Now I have nothing against the UK but trade negotiation is an art. Sadly, this has been going on for the last few years. The politicians in India fool the public by always telling of future trade deals. Sadly, as any businessman knows, once you have compromised, you always have to compromise. And the more you compromise, the more you weaken the hand for any future trade deals. IIT pupil tries to sell kidney to repay loan, but no takers for Dalit organ.
The above was from yesterday s Times of India. Just goes to show how much people are suffering. There have been reports in vernacular papers of quite a few people from across regions and communities are doing this so they can live without pain a bit.
Almost all the time, the politicians are saved as only few understand international trade, the diplomacy and the surrounding geopolitics around it. And this sadly, is as much to do with basic education as much as it is to any other factor
Suli Deals
About a month back on the holy day of Ramzan or Ramadan as it is known in the west, which is beloved by Muslims, a couple of Muslim women were targeted and virtually auctioned. Soon, there was a flood and a GitHub repository was created where hundreds of Muslim women, especially those who have a voice and fearlessly talk about their understanding about issues and things, were being virtually auctioned. One week after the FIR was put up, to date none of the people mentioned in the FIR have been arrested. In fact, just yesterday, there was an open letter which was published by livelaw. I have saved a copy on WordPress just in case something does go wrong. Other than the disgust we feel, can t say much as no action being taken by GOI and police.
IT Rules 2021 and Big Media
After almost a year of sleeping when most activists were screaming hoarsely about how the new IT rules are dangerous for one and all, big media finally woke up a few weeks back and listed a writ petition in Madras High Court of the same. Although to be frank, the real writ petition was filed In February 2021, classical singer, performer T.M. Krishna in Madras High Court. Again, a copy of the writ petition, I have hosted on WordPress. On 23rd June 2021, a group of 13 media outlets and a journalist have challenged the IT Rules, 2021.
The Contention came from Digital News Publishers Association which is made up of the following news companies: ABP Network Private Limited, Amar Ujala Limited, DB Corp Limited, Express Network Pvt Ltd, HT Digital Streams Limited, IE Online Media Services Pvt Ltd, Jagran Prakashan Limited, Lokmat Media Private Limited, NDTV Convergence Limited, TV Today Network Limited, The Malayala Manorama Co (P) Ltd, Times Internet Limited, and Ushodaya Enterprises Private Limited. All the above are heavyweights in the markets where they operate. The reason being simple, when these media organizations came into being, the idea was to have self-regulation, which by and large has worked. Now, the present Govt. wants each news item to be okayed by them before publication. This is nothing but blatant misuse of power and an attempt at censorship. In fact, the Tamil Nadu BJP president himself made a promise of the same. And of course, what is true and what is a lie, only GOI knows and will decide for the rest of the country. If somebody remembers Joseph Goebbels at this stage, it is merely a coincidence. Anyways, 3 days ago Supreme Court on 14th July the Honorable Supreme Court asked the Madras High Court to transfer all the petitions to SC. This, the Madras High Court denied as cited/shared by Meera Emmanuel, a reporter who works with barandbench. The Court says nothing doing, let this happen and then the SC can entertain the motion of doing it that level. At the same time, they would have the benefit of Madras High Court opinion as well. It gave the center two weeks to file a reply. So, either of end-week of July or latest by August first week, we might be able to read the Center s reply on the same. The SC could do a forceful intervention, but it would lead to similar outrage as has been witnessed in the past when a judge commented that if the SC has to do it all, then why do we need the High Courts, district courts etc. let all the solutions come from SC itself. This was, admittedly, frustration on the part of the judge, but due in part to the needless intervention of SC time and time again. But the concerns had been felt around all the different courts in the country.
Sedition Law
A couple of days ago, the Supreme Court under the guidance of Honorable CJI NV Ramanna, entertained the PIL filed by Maj Gen S G Vombatkere (Retd.) which asked simply that the sedition law which was used in the colonial times by the British to quell dissent by Mahatma Gandhi and Bal Gangadhar Tilak during the Indian freedom struggle. A good background filler article can be found on MSN which tells about some recent cases but more importantly how historically the sedition law was used to quell dissent during India s Independence. Another article on MSN actually elaborates on the PIL filed by Maj Gen S. G. Vombatkere. Another article on MSN tells how sedition law has been challenged and changed in 10 odd countries. I find it equally sad and equally hilarious that the Indian media whose job is to share news and opinion on this topic is being instead of being shared more by MSN. Although, I would be bereft of my duty if I did not share the editorial on the same topic by the Hindu and Deccan Chronicle. Also, an interesting question to ask is, are there only 10 countries in the world that have sedition laws? AFAIK, there are roughly 200 odd countries as recognized by WTO. If 190 odd countries do not have sedition laws, it also tells a lot about them and a lot about the remaining 10. Also, it came to light that police are still filing laws under sec66A which was declared null and void a few years ago. It was replaced with section 124A if memory serves right and it has more checks and balances.
Danish Siddiqui, Pulitzer award-winning and death in Afghanistan
Before I start with Danish Siddiqui, let me share an anecdote that I think I have shared on the blog years ago about how photojournalists are. Again, those who know me and those who follow me know how much I am mad both about trains and planes (civil aviation). A few months back, I had shared a blog post about some of the biggest railway systems in the world which shows that privatization of Railways doesn t necessarily lead to up-gradation of services but definitely leads to an increase in tariff/fares. Just had a conversation couple of days ago on Twitter and realized that need to also put a blog post about civil aviation in India and the problems it faces, but I digress.
This was about a gentleman who wanted to take a photo of a particular train coming out of a valley at a certain tunnel at two different heights, one from below and one from above the train. This was several years ago, and while I did share that award-winning photograph then, it probably would take me quite a bit of time and effort to again look it up on my blog and share.
The logistics though were far more interesting and intricate than I had first even thought of. We came around a couple of days before the train was supposed to pass that tunnel and the valley. More than half a dozen or maybe more shots were taken throughout the day by the cameras. The idea was to see how much light was being captured by the cameras and how much exposure was to be given so that the picture isn t whitened out or is too black.
Weather is the strangest of foes for a photojournalist or even photographers, and the more you are in nature, the more unpredictable it is and can be. We were also at a certain height, so care had to be taken in case light rainfall happens or dew falls, both not good for digital cameras.
And dew is something which will happen regardless of what you want. So while the two days our gentleman cameraman fiddled with the settings to figure out correct exposure settings, we had one other gentleman who was supposed to take the train from an earlier station and apprise us if the train was late or not.
The most ideal time would be at 0600 hrs. When the train would enter the tunnel and come out and the mixture of early morning sun rays, dew, the flowers in the valley, and the train would give a beautiful effect. We could stretch it to maybe 0700 hrs.
Anything after that would just be useless, as it wouldn t have the same effect. And of all this depended on nature. If the skies were to remain too dark, nothing we could do about it, if the dewdrops didn t fall it would all be over.
On the day of the shoot, we were told by our compatriot that the train was late by half an hour. We sank a little on hearing that news. Although Photoshop and others can do touch-ups, most professionals like to take as authentic a snap as possible. Everything had been set up to perfection. The wide-angle lenses on both the cameras with protections were set up. The tension you could cut with a knife. While we had a light breakfast, I took a bit more and went in the woods to shit and basically not be there. This was too tensed up for me. Returned an hour to find everybody in a good mood. Apparently, the shoot went well. One of the two captured it for good enough. Now, this is and was in a benign environment where the only foe was the environment. A bad shot would have meant another week in the valley, something which I was not looking forward to. Those who have lived with photographers and photojournalists know how self-involved they can be in their craft, while how grumpy they can be if they had a bad shoot. For those, who don t know, it is challenging to be friends with such people for a long time. I wish they would scream more at nature and let out the frustrations they have after a bad shoot. But again, this is in a very safe environment.
Now let s cut to Danish Siddiqui and the kind of photojournalism he followed. He followed a much more riskier sort of photojournalism than the one described above. Krittivas Mukherjee in his Twitter thread shared how reporters in most advanced countries are trained in multiple areas, from risk assessment to how to behave in case you are kidnapped, are in riots, hostage situations, etc. They are also trained in all sorts of medical training from treating gunshot wounds, CPR, and other survival methods. They are supposed to carry medical equipment along with their photography equipment. Sadly, these concepts are unknown in India. And even then they get killed. Sadly, he attributes his death to the thrill of taking an exclusive photograph. And the gentleman s bio reads that he is a diplomat. Talk about tone-deafness
On another completely different level was Karen Hao who was full of empathy as she shared the humility, grace, warmth and kinship she describes in her interaction with the photojournalist. His body of work can be seen via his ted talk in 2020 where he shared a brief collage of his works. Latest, though in a turnaround, the Taliban have claimed no involvement in the death of photojournalist Danish Siddiqui. This could be in part to show the Taliban in a more favorable light as they do and would want to be showcased as progressive, even though they are forcing that all women within a certain age become concubines or marry the fighters and killing the minority Hazaras or doing vile deeds with them. Meanwhile, statements made by Hillary Clinton almost a decade, 12 years ago have come back into circulation which stated how the U.S. itself created the Taliban to thwart the Soviet Union and once that job was finished, forgot all about it. And then in 2001, it landed back in Afghanistan while the real terrorists were Saudi. To date, not all documents of 9/11 are in the public domain. One can find more information of the same here. This is gonna take probably another few years before Saudi Arabia s whole role in the September 11 attacks will be known.
Last but not the least, came to know about the Pegasus spyware and how many prominent people in some nations were targeted, including in mine India. Will not talk more as it s already a big blog post and Pegasus revelations need an article on its own.
A new (and genuinely) minor release of our ttdo package arrived on CRAN today. The ttdo package extends the most excellent (and very minimal / zero depends) unit testing package tinytest by Mark van der Loo with the very clever and well-done diffobj package by Brodie Gaslam to give us test results with visual diffs (as shown in the screenshot below) which seemingly is so compelling an idea that it eventually got copied by another package
This release cleans up one microscopic wart of an R warning when installing and byte-compiling the package due to a sprintf call with an unused argument.
And once again, this release gets a #ThankYouCRAN mark as it was processed in a fully automated and intervention-free manner in a matter of minutes.
As usual, the NEWS entry follows.
Armadillo is a powerful and expressive C++ template library for linear algebra aiming towards a good balance between speed and ease of use with a syntax deliberately close to a Matlab. RcppArmadillo integrates this library with the R environment and language and is widely used by (currently) 882 other packages on CRAN.
This new release gets us Armadillo 10.6.0 which was released yesterday. We did the usual reverse dependency checks (which came out spotless and clean), and had also just done even fuller checks for Rcpp 1.0.7.
Since the previous RcppArmadillo 0.10.5.0.0 release we made a few interim releases to the drat repo. In general, Conrad is a little more active than we want to be with (montly or less frequent) CRAN updates so keep and eye on the drat repo (or follow the GitHub repo) for a higher-frequence cadence. To use the drat repo, use install.packages("RcppArmadillo", repos="https://RcppCore.github.io/drat") or update.packages() with a similar repos argument.
The full set of changes follows. We include the last interim release as well.
Changes in RcppArmadillo version 0.10.6.0.0 (2021-07-16)
Upgraded to Armadillo release 10.6.0 (Keep Calm)
expanded chol() to optionally use pivoted decomposition
expanded vector, matrix and cube constructors to allow element initialisation via fill::value(scalar), eg. mat X(4,5,fill::value(123))
faster loading of CSV files when using OpenMP
added csv_opts::semicolon option to allow saving/loading of CSV files with semicolon (;) instead of comma (,) as the separator
Changes in RcppArmadillo version 0.10.5.3.0 (2021-07-01)
Upgraded to Armadillo release 10.5.3 (Antipodean Fortress)
GitHub-only release
Extended test coverage with several new tests, added a coverage badge.
Back in the days of Etch, I converted this blog from Drupal to
ikiwiki. I remember being very excited about this brand
new concept of static web sites derived from content stored in a version
control system.
And now over a decade later I ve moved to hugo.
I feel some loyalty to ikiwiki and Joey Hess for opening my eyes to the static
web site concept. But ultimately I grew tired of splitting my time and energy
between learning ikiwiki and hugo, which has been my tool of choice for new
projects. When I started getting strange emails that I suspect had something to
do with spammers filling out ikiwiki s commenting registration system, I choose
to invest my time in switching to hugo over debugging and really understanding
how ikiwiki handles user registration.
I carefully reviewed anarcat s blog on converting from ikiwiki to
hugo and learned
about a lot of ikiwiki features I am not using. Wow, it s times like these that
I m glad I keep it really simple. Based on the various ikiwiki2hugo python
scripts I studied, I eventually wrote a far simpler
one
tailored to my needs.
Also, in what could only be called a desperate act of procrastination combined
with a touch of self-hatred (it s been a rough week) I rejected all the
commenting options available to me and choose to implement my own in
PHP.
What?!?! Why would anyone do such a thing?
I refer you to my previous sentence about desperate procrastination. And
also I know it s fashionable to hate PHP, but honestly as the first
programming language I learned, there is something comforting and familiar
about it. And, on a more objective level, I can deploy it easily to just about
any hosting provider in the world. I don t have to maintain a unicorn service
or a nodejs service and make special configuration entries in my web
configuration. All I have to do is upload the php files and I m done.
Well, I m sure I ll regret this decision.
Special thanks to Alexander Bilz for the
anatole hugo theme. I choose it via a
nearly random click to avoid the rabbit hole of choosing a theme. And, by luck,
it has turned out quite well. I only had to override the commento partial theme
page to hijack it for my own commenting system s use.
Github recently announced Copilot, a machine learning system that makes suggestions for you when you're writing code. It's apparently trained on all public code hosted on Github, which means there's a lot of free software in its training set. Github assert that the output of Copilot belongs to the user, although they admit that it may occasionally produce output that is identical to content from the training set.
Unsurprisingly, this has led to a number of questions along the lines of "If Copilot embeds code that is identical to GPLed training data, is my code now GPLed?". This is extremely understandable, but the underlying issue is actually more general than that. Even code under permissive licenses like BSD requires retention of copyright notices and disclaimers, and failing to include them is just as much a copyright violation as incorporating GPLed code into a work and not abiding by the terms of the GPL is.
But free software licenses only have power to the extent that copyright permits them to. If your code isn't a derived work of GPLed material, you have no obligation to follow the terms of the GPL. Github clearly believe that Copilot's output doesn't count as a derived work as far as US copyright law goes, and as a result the licenses on the training data don't apply to the output. Some people have interpreted this as an attack on free software - Copilot may insert code that's either identical or extremely similar to GPLed code, and claim that there are no license obligations created as a result, effectively allowing the laundering of GPLed code into proprietary software.
I'm completely unqualified to hold a strong opinion on whether Github's legal position is justifiable or not, and right now I'm also not interested in thinking about it too much. What I think is more interesting is what the impact of either position has on free software. Do we benefit more from a future where the output of Copilot (or similar projects) is considered a derived work of the training data, or one where it isn't? Having been involved in a bunch of GPL enforcement activities, it's very easy to think of this as something that weakens the GPL and, as a result, weakens free software. That was my initial reaction, but that's shifted over the past few days.
Let's look at the GNU manifesto, specifically this section:
The fact that the easiest way to copy a program is from one neighbor to another, the fact that a program has both source code and object code which are distinct, and the fact that a program is used rather than read and enjoyed, combine to create a situation in which a person who enforces a copyright is harming society as a whole both materially and spiritually; in which a person should not do so regardless of whether the law enables him to.
The GPL makes use of copyright law to ensure that GPLed work can't be taken from the commons. Anyone who produces a derived work of GPLed code is obliged to provide that work under the same terms. If software weren't copyrightable, the GPL would have no power. But this is the outcome Stallman wanted! The GPL doesn't exist because copyright is good, it exists because software being copyrightable is what enables the concept of proprietary software in the first place.
The powers that the GPL uses to enforce sharing of code are used by the authors of proprietary software to reduce that sharing. They attempt to forbid us from examining their code to determine how it works - they argue that anyone who does so is tainted, unable to contribute similar code to free software projects in case they produce a derived work of the original. Broadly speaking, the further the definition of a derived work reaches, the greater the power of proprietary software authors. If Oracle's argument that APIs are copyrightable had prevailed, it would have been disastrous for free software. If the Apple look and feel suit had established that Microsoft infringed Apple's copyright, we might be living in a future where we had no free software desktop environments.
When we argue for an interpretation of copyright law that enhances the power of the GPL, we're also enhancing the power of giant corporations with a lot of lawyers on hand. So let's look at this another way. If Github's interpretation of copyright law holds, we can train a model on proprietary code and extract concepts without having to worry about being tainted. The proprietary code itself won't enter the commons, but the ideas it embodies will. No more worries about whether you're literally copying the code that implements an algorithm you want to duplicate - simply start typing and let the model remove the risk for you.
There's a reasonable counter argument about equality here. How much GPL-influenced code is going to end up in proprietary projects when compared to the reverse? It's not an easy question to answer, but we should bear in mind that the majority of public repositories on Github aren't under an open source license. Copilot is already claiming to give us access to the concepts embodied in those repositories. Do these provide more value than is given up? I honestly don't know how to measure that. But what I do know is that free software was founded in a belief that software shouldn't be constrained by copyright, and our default stance shouldn't be to argue against the idea that copyright is weaker than we imagined.
(Edit: this post by Julia Reda makes some of the same arguments, but spends some more time focusing on a legal analysis of why having copyright cover the output of Copilot would be a problem)
This is not only the Year of the Ox, but also the year of Debian 11, code-named
bullseye. The release lies ahead, full freeze starts this week.
A good opportunity to take a look at what is new in bullseye.
In this post new programs and new software versions related to
XMPP, also known as Jabber are presented.
XMPP exists since 1999, and has a diverse and active developers community. It is a
universal communication protocol, used for instant messaging, IoT, WebRTC, and social
applications.
You probably will encounter some oxen in this post.
biboumi, XMPP gateway to connect to IRC servers: 8.3 9.0
The biggest change for users is SASL support: A new field in the Configure ad-hoc
command lets you set a password that will be used to authenticate to the nick service,
instead of using the cumbersome NickServ method.
Many more changes are listed in the changelog.
Dino, modern XMPP client: 0.0.git20181129 0.2.0
Dino in Debian 10 was practically a technology preview. In Debian 11 it is already a
fully usable client, supporting OMEMO encryption, file upload, image preview, message
correction and many more features in a clean and beautiful user interface.
ejabberd, the extensible realtime platform: 18.12.1 21.01.
Probably the most important improvement for end-users is
XEP-0215 support to facilitate modern WebRTC-style audio/video calls.
ejabberd also integrates more nicely with systemd (e.g., the watchdog feature if
supported, now). Apart from that, a new configuration validator was introduced, which
brings a more flexible (but mostly backwards-compatible) syntax. Also, error reporting
in case of misconfiguration should be way more helpful, now. As a new authentication
backend, JSON Web Tokens (JWT) can be used. In addition to the XMPP and SIP support,
ejabberd now includes a full-blown MQTT server. A large number of smaller features has
been added, performance was improved in many ways, and several bugs were fixed.
See the long list of changes.
Gajim, a GTK+-based Jabber client: 1.1.2 1.3.1
The new Debian release brings many improvements. Gajim s network code has been
completely rewritten, which leads to faster connections, better recovery from network
loss, and less network related hiccups. Customizing Gajim is now easier than ever.
Thanks to the new settings backend and a completely reworked Preferences window, you
can adapt Gajim to your needs in just a few seconds.
Good for newcomers: account creation is now a lot easier with Gajim s new assistant.
The new Profile window gives you many options to tell people more about yourself. You
can now easily crop your own profile picture before updating it.
Group chats actions have been reorganized. It s now easier to send invitations or
change your nickname for example. Gajim also received support for chat markers, which
enables you to see how far your contact followed the conversation. But this is by far
not everything the new release brings. There are many new and helpful features, such
as pasting images from your clipboard directly into the chat or playing voice messages
directly from the chat window.
Read more about the new Gajim release in Debian 11 here.
Furthermore, three more Gajim plugins are now in Debian: gajim-lengthnotifier,
gajim-openpgp for OX (XEP-0373: OpenPGP for XMPP) and gajim-syntaxhighlight.
NEW Kaidan Simple and user-friendly Jabber/XMPP client 0.7.0
Kaidan is a simple, user-friendly and modern XMPP chat client. The user interface
makes use of Kirigami and QtQuick, while the back-end of Kaidan is entirely written in
C++ using Qt and the Qt-based XMPP library QXmpp. Kaidan runs on mobile and desktop
systems including Linux, Windows, macOS, Android, Plasma Mobile and Ubuntu Touch.
mcabber, small Jabber (XMPP) console client: 1.1.0 1.1.2
A theme for 256 color terminals is now included, the handling of carbon message copies
has been improved, and various minor issues have been fixed.
Poezio, Console-based XMPP client: 0.12.1 0.13.1
This new release brings many improvements, such as Message Archive (XEP-0313) support,
initial support for OMEMO (XEP-0384) through a plugin, HTTP File Upload support,
Consitent Color Generation (XEP-0392), and plenty of internal changes and bug fixes.
Not all changes in 0.13 and 0.13.1 can be listed, see the CHANGELOG
for a more extensive summary.
Profanity, the console based XMPP client: 0.6.0 0.10.0
We can not list all changes which have been done,
but here are some highlights.
Support of OMEMO Encryption (XEP-0384). Consistent Color Generation (XEP-0392),
be aware of the changes in the command to standardize the names of commands. A
clipboard feature has been added. Highlight unread messages with a different
color in /wins. Keyboard switch to select the next window with unread messages
with alt + a. Support for Last Message Correction (XEP-0308), Allow UTF-8
symbols as OMEMO/OTR/PGP indicator char. Add option to open avatars directly
(XEP-0084). Add option to define a theme at startup and some changes to improve
themes. Add possibility to easily open URLs. Add experimental OX (XEP-0373,
XEP-0374) support. Add OMEMO media sharing support, ...
There is also a Profanity light package in Debian now, the
best option for systems with tight limits on resources.
Prosody, the lightweight extensible XMPP server: 0.11.2 0.11.9
Upgrading to the latest stable release of Prosody brings a whole load of
improvements in the stability, usability and performance departments. It especially
improves the performance of websockets, and PEP performance for users with many
contacts. It includes interoperability improvements for a range of clients.
Psi, Qt-based XMPP client: 1.3 1.5
The new version contains important bug fixes.
salutatoi, multi-frontends, multi-purposes communication tool: 0.7.0a4 0.8.0~hg3453
This version is now fully running on Python 3, and has full OMEMO support (one2one,
groups and files). The CLI frontend (jp) has among new commands a "jp file get" one
which is comparable to wget with OMEMO support. A file sharing component is included,
with HTTP Upload and Jingle support. For a list of other improvements, please consult
the changelog.
Note, that the upstream project has been renamed to "Libervia".
NEW sms4you, Personal gateway connecting SMS to XMPP or email 0.0.7
It runs with a GSM device over ModemManager and uses a lightweight
XMPP server or a single email account to handle communication in both directions.
NEW xmppc, XMPP Command Line Client 0.1.0
xmppc is a new command line tool for XMPP. It supports some
basic features of XMPP (request your roster, bookmarks, OMEMO Devices and
fingerprints). You can send messages with both legacy PGP (XEP-0027) and the
new OX (XEP-0373: OpenPGP for XMPP).
That's all for now. Enjoy Debian 11 bullseye and Happy Chatting!
How are you meant to write about an unfinished emancipation? Bodies of Light is a 2014 book by Glasgow-born Sarah Moss on the stirrings of women's suffrage in an arty clique in nineteenth-century England. Set in the intellectually smoggy cities of Manchester and London, we follow the studious and intelligent Alethia 'Ally' Moberly, who is struggling to gain the acceptance of herself, her mother and the General Medical Council.
'Alethia' may be the Greek goddess of truth, but our Ally is really searching for wisdom. Her strengths are her patience and bookish learning, and she acquires Latin as soon as she learns male doctors will use it to keep women away from the operating theatre. In fact, Ally's acquisition of language becomes a recurring leitmotif: replaying a suggestive dream involving a love interest, for instance, Ally thinks of 'dark, tumbling dreams for which she has a perfectly adequate vocabulary'. There are very few moments of sensuality in the book, and pairing it with Ally's understated wit achieves a wonderful effect.
The amount we learn about a character is adapted for effect as well. There are few psychological insights about Ally's sister, for example, and she thus becomes a fey, mysterious and almost Pre-Raphaelite figure below the surface of a lake to match the artistic movement being portrayed. By contrast, we get almost the complete origin story of Ally's mother, Elizabeth, who also constitutes of those rare birds in literature: an entirely plausible Christian religious zealot. Nothing Ally does is ever enough for her, but unlike most modern portrayals of this dynamic, neither of them are aware of what is going, and it is conveyed in a way that is chillingly... benevolent. This was brought home in the annual 'birthday letters' that Elizabeth writes to her daughter:
Last year's letter said that Ally was nervous, emotional and easily swayed, and that she should not allow her behaviour to be guided by feeling but remember always to assert her reason. Mamma would help her with early hours, plain food and plenty of exercise. Ally looks at the letter, plump in its cream envelope. She hopes Mamma wrote it before scolding her yesterday.
The book makes the implicit argument that it is a far more robust argument against pervasive oppression to portray a character in, say, 'a comfortable house, a kind husband and a healthy child', yet they are nonetheless still deeply miserable, for reasons they can't quite put their finger on. And when we see Elizabeth perpetuating some generational trauma with her own children, it is telling that is pattern is not short-circuited by an improvement in their material conditions. Rather, it is arrested only by a kind of political consciousness in Ally's case, the education in a school. In fact, if there is a real hero in Bodies of Light, it is the very concept of female education.
There's genuine shading to the book's ideological villains, despite finding their apotheosis in the jibes about 'plump Tories'. These remarks first stuck out to me as cheap thrills by the author; easy and inexpensive potshots that are unbecoming of the pages around them. But they soon prove themselves to be moments of much-needed humour. Indeed, when passages like this are read in their proper context, the proclamations made by sundry Victorian worthies start to serve as deadpan satire:
We have much evidence that the great majority of your male colleagues regard you as an aberration against nature, a disgusting, unsexed creature and a danger to the public.
Funny as these remarks might be, however, these moments have a subtler and more profound purpose as well. Historical biography always has the risk of allowing readers to believe that the 'issue' has already been solved hence, perhaps, the enduring appeal of science fiction. But Moss providing these snippets from newspapers 150 years ago should make a clear connection to a near-identical moral panic today.
On the other hand, setting your morality tale in the past has the advantage that you can show that progress is possible. And it can also demonstrate how that progress might come about as well. This book makes the argument for collective action and generally repudiates individualisation through ever-fallible martyrs. Ally always needs 'allies' not only does she rarely work alone, but she is helped in some way by almost everyone around her. This even includes her rather problematic mother, forestalling any simplistic proportioning of blame. (It might be ironic that Bodies of Light came out in 2014, the very same year that Sophia Amoruso popularised the term 'girl boss'.) Early on, Ally's schoolteacher is coded as the primary positive influence on her, but Ally's aunt later inherits this decisive role, continuing Ally's education on cultural issues and what appears to be the Victorian version of 'self-care'. Both the aunt and the schoolteacher are, of course, surrogate mother figures.
After Ally arrives in the cut-throat capital, you often get the impression you are being shown discussions where each of the characters embodies a different school of thought within first-wave feminism. This can often be a fairly tedious device in fiction, the sort of thing you would find in a Sally Rooney novel, Pilgrim's Progress or some other ponderously polemical tract. Yet when Ally appears to 'win' an argument, it is only in the sense that the narrator continues to follow her, implicitly and lightly endorsing her point. Perhaps if I knew my history better, I might be able to associate names with the book's positions, but perhaps it is better (at least for the fiction-reading experience...) that I don't, as the baggage of real-world personalities can often get in the way. I'm reminded here of Regina King's One Night in Miami... (2020), where caricatures of Malcolm X, Muhammad Ali, Jim Brown and Sam Cooke awkwardly replay various arguments within an analogous emancipatory struggle.
Yet none of the above will be the first thing a reader will notice. Each chapter begins with a description of an imaginary painting, providing a title and a date alongside a brief critical exegesis. The artworks serve a different purpose in each chapter: a puzzle to be unlocked, a fear to be confirmed, an unsolved enigma. The inclusion of (artificial) provenances is interesting as well, not simply because they add colour and detail to the chapter to come, but because their very inclusion feels reflective of how we see art today.
Orphelia (1852) by Sir John Everett Millais.
To continue the question this piece began, how should an author conclude a story about an as-yet-unfinished struggle for emancipation? How can they? Moss' approach dares you to believe the ending is saccharine or formulaic, but what else was she meant to turn in yet another tale of struggle and suffering? After all, Thomas Hardy has already written Tess of the d'Urbervilles. All the same, it still feels slightly unsatisfying to end merely with Ally's muted, uncelebrated success.
Nevertheless, I suspect many readers will dislike the introduction of a husband in the final pages, taking it as a betrayal of the preceding chapters. Yet Moss denies us from seeing the resolution as a Disney-style happy ending. True, Ally's husband turns out to be a rather dashing lighthouse builder, but isn't it Ally herself who is lighting the way in their relationship, warning other women away from running aground on the rocks of mental illness? And Tom feels more of a reflection of Ally's newly acquired self-acceptance instead of that missing piece she needed all along. We learn at one point that Tom's 'importance to her is frightening' this is hardly something a Disney princess would say.
In fact, it is easy to argue that a heroic ending for Ally might have been an even more egregious betrayal. The evil of saints is that you can never live up to them, for the concept of a 'saint' embodies an unreachable ideal that no human can begin to copy. By being taken as unimpeachable and uncorrectable as well, saints preclude novel political action, and are therefore undoubtedly agents of reaction. Appreciating historical figures as the (flawed) people that they really were is the first step if you wish to continue or adapt their political ideas.
I had acquired Bodies of Light after enjoying Moss' Summerwater (2020), which had the dubious honour of being touted as the 'first lockdown novel', despite it being finished before Covid-19. There are countless ways one might contrast the two, so I will limit myself to the sole observation that the strengths of one are perhaps the weaknesses of the other. It's not that Bodies of Light ends with a whimper, of course, as it quietly succeeds in concert with Ally. But by contrast, the tighter arc of Summerwater (which is set during a single day, switches protagonist between chapters, features a closed-off community, etc.) can reach a higher high with its handful of narrative artifices. Summerwater is perhaps like Phil Collins' solo career: 'more satisfying, in a narrower way.'
A new minor release of drat arrived on CRAN overnight. This is a minor update relative to the 0.2.0 release in April. This release will now create an empty file index.html in the top-level (when initRepo() is called), and check for presence of such a file when adding files to a repo (via insertPackage()). This helps to avoid getting 404 results when (perfectly valid) drat repos are checking by accessing the top-level URL, as for example CRAN does when testing if an Additional_repositoiries is reachable. The step-by-step vignette had already suggested creating one by hand, this is now done programmatically (and one is present in the repo suggsted to fork from too).
drat stands for drat R Archive Template, and helps with easy-to-create and easy-to-use repositories for R packages. Since its inception in early 2015 it has found reasonably widespread adoption among R users because repositories with marked releases is the better way to distribute code. See below for a few custom reference examples.
Because for once it really is as your mother told you: Friends don t let friends install random git commit snapshots. Properly rolled-up releases it is. Just how CRAN shows us: a model that has demonstrated for two-plus decades how to do this. And you can too: drat is easy to use, documented by six vignettes and just works.
The NEWS file summarises the release as follows:
Changes in drat version 0.2.1 (2021-07-09)
Two internal functions now have a note in their documentation stating them as not exported (Dirk in response to #123)
Repositories created by initRepo now have an placeholder index.html to not trigger a curl check at CRAN (Dirk)
Adding to a repository now checks for a top-level index.html and displays a message if missing (Dirk)
The DratStepByStep.Rmd vignette mentions the added index.html file
The Rcpp team is pleased to announce release 1.0.7 of Rcpp which arrived at CRAN earlier today, and will be uploaded to Debian shortly. Windows and macOS builds should appear at CRAN in the next few days. This release continues with the six-months cycle started with release 1.0.5 last July. As a reminder, interim dev or rc releases will alwasys be available in the Rcpp drat repo; this cycle there were seven (!!). These rolling release tend to work just as well, and are also fully tested against all reverse-dependencies.
Rcpp has become the most popular way of enhancing R with C or C++ code. As of today, 2323 packages on CRAN depend on Rcpp for making analytical code go faster and further, along with 227 in BioConductor.
This release contains a change which Luke Tierney urged us to make a good year ago in #1081) (and which we had looked at earlier in #382). Implementing the change in a regular update proved a little tricky, and my initial branch lay dormant until I aki revived it, and finished the transition (which we then did in two PRs). The change concerns how Rcpp grows internal objects, and the new approach (thanks to the hint by Luke) closer to what R does guaranteeing linear behaviour. It turns out that we overlooked one aspect (of coping with Modules built under earlier Rcpp releases) so the initial upload to CRAN on Saturday revealed that we needed a small adjustment that we made for the final release. This version should now be more performant, and rest on a stable API. Based on the reverse depends checks by both us and CRAN (using the updated version), we expect no issues with existing code. However, it something does act up a fresh compilation of the packages using Rcpp may help.
We also made a few other minor changes in the API such as silencing a silly compiler warning, ensuring global Rcout and Rcerr objects, adding support for a new Rcpp::message() call, completing a switch to uint32_t instead of unsigned int and including the cfloat header (which relates to STRICT_R_HEADERS discussed below). Similarly, the Rcpp Attributes feature was enhanced by coping better with packages with a dot in their name and their for per-package include files, along with support for more quiet compilation if desired.
As some Rcpp users may have seen, we plan to enable STRICT_R_HEADERS by the next release (expected in January 2022). A long issue tick #1158 is laying the ground work. Maintainers of 81 packages which are affected and need a small change (such as for example switching from PI to M_PI); of these 56 have already responded. We plan to be in touch in the fall. Adding the cfloat header is one help in this transition (as the corresponding C header was pulled in when STRICT_R_HEADERS is not defined) as it ensures DBL_EPSILON and alike are defined.
Last but not least this is also the first relase in which we welcome I aki as a new member of the Rcpp Core team. Yay!
The NEWS file entries follow summarizing the nine key PRs in this release.
Changes in Rcpp release version 1.0.7 (2021-07-06)
Changes in Rcpp API:
Refactored Rcpp_PreserveObject and Rcpp_ReleaseObject which are now O(1) (Dirk and I aki in #1133 and #1135 fixing #382 and #1081).
A spuriously assigned variable was removed (Dirk in #1138 fixing #1137).
Global Rcout and Rcerr objects are supported via a compiler directive (I aki in #1139 fixing #928)
Add support for Rcpp::message (Dirk in #1146 fixing #1145).
The uint32_t type is used throughout instead of unsigned int (Dirk in #1153 fixing #1152).
The cfloat header for floating point limits is now included (Dirk in #1162 fixing #1161).
Changes in Rcpp Attributes:
Packages with dots in their name can now have per-package include files (Dirk in #1132 fixing #1129).
New argument echo to quieten optional evaluation in sourceCpp (Dirk in #1138 fixing #1126).
Forthcoming Changes in Rcpp API:
Starting with Rcpp 1.0.8 anticipated in January 2022, STRICT_R_HEADERS will be enabled by default, see #1126.
A new minor release of our ttdo package arrived on CRAN today. The ttdo package extends the most excellent (and very minimal / zero depends) unit testing package tinytest by Mark van der Loo with the very clever and well-done diffobj package by Brodie Gaslam to give us test results with visual diffs (as shown in the screenshot here) which seemingly is so compelling an idea that another package decided to copied it more recently:
This release is mostly procedural to accomodate changes in tinytest 1.3.1 released today, and brought to us via a pull request by Mark himself. Other than that we also updated the CI runner to use r-ci and accomodated new CRAN check for a superfluous LazyData: field in a package without a data/ directory.
This release also gets another #ThankYouCRAN mark as it was once again fully automated and intervention-free (once the new tinytest release hit CRAN).
As usual, the NEWS entry follows.
Changes in ttdo version 0.0.7 (2021-07-06)
The CI setup was updated to use run.sh from r-ci (Dirk).
The package was updated for an API extension in tinytest 1.3.1 or later (Mark van der Look in #7)
The unused LazyData field was removed from DESCRIPTION (Dirk)
Continuing with #newinbullseye. One package that isn t new but its tools are used by many of us is util-linux, providing many essential system utilities. There is util-linux v2.33.1 in Debian/buster and util-linux v2.36.1 in Debian/bullseye, and as usual there are many new features and options available.
I don t want to replicate the release notes provided by upstream, instead make sure to check out the Release highlights sections in the following release notes:
Tools that have been taken over from / moved to other packages
Debian s util-linux source package provides new binary packages: eject (and eject-udeb) and bsdextrautils. The util-linux implementation of /usr/bin/eject is used now, replacing the one previously provided by the eject source package.
Overall, from a util-linux perspective the following shifts took place:
col, colcrt, colrm, column: moved from binary package bsdmainutils to bsdextrautils
eject: moved to binary package eject
hd: moved from binary package bsdmainutils to bsdextrautils
hexdump: moved from binary package bsdmainutils to bsdextrautils
look: moved from binary package bsdmainutils to bsdextrautils
ul: moved from binary package bsdmainutils to bsdextrautils
write(.ul): moved from binary package bsdmainutils (named bsd-write) to bsdextrautils
Deprecated / removed tools
Tools that are no longer shipped as of Debian/bullseye:
/usr/bin/rename.ul (rename files): use e.g. rename package instead, see #926637 for details regarding the removal
/usr/bin/volname (return volume name for a device formatted with an ISO-9660 file system): use blkid -s LABEL -o value $filename instead
/usr/lib/eject/dmcrypt-get-device: no replacement available
New tools
Debian s bsdutils package (which is provided by the util-linux source package) provides a new tool from util-linux:
scriptlive: re-execute stdin log by a shell in PTY session
The new tools lsirq + irqtop (to monitor kernel interrupts) sadly didn t make it into util-linux s packaging of Debian/bullseye (as without per-CPU data they do not seem mature at this time). The new hardlink tool (to consolidate duplicate files via hardlinks) won t be shipped, as there s an existing hardlink package already.
New features/options
agetty + getty:
--show-issue display issue file and exit
blkdiscard:
--force disable all checking
blkid:
-D, --no-part-details don't print info from partition table
blkzone:
Commands:
open Open a range of zones.
close Close a range of zones.
finish Set a range of zones to Full.
Options:
-f, --force enforce on block devices used by the system
cfdisk:
--lock[=<mode>] use exclusive device lock (yes, no or nonblock)
dmesg:
--noescape don't escape unprintable character
-W, --follow-new wait and print only new messages
fdisk:
-x, --list-details like --list but with more details
-n, --noauto-pt don't create default partition table on empty devices
--lock[=<mode>] use exclusive device lock (yes, no or nonblock)
fstrim:
-I, --listed-in <list> trim filesystems listed in specified files
--quiet-unsupported suppress error messages if trim unsupported
lsblk:
Options:
-E, --dedup <column> de-duplicate output by <column>
(for example 'lsblk --dedup WWN' to de-duplicate devices by WWN number, e.g. multi-path devices)
-M, --merge group parents of sub-trees (usable for RAIDs, Multi-path)
see http://karelzak.blogspot.com/2018/11/lsblk-merge.html
New output columns:
FSVER filesystem version
PARTTYPENAME partition type name
DAX dax-capable device
lscpu:
Options:
-B, --bytes print sizes in bytes rather than in human readable format
-C, --caches[=<list>] info about caches in extended readable format
--output-all print all available columns for -e, -p or -C
Available output columns for -C:
ALL-SIZE size of all system caches
LEVEL cache level
NAME cache name
ONE-SIZE size of one cache
TYPE cache type
WAYS ways of associativity
ALLOC-POLICY allocation policy
WRITE-POLICY write policy
PHY-LINE number of physical cache line per cache t
SETS number of sets in the cache; set lines has the same cache index
COHERENCY-SIZE minimum amount of data in bytes transferred from memory to cache
lslogins:
--lastlog <path> set an alternate path for lastlog
lsns:
-t, --type time namespace type time is also supported now (next to mnt, net, ipc, user, pid, uts, cgroup)
mkswap:
--lock[=<mode>] use exclusive device lock (yes, no or nonblock)
more:
Options:
-n, --lines <number> the number of lines per screenful
New long options (in addition to the listed equivalent short options):
--silent - equivalent to -d
--logical - equivalent to -f
--no-pause - equivalent to -l
--print-over - equivalent to -c
--clean-print - equivalent to -p
--squeeze - equivalent to -s
--plain - equivalent to -u
mount:
Options:
--target-prefix <path> specifies path use for all mountpoints
Source:
ID=<id> specifies device by udev hardware ID
mountpoint:
--nofollow do not follow symlink
nsenter:
-T, --time[=<file>] enter time namespace
script:
-I, --log-in <file> log stdin to file
-O, --log-out <file> log stdout to file (default)
-B, --log-io <file> log stdin and stdout to file
-T, --log-timing <file> log timing information to file
-m, --logging-format <name> force to 'classic' or 'advanced' format
-E, --echo <when> echo input (auto, always or never)
sfdisk:
--disk-id <dev> [<str>] print or change disk label ID (UUID)
--relocate <oper> <dev> move partition header
--move-use-fsync use fsync after each write when move data
--lock[=<mode>] use exclusive device lock (yes, no or nonblock)
unshare:
-T, --time[=<file>] unshare time namespace
--map-user=<uid> <name> map current user to uid (implies --user)
--map-group=<gid> <name> map current group to gid (implies --user)
-c, --map-current-user map current user to itself (implies --user)
--keep-caps retain capabilities granted in user namespaces
-R, --root=<dir> run the command with root directory set to <dir>
-w, --wd=<dir> change working directory to <dir>
-S, --setuid <uid> set uid in entered namespace
-G, --setgid <gid> set gid in entered namespace
--monotonic <offset> set clock monotonic offset (seconds) in time namespaces
--boottime <offset> set clock boottime offset (seconds) in time namespaces
wipefs:
--lock[=<mode>] use exclusive device lock (yes, no or nonblock)
I woke up this morning to a lovely little gallery of pictures of our children that my wife had sent me via WhatsApp.
This has become the most common way we interact with family photos. We regularly send and receive photos to and from our families via WhatsApp, which re-compresses them for transit and temporary storage across their network.
The original photos, wherever they are, will be in a very high quality (as you get on most modern cameras) and will be backed up in perfect fidelity to either Apple or Google s photo storage solutions. But all of that seems moot, when the most frequent way we engage with the pictures is via a method which compresses so aggressively that you can clearly see the artefacts, even thumbnailed on a phone screen.
I still don t feel particularly happy with the solution in place for backing up the photos (or even: getting them off the phone). Both Apple and Google make it less than convenient to get them out of their respective walled gardens. I ve been evaluating the nextCloud app and a Nextcloud instance on my home NAS as a possible alternative.
OS security features and server class systems are things that surely belong together. If a program is important enough to buy expensive servers to run it then it s important enough that you want to have all the OS security features enabled. For such an important program you will also want to have all possible monitoring systems running so you can predict hardware failures etc. Therefore you would expect that you could buy a server, setup the vendor s management software, configure your Linux kernel with security features such as lockdown (a LSM that restricts access to /dev/mem, the iopl() system call, and other dangerous things [1]), and have it run nicely! You will be disappointed if you try doing that on a HP or Dell server though.
HP Problems
[370742.622525] Lockdown: hpasmlited: raw io port access is restricted; see man kernel_lockdown.7
The above message is logged when trying to INSTALL (not even run) the hp-health package from the official HP repository (as documented in my previous blog post about the HP ML-110 Gen9 [2]) with lockdown=integrity (the less restrictive lockdown option). Now the HP package in question is in their repository for Debian/Stretch (released in 2017) and the Lockdown LSM was documented by LWN as being released in 2019, so not supporting a Debian/Bullseye feature in Debian/Stretch packages isn t inherently a bad thing apart from the fact that they haven t released a new version of that package since. The Stretch package that I am testing now was released in 2019. Also it s been regarded as best practice to have device drivers for this sort of thing since long before 2017.
# hplog -v
ERROR: Could not open /dev/cpqhealth/cdt.
Please make sure the Health Monitor is started.
Attempting to run the hplog -v command (to view the HP hardware log) gives the above error. Strace reveals that it could and did open /dev/cpqhealth/cdt but had problems talking to something (presumably the Health Monitor daemon) over a Unix domain socket. It would be nice if they could at least get the error message right!
Dell Problems
[ 13.811165] Lockdown: smbios-sys-info: /dev/mem,kmem,port is restricted; see man kernel_lockdown.7
[ 13.820935] Lockdown: smbios-sys-info: raw io port access is restricted; see man kernel_lockdown.7
[ 18.118118] Lockdown: dchcfg: raw io port access is restricted; see man kernel_lockdown.7
[ 18.127621] Lockdown: dchcfg: /dev/mem,kmem,port is restricted; see man kernel_lockdown.7
[ 19.371391] Lockdown: dsm_sa_datamgrd: raw io port access is restricted; see man kernel_lockdown.7
[ 19.382147] Lockdown: dsm_sa_datamgrd: /dev/mem,kmem,port is restricted; see man kernel_lockdown.7
Above is a sample of the messages when booting a Dell PowerEdge R710 with lockdown=integrity with the srvadmin-omacore package installed from the official Dell repository I describe in my blog post about the Dell PowerEdge R710 [3]. Now that repository is for Ubuntu/Xenial which was released in 2015, but again it was best practice to have device drivers for this many years ago. Also the newest Debian based releases that Dell apparently supports are Ubuntu/Xenial and Debian/Jessie which were both released in 2015.
# omreport system esmlog
Error! No Embedded System Management (ESM) log found on this system.
Above is the result when I try to view the ESM log (the Dell hardware log).
How Long Should Server Support Last?
The Wikipedia List of Dell PowerEdge Servers shows that the R710 is a Generation 11 system. Generation 11 was first released in 2010 and Generation 12 was first released in 2012. Generation 13 was the latest hardware Dell sold in 2015 when they apparently ceased providing newer OS support for Generation 11. Dell currently sells Generation 15 systems and provides more recent support for Generation 14 and Generation 15. I think it s reasonable to debate whether Dell should support servers for 4 generations. But given that a major selling point of server class systems is that they have long term support I think it would make sense to give better support for this and not drop support when it s only 2 versions from the latest release! The support for Dell Generation 11 hardware only seems to have lasted for 3 years after Generation 12 was first released. Also it appears that software support for Dell Generation 13 ceased before Generation 14 was released, that sucks for the people who bought Generation 13 when they were new!
HP is currently selling Gen 10 servers which were first released at the end of 2017. So it appears that HP stopped properly supporting Gen 9 servers as soon as Gen 10 servers were released!
One thing to note about these support times, when the new generation of hardware was officially released the previous generation was still on sale. So while HP Gen 10 servers officially came out in 2017 that doesn t necessarily mean that someone who wanted to buy a ML-110 Gen10 could actually have done so.
For comparison Red Hat Enterprise Linux has been supported for 4-6 years for every release they made since 2005 and Ubuntu has always had a 5 year LTS support for servers.
How To Do It Properly
The correct way of interfacing with hardware is via a device driver that is supported in the kernel.org tree. That means it goes through the usual kernel source code quality checks which are really good at finding bugs and gives users an assurance that the code won t cause security problems. Generally nothing about the code from Dell or HP gives me confidence that it should be directly accessing /dev/kmem or raw IO ports without risk of problems.
Once a driver is in the kernel.org tree it will usually stay there forever and not require further effort from the people who submit it. Then it just works for everyone and tends to work with any other kernel features that people use, like LSMs.
If they released the source code to the management programs then it would save them even more effort as they could be maintained by the community.
As part of my GSoC project, my first task includes that user could login into their account on debci using their Debian Salsa account (collaborative development server for Debian based on the GitLab software).
The task is officially completed using OmniAuth library and while implementing it, I found that the documentation of OmniAuth is quite a mix-match and more focused on using it with Rails app and this gives me a idea to write a tutorial for people looking to integrate OmniAuth with Sinatra application. So here it is.
Now, depending on the provider, Omniauth requires specific strategy which are generally released individually as RubyGems. For this tutorial, I would be using omniauth-gitlab which I used for Debian Salsa in my project, omniauth-twitter and a developer strategy which could be used for project in development mode and comes with omniauth gem itself.
For simplicity purposes, I have included all routes and OmniAuth configurations in a single file app.rb.
Let s start.
Register your application
This can be easily done just head over to provider (Twitter, Salsa) website and find the option to create a new application and fill in the form. In the callback URL field, you need to append /auth/:provider/callback to whichever URL you used in the website field.
The client-id and client-secret is obtained from the console which is used in further step to set up OmniAuth.
Gems
At the top of the file, we require the necessary gems of for our project
Enable sessions
In order for OmniAuth to work and to store the logged in user across requests, sessions need to be activated and if activated, you have one session hash per user session.
configure do
set :sessions, trueend
Set up OmniAuth configurations
OmniAuth::Builder Rack middleware build up your list of OmniAuth strategies for use in your application:
Here, few things to note could be the extra options used with providers, as
In developer: fields : to specify the form fields for login in developer mode and by default it has name and email. uid_field: to specify that which field s value could be obtained as uid and by default it is email.
In gitlab: scope: to limit the scope of application, by default, the api scope is requested and must be allowed in GitLab s application configuration. client_options: to specify the server being used as client based on Gitlab software.
Note: In case you want to have a different callback url other than the default /auth/:provider/callback, it can be specified using redirect_url option in case of gitlab provider and accordingly update it in your Application Configuration at provider s console.
Extra configurations
To redirect to auth/failure route in case of failure even in developer mode, following could be added:
OmniAuth.config.on_failure = proc doenvOmniAuth::FailureEndpoint.new(env).redirect_to_failure
end
By default, OmniAuth will log to STDOUT but you can configure this. If you don t want OmniAuth to log to STDOUT, following could be used:
OmniAuth.config.logger.level =Logger::UNKNOWN
Setting up routes
Login route
Starting with GET /login route, where you can specify the options available to login:
get '/login'do<<~HTML<form method='post' action='/auth/gitlab'><input type="hidden" name="authenticity_token" value='# request.env["rack.session"]["csrf"] '><button type='submit'>Login with Salsa</button>
</form><form method='post' action='/auth/twitter'><input type="hidden" name="authenticity_token" value='# request.env["rack.session"]["csrf"] '><button type='submit'>Login with Twitter</button>
</form><form method='post' action='/auth/developer'><input type="hidden" name="authenticity_token" value='# request.env["rack.session"]["csrf"] '><button type='submit'>Login with Developer</button>
</form>HTMLend
The auth/:provider path is created and configured automatically by OmniAuth, so you just need to send the request to that paths and auth process will start. As per the OmniAuth version 2.0, OmniAuth now defaults to only POST as allowed request_phase methods and authenticity_token is required to validate your requests so make sure to take care of this.
Callback routes:
On success from authentication, Omniauth will return the hash of information to the auth/:provider/callback in the Rack environment under the key omniauth.auth so this is what you can use in your desired way like creating a entry to your database and storing the current user in session params.
get '/auth/:provider/callback'do
erb "
<h1>Hello # request.env['omniauth.auth']['info']['name']</h1>"end
post '/auth/developer/callback'do
erb "
<h1>Hello # request.env['omniauth.auth']['info']['name']</h1>"end
Here, POST request method is used for developer strategy and GET request method for twitter and gitlab as that is how their working is defined in their respective strategies.
Failure route:
If user authentication fails on the provider side, OmniAuth will catch the response and then redirect the request to the path /auth/failure, passing a corresponding error message in a parameter named message.
get '/auth/failure'do
halt(403, erb("<h2>Authentication Failed</h2><h4>Reason: </h4><pre># params[:message]</pre>"))
end
Final result
That s it, our application is all set to be tested. And this is how it will be working.
Here, I have already logged into my respective accounts so page for entring credentials does not show up but if the user has not logged into his account, he would be first asked to log in.
For complete code, you can check out:
OmniAuth with Sinatra Tutorial
So this completes the tutorial. I hope it helps others who are looking to integrate their Ruby Applications with OmniAuth. If you have any feedback, feel free to let me know.
See you next time!
CVE-2020-27751
A flaw was found in MagickCore/quantum-export.c. An attacker who submits a
crafted file that is processed by ImageMagick could trigger undefined behavior
in the form of values outside the range of type
unsigned long long as well as a shift exponent that is too large for
64-bit type. This would most likely lead to an impact to application availability,
but could potentially cause other problems related to undefined behavior.
CVE-2021-20243
A flaw was found in MagickCore/resize.c. An attacker who submits a crafted
file that is processed by ImageMagick could trigger undefined behavior
in the form of math division by zero.
CVE-2021-20245
A flaw was found in coders/webp.c. An attacker who submits a crafted file that
is processed by ImageMagick could trigger undefined behavior in the form of
math division by zero.
CVE-2021-20309
A division by zero in WaveImage() of MagickCore/visual-effects.c may trigger
undefined behavior via a crafted image file submitted to an application using
ImageMagick.
CVE-2021-20312
An integer overflow in WriteTHUMBNAILImage of coders/thumbnail.c may trigger
undefined behavior via a crafted image file that is submitted by an attacker
and processed by an application using ImageMagick.
CVE-2021-20313
A potential cipher leak when the calculate signatures in TransformSignature is possible.
CVE-2018-25009
An out-of-bounds read was found in function WebPMuxCreateInternal.
The highest threat from this vulnerability is to data confidentiality
and to the service availability.
CVE-2018-25010
An out-of-bounds read was found in function ApplyFilter.
The highest threat from this vulnerability is to data confidentiality
and to the service availability.
CVE-2018-25011
A heap-based buffer overflow was found in PutLE16().
The highest threat from this vulnerability is to data confidentiality
and integrity as well as system availability.
CVE-2018-25012
An out-of-bounds read was found in function WebPMuxCreateInternal.
The highest threat from this vulnerability is to data confidentiality
and to the service availability.
CVE-2018-25013
An out-of-bounds read was found in function ShiftBytes.
The highest threat from this vulnerability is to data confidentiality
and to the service availability.
CVE-2018-25014
An unitialized variable is used in function ReadSymbol.
The highest threat from this vulnerability is to data confidentiality
and integrity as well as system availability.
CVE-2020-36328
A heap-based buffer overflow in function WebPDecodeRGBInto is possible
due to an invalid check for buffer size. The highest threat from this
vulnerability is to data confidentiality and integrity as well as system
availability.
CVE-2020-36329
A use-after-free was found due to a thread being killed too early.
The highest threat from this vulnerability is to data confidentiality
and integrity as well as system availability.
CVE-2020-36330
An out-of-bounds read was found in function ChunkVerifyAndAssign.
The highest threat from this vulnerability is to data confidentiality
and to the service availability.
CVE-2020-36331
An out-of-bounds read was found in function ChunkAssignData.
The highest threat from this vulnerability is to data confidentiality
and to the service availability.
CVE-2020-36332 was marked as ignored
for stretch due to too disruptive patch for older versions of libwebp.
CVE-2021-32917
The proxy65 component allows open access by default, even if neither
of the users has an XMPP account on the local server, allowing
unrestricted use of the server s bandwidth.
CVE-2021-32921
Authentication module does not use a constant-time algorithm for
comparing certain secret strings when running under Lua 5.2 or later.
This can potentially be used in a timing attack to reveal the
contents of secret strings to an attacker.
DLA-2687-2 prosody_0.9.12-2+deb9u4
Upload prosody_0.9.12-2+deb9u3 introduced a regression in the
mod_auth_internal_hashed module. Big thanks to Andre Bianchi for the
reporting an issue and for testing the update.
CVE-2021-32918,
CVE-2021-32920,
were marked as ignored for stretch: the affected code is not existing in that version of prosody.
LTS-Meeting
I attended the Debian LTS team Jitsi-meeting.
Debian Science Team
openpiv-python
I started to package python-openpiv.
The software implements PIV (Particle Image Velocimetry) method to compare two images and
obtain velocity field.
Other FLOSS activities
Admesh
Admesh is the first package which I adopted over 10 years ago! Upstream is not active for a very long time,
so I created a github-repo back in 2013.
The software helps to manipulate STL-files. STL is the file format for meshes, mostly developed for CAD programs.
This month I decided to clean the build system. It was switched to cmake. CI was updated, now it compiles
the sources under Linux/Windows environment, runs tests, AddressSanitizer and UndefinedBehaviourSanitizer
were employed. Work is ongoing.
This is part of a series of posts on ideas for an ansible-like provisioning
system, implemented in Transilience.
Unit testing some parts of Transilience, like the
apt and
systemd
actions, or remote Mitogen connections, can really use a containerized system
for testing.
To have that, I reused my work on nspawn-runner.
to build a simple and very fast system of ephemeral containers, with minimal dependencies,
based on systemd-nspawn
and btrfs snapshots:
Setup
To be able to use systemd-nspawn --ephemeral, the chroots needs to be btrfs
subvolumes. If you are not running on a btrfs filesystem, you can create one to
run the tests, even on a file:
I created a script to setup the test environment, here is an extract:
mkdir -p test_chroots
cat << EOF > "test_chroots/CACHEDIR.TAG"Signature: 8a477f597d28d172789f06886806bc55# chroots used for testing transilience, can be regenerated with make-test-chrootEOF
btrfs subvolume create test_chroots/buster
eatmydata debootstrap --variant=minbase --include=python3,dbus,systemd buster test_chroots/buster
CACHEDIR.TAG is a nice trick to tell backup software not to bother backing up
the contents of this directory, since it can be easily regenerated.
eatmydata is optional, and it speeds up debootstrap quite a bit.
Running unittest with sudo
Here's a simple helper to drop root as soon as possible, and regain it only
when needed. Note that it needs $SUDO_UID and $SUDO_GID, that are set by
sudo, to know which user to drop into:
classProcessPrivs:""" Drop root privileges and regain them only when needed """def__init__(self):self.orig_uid,self.orig_euid,self.orig_suid=os.getresuid()self.orig_gid,self.orig_egid,self.orig_sgid=os.getresgid()if"SUDO_UID"notinos.environ:raiseRuntimeError("Tests need to be run under sudo")self.user_uid=int(os.environ["SUDO_UID"])self.user_gid=int(os.environ["SUDO_GID"])self.dropped=Falsedefdrop(self):""" Drop root privileges """ifself.dropped:returnos.setresgid(self.user_gid,self.user_gid,0)os.setresuid(self.user_uid,self.user_uid,0)self.dropped=Truedefregain(self):""" Regain root privileges """ifnotself.dropped:returnos.setresuid(self.orig_suid,self.orig_suid,self.user_uid)os.setresgid(self.orig_sgid,self.orig_sgid,self.user_gid)self.dropped=False@contextlib.contextmanagerdefroot(self):""" Regain root privileges for the duration of this context manager """ifnotself.dropped:yieldelse:self.regain()try:yieldfinally:self.drop()@contextlib.contextmanagerdefuser(self):""" Drop root privileges for the duration of this context manager """ifself.dropped:yieldelse:self.drop()try:yieldfinally:self.regain()privs=ProcessPrivs()privs.drop()
As soon as this module is loaded, root privileges are dropped, and can be
regained for as little as possible using a handy context manager:
classChroot:""" Manage an ephemeral chroot """running_chroots:Dict[str,"Chroot"]=def__init__(self,name:str,chroot_dir:Optional[str]=None):self.name=nameifchroot_dirisNone:self.chroot_dir=self.get_chroot_dir(name)else:self.chroot_dir=chroot_dirself.machine_name=f"transilience- uuid.uuid4() "defstart(self):""" Start nspawn on this given chroot. The systemd-nspawn command is run contained into its own unit using systemd-run """unit_config=['KillMode=mixed','Type=notify','RestartForceExitStatus=133','SuccessExitStatus=133','Slice=machine.slice','Delegate=yes','TasksMax=16384','WatchdogSec=3min',]cmd=["systemd-run"]forcinunit_config:cmd.append(f"--property= c ")cmd.extend(("systemd-nspawn","--quiet","--ephemeral",f"--directory= self.chroot_dir ",f"--machine= self.machine_name ","--boot","--notify-ready=yes"))log.info("%s: starting machine using image %s",self.machine_name,self.chroot_dir)log.debug("%s: running %s",self.machine_name," ".join(shlex.quote(c)forcincmd))withprivs.root():subprocess.run(cmd,check=True,capture_output=True)log.debug("%s: started",self.machine_name)self.running_chroots[self.machine_name]=selfdefstop(self):""" Stop the running ephemeral containers """cmd=["machinectl","terminate",self.machine_name]log.debug("%s: running %s",self.machine_name," ".join(shlex.quote(c)forcincmd))withprivs.root():subprocess.run(cmd,check=True,capture_output=True)log.debug("%s: stopped",self.machine_name)delself.running_chroots[self.machine_name]@classmethoddefcreate(cls,chroot_name:str)->"Chroot":""" Start an ephemeral machine from the given master chroot """res=cls(chroot_name)res.start()returnres@classmethoddefget_chroot_dir(cls,chroot_name:str):""" Locate a master chroot under test_chroots/ """chroot_dir=os.path.abspath(os.path.join(os.path.dirname(__file__),"..","test_chroots",chroot_name))ifnotos.path.isdir(chroot_dir):raiseRuntimeError(f" chroot_dir does not exists or is not a chroot directory")returnchroot_dir# We need to use atextit, because unittest won't run# tearDown/tearDownClass/tearDownModule methods in case of KeyboardInterrupt# and we need to make sure to terminate the nspawn containers at exit@atexit.registerdefcleanup():# Use a list to prevent changing running_chroots during iterationforchrootinlist(Chroot.running_chroots.values()):chroot.stop()
And here's a TestCase
mixin that starts a containerized systems and opens a Mitogen
connection to it:
classChrootTestMixin:""" Mixin to run tests over a setns connection to an ephemeral systemd-nspawn container running one of the test chroots """chroot_name="buster"@classmethoddefsetUpClass(cls):super().setUpClass()importmitogenfromtransilience.systemimportMitogencls.broker=mitogen.master.Broker()cls.router=mitogen.master.Router(cls.broker)cls.chroot=Chroot.create(cls.chroot_name)withprivs.root():cls.system=Mitogen(cls.chroot.name,"setns",kind="machinectl",python_path="/usr/bin/python3",container=cls.chroot.machine_name,router=cls.router)@classmethoddeftearDownClass(cls):super().tearDownClass()cls.system.close()cls.broker.shutdown()cls.chroot.stop()
Running tests
Once the tests are set up, everything goes on as normal, except one needs to
run nose2 with sudo:
sudo nose2-3
Spin up time for containers is pretty fast, and the tests drop root as soon as
possible, and only regain it for as little as needed.
Also, dependencies for all this are minimal and available on most systems, and
the setup instructions seem pretty straightforward