TLDR: Freexian s success means that we have resources to invest into Debian projects. Plainly offering money has not worked so far, so I am looking to hire a project manager whose work would be to help spend that money in useful ways. At the same time, Freexian needs to adapt to cope with the growth: with new employees, with new infrastructure and a new offering. I want to give an idea of where we are headed, to try to inspire persons that share our values and our desire to improve Debian. Read on if you are interested. Note: The original text has been split into 4 blog posts that will be published over a few days. Introduction

Freexian is an IT service company specialized in Debian. We provide technical support by email on Debian, we create and maintain Debian packages requested by our customers, we also help organizations run an entire Debian derivative (Kali Linux being the most notable one).

On top of this, it runs the commercial part of the Debian LTS service : Freexian invoices many sponsors that need long term support, and uses the money to pay Debian contributors (about 12 currently) to make sure that Debian releases are supported for 5 years instead of 3. With the Extended LTS service, we push that further to 7 years, however only for a smaller subset of packages and in a repository that is hosted outside of debian.org. Freexian s purpose When I created Freexian, it was out of a desire to be paid to work on Debian, and to be able to contribute during work time to the project that was so important to me. That goal has been met a long time ago. But ultimately what I strive to achieve for Debian is not entirely aligned with the work that Freexian s customers are requesting. That s why, in the long term projects of Freexian, I always kept find a business model that can fund the Debian projects that I would like to do , as well as if that model works for me, build something so that other can benefit from it too . The first occasion to experiment something appeared when Debian discussed Long Term Support and when I stepped up to setup a commercial offer to pay Debian contributors. Step 1: Paying Debian contributors for LTS work When we started the Debian LTS service, I voluntarily opted to use an hourly rate that was rather high so that any Debian developer regardless of their geographical location, could participate and not earn (much) less than what they would have from working on other tasks. This choice did imply paying a very high rate for some countries, but I didn t see that as a problem, quite the contrary: if a Debian developer can earn enough money to cover their cost of living with 15h of Debian LTS, and then spend the rest of their month contributing on Debian, all the better! I m not sure if anyone made this choice, but that was a dream of my younger self From a personal standpoint, the launch of Debian LTS has meant less free time, more administrative work, new duties to coordinate a team of paid contributors, more communication with many Debian-using companies, and many new opportunities too! This ultimately resulted in the launch of Extended LTS and PHP LTS, both of which have been rather successful so far. Step 2: Funding Debian projects With the growth of the Debian LTS service, and given that we have reached the required funding level, we decided to put a small share of the revenues aside and use that to fund useful Debian projects, typically in areas that were affected by our Debian LTS work. This effort was fully formalized in the project-funding git repository. We announced this process in November 2020, and we have kept mentioning it in our monthly LTS reports ever since, but so far only a single project has benefited from this. This is really the dream offer that I wish had existed when I was younger and was still struggling to get enough customers: hence I don t really understand this lack of interest. You can find some discussions over the reasons why this offer has not (yet) found its target audience in this debian-vote thread. I was hoping that spending money would be easy, but I now realize I was wrong! I m positive that I could find dozens of useful projects to fund, but I just don t have the time for this extra effort on top of my regular Freexian duties. I still really want to put this money to good use, which is why I m looking into some solutions. This article is to be continued in another upcoming post, stay tuned!

How can you help free software? Aside from donating to the excellent free software nonprofits out there, and contributing to a project by building software or other resources, there are things you can do to help the free software cause. The two biggest things I think are providing mentorship and gently normalizing free software.

Providing Mentorship Allison Randal introduced me to the idea that mentorship doesn t have to be an ongoing process. This is to say, you don t have to sign up to be someone s best friend and advisor for life (though you certainly can). Providing short term, project or skill based, or one off mentorship is useful for building community because it makes people feel welcome and cared for and helps build skills that benefit free software. I do a lot of proofreading and editing of people s writing especially people with minimal writing experience and/or non-native English speakers who are writing important documents in English. If the person is interested, I try to talk to them about their writing and why I m making these particular suggestions. I hope this helps them with their writing in the future. Other examples are working on a particular project or skill, this can be helping them develop a particular skill (e.g. git outside of the command line), or giving advice on a project with a level of specificity and detail you re both comfortable with. These can, again, be one off things or things that require minimal effort/occasional conversation. I have some friends who I consider my Debian mentors who just answer functional questions whenever I have trouble doing something. I also love love love talking with people about their free software trajectories, their goals and desires and dreams for their involvement in free software, whether that s finding a place in a community, developing a skill set, or other things about their future (like job hopes, schooling, etc). These conversations have been so helpful for me personally, and I like to think they help others.

Gently Normalizing Free Software I think normalizing free software is very important to its success and adoption. It s not helpful to insist someone who has never done so before to create a Debian boot disk and install it. It is helpful to suggest using Big Blue Button or jitsi. If a friend wants help finding audio editing software, suggest they try audacity. I d go as far as to suggest doing this without explaining that it s free software, and instead focus on why it ll work and that it s available at no cost. If they like it, then it s a great time to talk about rights and freedoms. Of course if they already care about these sorts of things, if you re discussing privacy software, if anti-surveillance is an issue, or any number of other things, software freedom is a great thing to bring up!

Above all, just be nice. Be nice. It s basically the best thing you can do for free software.

I am slowly working towards collecting twenty years of static web site generation and release management tools into DocKnot. Pod::Thread is the component that turns POD documentation into HTML by converting it to thread, the macro language that is the backbone of my static site generator, and then letting spin turn the results into HTML. I wrote this module years ago and have had it around as a private Perl module, but since the version of DocKnot that incorporates spin will have it as a dependency, it seemed time to publicly release it. Compared to the last released version in 2013, it also has a few bug fixes and improvements. The module now internally handles the navbar and table of contents generation (by deferring output) rather than using a pre-scanning pass in the driver script, and fixes the title casing of DB_CONFIG in the output from one INN manual page. I also did a lot of modernization and improvements to the test suite. You can get the latest version from CPAN or the Pod::Thread distribution page.

Too much has been written, a war of support letters is going on, Debian is heading head first like Lemmings into the same war. And then, there is this by the first female President of the American Civil Liberties Union (ACLU): Civil Rights Activist Nadine Strossen s Response To #CancelStallman:

I find it so odd that the strong zeal for revenge and punishment if someone says anything that is perceived to be sexist or racist or discriminatory comes from liberals and progressives. There are so many violations [in cases like Stallman s] of such fundamental principles to which progressives and liberals cling in general as to what is justice, what is fairness, what is due process.
One is proportionality: that the punishment should be proportional to the offense. Another one is restorative justice: that rather than retribution and punishment, we should seek to have the person constructively come to understand, repent, and make amends for an infraction. Liberals generally believe society to be too punitive, too harsh, not forgiving enough. They are certainly against the death penalty and other harsh punishments even for the most violent, the mass murderers. Progressives are right now advocating for the release of criminals, even murderers. To then have exactly the opposite attitude towards something that certainly is not committing physical violence against somebody, I don t understand the double standard!
Another cardinal principle is we shouldn t have any guilt by association. [To hold culpable] these board members who were affiliated with him and ostensibly didn t do enough to punish him for things that he said which by the way were completely separate from the Free Software Foundation is multiplying the problems of unwarranted punishment. It extends the punishment where the argument for responsibility and culpability becomes thinner and thinner to the vanishing point. That is also going to have an enormous adverse impact on the freedom of association, which is an important right protected in the U.S. by the First Amendment.
The Supreme Court has upheld freedom of association in cases involving organizations that were at the time highly controversial. It started with NAACP (National Association for the Advancement of Colored People) during the civil rights movement in the 1950s and 60s, but we have a case that s going to the Supreme Court right now regarding Black Lives Matter. The Supreme Court says even if one member of the group does commit a crime in both of those cases physical violence and assault that is not a justification for punishing other members of the group unless they specifically intended to participate in the particular punishable conduct.
Now, let s assume for the sake of argument, Stallman had an attitude that was objectively described as discriminatory on the basis on race and gender (and by the way I have seen nothing to indicate that), that he s an unrepentant misogynist, who really believes women are inferior. We are not going to correct those ideas, to enlighten him towards rejecting them and deciding to treat women as equals through a punitive approach! The only approach that could possibly work is an educational one! Engaging in speech, dialogue, discussion and leading him to re-examine his own ideas.
Even if I strongly disagree with a position or an idea, an expression of an idea, advocacy of an idea, and even if the vast majority of the public disagrees with the idea and finds it offensive, that is not a justification for suppressing the idea. And it s not a justification for taking away the equal rights of the person who espouses that idea including the right to continue holding a tenured position or other prominent position for which that person is qualified.
But a number of the ideas for which Richard Stallman has been attacked and punished are ideas that I as a feminist advocate of human rights find completely correct and positive from the perspective of women s equality and dignity! So for example, when he talks about the misuse and over use and flawed use of the term sexual assault, I completely agree with that critique! People are indiscriminantly using that term or synonyms to describe everything from the most appaulling violent abuse of helpless vulnerable victims (such as a rape of a minor) to any conduct or expression in the realm of gender or sexuality that they find unpleasant or disagreeable.
So we see the term sexual assault and sexual harrassment used for example, when a guy asks a woman out on a date and she doesn t find that an appealing invitation. Maybe he used poor judgement in asking her out, maybe he didn t, but in any case that is NOT sexual assault or harassment. To call it that is to really demean the huge horror and violence and predation that does exist when you are talking about violent sexual assault. People use the term sexual assault/ sexual harassment to refer to any comment about gender or sexuality issues that they disagree with or a joke that might not be in the best taste, again is that to be commended? No! But to condemn it and equate it with a violent sexual assault again is really denying and demeaning the actual suffering that people who are victims of sexual assault endure. It trivializes the serious infractions that are committed by people like Jeffrey Epstein and Harvey Weinstein. So that is one point that he made that I think is very important that I strongly agree with.
Secondly and relatedly, [Richard Stallman] never said that he endorse child pornography, which by definition the United States Supreme Court has defined it multiple times is the sexual exploitation of an actual minor. Coerced, forced, sexual activity by that minor, with that minor that happens to be filmed or photographed. That is the definition of child pornography. He never defends that! What the point he makes, a very important one, which the U.S. Supreme Court has also made, is mainly that we overuse and distort the term child pornography to refer to any depiction of any minor in any context that is even vaguely sexual.
So some people have not only denounced as child pornography but prosecuted and jailed loving devoted parents who committed the crime of taking a nude or semi-nude picture of their own child in a bathtub or their own child in a bathing suit. Again it is the hysteria that has totally refused to draw an absolutely critical distinction between actual violence and abuse, which is criminal and should be criminal, to any potentially sexual depiction of a minor. And I say potentially because I think if you look at a picture a parent has taken of a child in a bathtub and you see that as sexual, then I d say there s something in your perspective that might be questioned or challenged! But don t foist that upon the parent who is lovingly documenting their beloved child s life and activities without seeing anything sexual in that image.
This is a decision that involves line drawing. We tend to have this hysteria where once we hear terms like pedophilia of course you are going to condemn anything that could possibly have that label. Of course you would. But societies around the world throughout history various cultures and various religions and moral positions have disagreed about at what age do you respect the autonomy and individuality and freedom of choice of a young person around sexuality. And the U.S. Supreme Court held that in a case involving minors right to choose to have an abortion.
By the way, [contraception and abortion] is a realm of sexuality where liberals and progressives and feminists have been saying, Yes! If you re old enough to have sex. You should have the right to contraception and access to it. You should have the right to have an abortion. You shouldn t have to consult with your parents and have their permission or a judge s permission because you re sufficiently mature. And the Supreme Court sided in accord of that position. The U.S. Supreme Court said constitutional rights do not magically mature and spring into being only when someone happens to attain the state defined age of majority.
In other words the constitution doesn t prevent anyone from exercising rights, including Rights and sexual freedoms, freedom of choice and autonomy at a certain age! And so you can t have it both ways. You can t say well we re strongly in favor of minors having the right to decide what to do with their own bodies, to have an abortion what is in some people s minds murder but we re not going to trust them to decide to have sex with somewhat older than they are.
And I say somewhat older than they are because that s something where the law has also been subject to change. On all issues of when you obtain the age of majority, states differ on that widely and they choose different ages for different activities. When you re old enough to drive, to have sex with someone around your age, to have sex with someone much older than you. There is no magic objective answer to these questions. I think people need to take seriously the importance of sexual freedom and autonomy and that certainly includes women, feminists. They have to take seriously the question of respecting a young person s autonomy in that area.
There have been famous cases of 18 year olds who have gone to prison because they had consensual sex with their girlfriends who were a couple of years younger. A lot of people would not consider that pedophilia and yet under some strict laws and some absolute definitions it is. Romeo and Juliet laws make an exception to pedophilia laws when there is only a relatively small age difference. But what is relatively small? So to me, especially when he says he is re-examining his position, Stallman is just thinking through the very serious debate of how to be protective and respectful of young people. He is not being disrespectful, much less wishing harm upon young people, which seems to be what his detractors think he s doing.

Unfortunately, I don t think the Anti-Harassment Team of Debian and others of the usual group of warriors will ever read less understand what is written there. So sad.

The Debian CI platform is comprised of 30+ (virtual) machines. Maintaining this many machines, and being able to add new ones with some degree of reliability requires one to use some sort of configuration management. Until about a week ago, we were using Chef for our configuration management. I was, for several years, the main maintainer of Chef in Debian, so using it was natural to me, as I had used it before for personal and work projects. But last year I decided to request the removal of Chef from Debian, so that it won't be shipped with Debian 11 (bullseye). After evaluating a few options, I believed that the path of least resistance was to migrate to itamae. itamae was inspired by chef, and uses a DSL that is very similar to the Chef one. Even though the itamae team claim it's not compatible with Chef, the changes that I needed to do were relatively limited. The necessary code changes might look like a lot, but a large part of them could be automated or done in bulk, like doing simple search and replace operations, and moving entire directories around. In the rest of this post, I will describe the migration process, starting with the infrastructure changes, the types of changes I needed to make to the configuration management code, and my conclusions about the process. Infrastructure changes The first step was to add support for itamae to chake, a configuration management wrapper tool that I wrote. chake was originally written as a serverless remote executor for Chef, so this involved a bit of a redesign. I thought it was worth it to do, because at that point chake had gained several interesting managements features that we no directly tied to Chef. This work was done a bit slowly over the course of the several months, starting almost exactly one year ago, and was completed 3 months ago. I wasn't in a hurry and knew I had time before Debian 11 is released and I had to upgrade the platform. After this was done, I started the work of migrating the then Chef cookbooks to itamae, and the next sections present the main types of changes that were necessary. During the entire process, I sent a few patches out:

itamae #317 - Make output unbuffered
itamae #325 - file: add support for sensitive files
itamae #327 - file: improve diff output
itamae #337 - package: add :upgrade action. This is work in progress at the moment, and wasn't merged yet.
serverspec #727 - debian: add support for Debian testing and unstable

Code changes These are the main types of changes that were necessary in the configuration code to accomplish the migration to itamae. Replace cookbook_file with remote_file. The resource known as cookbook_file in Chef is called remote_file in itamae. Fixing this is just a matter of search and replace, e.g.:

-cookbook_file '/etc/apt/apt.conf.d/00updates' do
+remote_file '/etc/apt/apt.conf.d/00updates' do

Changed file locations The file structure assumed by itamae is a lot simpler than the one in Chef. The needed changes were:

static files and templates moved from cookbooks/$ cookbook / files,templates /default to cookbooks/$ cookbook / files,templates
recipes moved from cookbooks/$ cookbook /recipes/*.rb to cookbooks/$ cookbook /*.rb
host-specific files and templates are not supported directly, but can be implemented just by using an explicit source statement, like this:
```
remote_file "/etc/foo.conf" do
  source "files/host-# node['fqdn'] /foo.conf"
end
```

Explicit file ownership and mode Chef is usually design to run as root on the nodes, and files created are owned by root and have move 0644 by default. With itamae, files are by default owned by the user that was used to SSH into the machine. Because of this, I had to review all file creation resources and add owner, group and mode explicitly:

-cookbook_file '/etc/apt/apt.conf.d/00updates' do
-  source 'apt.conf'
+remote_file '/etc/apt/apt.conf.d/00updates' do
+  source 'files/apt.conf'
+  owner   'root'
+  group   'root'
+  mode    "0644"
 end

In the end, I guess being explicit make the configuration code more understandable, so I take that as a win. Different execution context One of the major differences between Chef itamae comes down the execution context of the recipes. In both Chef and itamae, the configuration is written in DSL embedded in Ruby. This means that the recipes are just Ruby code, and difference here has to do with where that code is executed. With Chef, the recipes are always execute on the machine you are configuring, while with itamae the recipe is executed on the workstation where you run itamae, and that gets translated to commands that need to be executed on the machine being configured. For example, if you need to configure a service based on how much RAM the machine has, with Chef you could do something like this:

total_ram = File.readlines("/proc/meminfo").find do  l 
  l.split.first == "MemTotal:"
end.split[1]
file "/etc/service.conf" do
  # use 20% of the total RAM
  content "cache_size = # ram / 5 KB"
end

With itamae, all that Ruby code will run on the client, so total_ram will contain the wrong number. In the Debian CI case, I worked around that by explicitly declaring the amount of RAM in the static host configuration, and the above construct ended up as something like this:

file "/etc/service.conf" do
  # use 20% of the total RAM
  content "cache_size = # node['total_ram'] / 5 KB"
end

Lessons learned This migration is now complete, and there are a few points that I take away from it:

The migration is definitely viable, and I'm glad I picked itamae after all.
Of course, itamae is way simpler than Chef, and has less features. On the other hand, this means that it a simple package to maintain, with less dependencies and keeping it up to date is a lot easier.
itamae is considerably slower than Chef. On my local tests, a noop execution (e.g. re-applying the configuration a second time) against local VMs with itamae takes 3x the time it takes with Chef.

All in all, the system is working just fine, and I consider this to have been a successful migration. I'm happy it worked out.

A long time, no blog entries. What's prompted a new missive? Two things:

my own hankering to get back into free software after too many years faced with daily struggles against proprietary software
the upcoming GR in Debian.

All the time I've worked with software and computers, the lack of diversity in the contributors has irked me. I started my career in pharmaceutical sciences where the mix of people, at university, would appear to be genuinely diverse. It was in the workplace that the problems started, especially in the retail portion of community pharmacy with a particular gender imbalance between management and counter staff. Then I started to pick up programming. I gravitated to free software for the simple reason that I could tinker with the source code. To do a career change and learn how to program with a background in a completely alien branch of science, the only realistic route into the field is to have access to the raw material - source code. Without that, I would have been seeking a sponsor, in much the same way as other branches of science need grants or sponsors to purchase the equipment and facilities to get a foot in the door. All it took from me was some time, a willingness to learn and a means to work alongside those already involved. That's it. No complex titration equipment, flasks, centrifuges, pipettes, spectrographs, or petri dishes. It's hard to do pharmaceutical science from home. Software is so much more accessible - but only if the software itself is free. Software freedom removes the main barrier to entry. Software freedom enables the removal of other barriers too. Once the software is free, it becomes obvious that the compiler (indeed the entire toolchain) needs to be free, to turn the software into something other people can use. The same applies to interpreters, editors, kernels and the entire stack. I was in a different branch of science whilst all that was being created and I am very glad that Debian Woody was available as a free cover disc on a software magazine just at the time I was looking to pick up software programming. That should be that. The only next step to be good enough to write free software was the "means to work alongside those already involved". That, it turns out, is much more than just a machine running Debian. It's more than just having an ISP account and working email (not commonplace in 2003). It's working alongside the people - all the people. It was my first real exposure to the toxicity of some parts of many scientific and technical arenas. Where was the diversity? OK, maybe it was just early days and the other barriers (like an account with an ISP) were prohibitive for many parts of the world outside Europe & USA in 2003, so there were few people from other countries but the software world was massively dominated by the white Caucasian male. I'd been insulated by my degree course, and to a large extent by my university which also had courses which already had much more diverse intakes - optics and pharmacy, business and human resources. Relocating from the insular world of a little town in Wales to Birmingham was also key. Maybe things would improve as the technical barriers to internet connectivity were lowered. Sadly, no. The echo chamber of white Caucasian input has become more and more diluted as other countries have built the infrastructure to get the populace online. Debian has helped in this area, principally via DebConf. Yet only the ethnicity seemed to change, not the diversity. Recently, more is being done to at least make Debian more welcoming to those who are brave enough to increase the mix. Progress is much slower than the gains seen in the ethnicity mix, probably because that was a benefit of a technological change, not a societal change. The attitudes so prevalent in the late 20th century are becoming less prevalent amongst, and increasingly abhorrent to, the next generation of potential community members. Diversity must come or the pool of contributors will shrink to nil. Community members who cling to these attitudes are already dinosaurs and increasingly unwelcome. This is a necessary step to retain access to new contributors as existing contributors age. To be able to increase the number of contributors, the community cannot afford to be dragged backwards by anyone, no matter how important or (previously) respected. Debian, or even free software overall, cannot change all the problems with diversity in STEM but we must not perpetuate the problems either. Those people involved in free software need to be open to change, to input from all portions of society and welcoming. Puerile jokes and disrespectful attitudes must be a thing of the past. The technical contributions do not excuse behaviours that act to prevent new people joining the community. Debian is getting older, the community and the people. The presence of spouses at Debian social events does not fix the problem of the lack of diversity at Debian technical events. As contributors age, Debian must welcome new, younger, people to continue the work. All the technical contributions from the people during the 20th century will not sustain Debian in the 21st century. Bit rot affects us all. If the people who provided those contributions are not encouraging a more diverse mix to sustain free software into the future then all their contributions will be for nought and free software will die. So it comes to the FSF and RMS. I did hear Richard speak at an event in Bristol many years ago. I haven't personally witnessed the behavioural patterns that have been described by others but my memories of that event only add to the reality of those accounts. No attempts to be inclusive, jokes that focused on division and perpetuating the white male echo chamber. I'm not perfect, I struggle with some of this at times. Personally, I find the FSFE and Red Hat statements much more in line with my feelings on the matter than the open letter which is the basis of the GR. I deplore the preliminary FSF statement on governance as closed-minded, opaque and archaic. It only adds to my concerns that the FSF is not fit for the 21st century. The open letter has the advantage that it is a common text which has the backing of many in the community, individuals and groups, who are fit for purpose and whom I have respected for a long time. Free software must be open to contributions from all who are technically capable of doing the technical work required. Free software must equally require respect for all contributors and technical excellence is not an excuse for disrespect. Diversity is the life blood of all social groups and without social cohesion, technical contributions alone do not support a community. People can change and apologies are welcome when accompanied by modified behaviour. The issues causing a lack of diversity in Debian are complex and mostly reflective of a wider problem with STEM. Debian can only survive by working to change those parts of the wider problem which come within our influence. Influence is key here, this is a soft issue, replete with unreliable emotions, unhelpful polemic and complexity. Reputations are hard won and easily blemished. The problems are all about how Debian looks to those who are thinking about where to focus in the years to come. It looks like the FSFE should be the ones to take the baton from the FSF - unless the FSF can adopt a truly inclusive and open governance. My problem is not with RMS himself, what he has or has not done, which apologies are deemed sincere and whether behaviour has changed. He is one man and his contributions can be respected even as his behaviour is criticised. My problem is with the FSF for behaving in a closed, opaque and divisive manner and then making a governance statement that makes things worse, whilst purporting to be transparent. Institutions lay the foundations for the future of the community and must be expected to hold individuals to account. Free and open have been contentious concepts for the FSF, with all the arguments about Open Source which is not Free Software. It is clear that the FSF do understand the implications of freedom and openness. It is absurd to then adopt a closed and archaic governance. A valid governance model for the FSF would never have allowed RMS back onto the board, instead the FSF should be the primary institution to hold him, and others like him, to account for their actions. The FSF needs to be front and centre in promoting diversity and openness. The FSF could learn from the FSFE. The calls for the FSF to adopt a more diverse, inclusive, board membership are not new. I can only echo the Red Hat statement:

in order to regain the confidence of the broader free software community, the FSF should make fundamental and lasting changes to its governance.

...

[There is] no reason to believe that the most recent FSF board statement signals any meaningful commitment to positive change.

And the FSFE statement:

The goal of the software freedom movement is to empower all people to control technology and thereby create a better society for everyone. Free Software is meant to serve everyone regardless of their age, ability or disability, gender identity, sex, ethnicity, nationality, religion or sexual orientation. This requires an inclusive and diverse environment that welcomes all contributors equally.

The FSF has not demonstrated the behaviour I expect from a free software institution and I cannot respect an institution which proclaims a mantra of freedom and openness that is not reflected in the governance of that institution. The preliminary statement by the FSF board is abhorrent. The FSF must now take up the offers from those institutions within the community who retain the respect of that community. I'm only one voice, but I would implore the FSF to make substantive, positive and permanent change to the governance, practices and future of the FSF or face irrelevance.

In 14 years since starting this blog, I changed the software several times, with each iteration simplifying requirements and setup. Wordpress In 2007, I started blogging. Back then I used a self-hosted Wordpress instance, running on my server. I was never really comfortable with the technology stack involved: a programming language I didn't speak, a DB I had little experience with, a relatively heavy setup and of course, the occasional security issues related to that setup. All that just to have a somewhat dynamic website with pingbacks and comments generated by your visitors... totally worth it and exciting times! You wrote something to the "lazyweb" department and people would actually comment with useful advice! Then came the spammers. First, it was just a little, and you could easily fight it off with the askimet plugin. Over the years, however the spam-to-useful-comments-ratio shifted to ridiculous levels and the comment section itself became a burden. Hosting the comments on a separate platform like disqus was not an option for me, so I finally turned the comments off. Without the need for a comment section, the whole idea of storing mostly textual content in a DB just to have it served as a website, suddenly seemed like overkill and an unnecessary security risk. So I started looking for alternatives. I wanted to maintain my content as plain text files. Text files are cool because they are future proof, portable and can be managed in git. Consequently, I was searching for a static site generator. Pelican Around 2016 I finally migrated away from Wordpress to Pelican. Pelican met all my requirements. I could store my content as plain Markdown files in a git repository. Pelican is written in Python, a language with which I'm very comfortable with. Pelican deals with blog posts and "pages" (i.e. non blog articles) and most importantly: Pelican just generates static HTML. So all that's required on the server side is a simple web server. Conveniently, it also comes with an importer that allowed me to export the content of my old Wordpress instance straight into Markdown files. Migrating to Pelican was a huge relieve: I finally had a all my content as plain text in a git repository, my server was a lot more secure without the Wordpress instance running and removing MySQL freed up a lot of resources on that machine. blag Fast forward to 2021. While I was quite happy with Pelican, it still had too many features I don't need. My blogging experience can be stripped down to a bunch of Markdown files that serve either as articles or pages, some static files, some templating for a decent design and an Atom feed. Particularly, I do not need pages/articles in multiple languages, a plugin system, support for multiple authors and tons of settings. So I did what every self-respecting nerd would do: I wrote my own site generator. Not to write something better than Pelican, but rather something lesser. Something that does less while still being functionally adequate. I called the result "blag", named after the blag of the webcomic xkcd. blag is a blog-aware, static site generator written in Python. It supports the basic features you'd expect from a blog, like an archive, tags and an Atom feed. It converts Markdown (with fenced code blocks) and supports Jinja2 templating. With this setup I'm quite happy for now. Maybe sometime there will be a simpler RSS/Atom-successor that does not require site-specific metadata (i.e. only stuff that can be generated from the articles themselves), so I could get rid of blag's only required configuration.

Context:

Richard Stallman is the founder of the Free Software movement, and commited his life to making what seemed like a ludicrous idea into a tangible reality. We owe him big time for that, and nothing somebody says or does will ever eclipse the fact.
But Richard Stallman has a very toxic personality. There is a long, well-known published list of abuse cases; if you must read more into it, some regarding his views on sex, consent, gender, and some other issues are published as a part of the open letter I am about to reference. I have witnessed quite a few; I won t disclose here the details, as many other incidents are already known. And I don t mean by this sexual abuse, although that s the twig that eventually broke the camel s back, but ranging from general rudeness to absolute lack of consideration for people around him.
In September 2019, Stallman was Forced to resign, first from his position at MIT, then as the president of the FSF. The direct cause was a comment where he defended the accusations on Minsky (a personal friend of his, and deceased three years prior to the fact) of sexual abuse.
Last week, 18 months after he was driven out of the FSF, and at LibrePlanet (FSF s signature conference, usually held at the MIT, this time naturally online only) Stallman announced his comeback to the Board of Directors of the FSF.

Many people (me included, naturally) in the Free Software world are very angry about this announcement. There is a call for signatures for a position statement presented by several free software leaders that has gathered, as I write this message, over 400 signatures. The Open Source Initiative has presented its institutional position statement. And I can only forecast this rejection will continue to grow. Free software was once the arena of young, raging alpha machos where a thick skin was an entry requirement. A good thing about growing up is that our community is now wiser, and although it still attracts younger people, there is a clear trend not to repeat our past ways. Free software has grown, and there is no place for a leader so disrespectful and hurting as many of us have witnessed Stallman to be. Again, the free software movement and the world as a whole owes a great deal to Stallman. He changed history. I admire his work, his persistence and his stubbornness. But I won t have him represent me.

Back in 2019, The Cure released a double-video-album featuring two live shows from 2018, their 40th anniversary concert at Hyde Park (which I attended) and their experimental "Curaetion" show from the Meltdown festival a few weeks earier. Rumours that there would be a release had been circulating since the gig itself, where video director Tim Pope was spotted filming the gig (sporting a tee shirt reading "Yes I'm Filming. No I don't know when you'll see it."). Earlier in 2019 the "Anniversary" half had a limited run in cinemas, but the home video release was not officially revealed until much later, and the inclusion of Curaetion an unexpected bonus. 'deluxe' releases The release is available in several configurations. The "standard" issue is a two-DVD (or Blu Ray) set in a hard-book case, with some liner notes in the middle. A "deluxe" issue is a roughly twice-as-large boxed set containing the two video discs and two CD-audio discs of the shows. When it comes to deluxe editions, I would historically have been the perfect "mark", but I've had some bad experiences with other releases (this is a story for another blog post), so for this release I pre-ordered the standard BR edition. Right up until release, I wondered whether I should have made the other choice. Eventually, pictures of other people's Deluxe Editions started to circulate, and I was re-assured that I made the right call. The larger box is just something harder to store, the larger booklet is an exact reproduction of the smaller, standard edition liner notes, which are not really notable. The shows I think the shows are great. The Hyde Park show is a fairly standard "best of" setlist, and my enjoyment of it might be coloured by having been at the show and having fond memories of it. But it's a great performance and very well captured, so I think there's something for people who like the Cure but weren't there to enjoy too. I wasn't at the Meltdown show. The setlist for that one is much more esoteric, and it's lovely to have renditions of songs that are rarely performed live (And: across the two sets you get If Only Tonight We Could Sleep twice). Getting the audio I knew in advance that I would likely listen to the shows more often than watch them, so the CDs in the Deluxe set would have been useful for that, but I was fairly sure the audio mix would be the same as on the video discs, so I could just extract that. I was half-right: the BR set (at least) has 5.1 surround and separate lossless stereo mixes, but they have been mastered quite differently to the CD audio, which appears to be brick-wall mastered, to such an extent that the audio distorts quite noticeably in a few places. The BR audio does not seem to suffer from the same problems. Getting the audio off the discs was interesting. On Linux, the tool of choice for decoding encrypted Blu Rays is libaacs, which requires a list of Blu Ray keys which you must source separately. With my BR drive I was not able to decode the discs using the keys I could find online. However, the separate tool MakeMKV was able to decode them. I don't know exactly how but I speculate that MakeMKV contains some leaked hardware player keys that are not in more general circulation. Once I had decoded copies of the two videos, the trusty tool ffmpeg is able to easily extract the PCM stereo audio track. Unfortunately it cannot, in one go, split the audio track at the chapter boundary points. This is easy to script around: one ffmpeg invocation to get the chapter offsets, then some scripting to turn that into seek and duration command-line arguments for further ffmpeg invocations. For space reasons, I opted to transcode to FLAC at the same time as extracting each track. I hit an interesting (although not fatal) problem: the time taken to extract each track got progressively slower, almost as if ffmpeg was transcoding the entire audio track up to the point specified as the offset to start extracting from. If I wanted to answer categorically whether the BR audio was preferable, I would need to perform a double-blind listening test, using a tool such as squishyball. I certainly can't be bothered: the BR audio is fine for me.

TL:DR; lost half my mail (150,000 messages, ~6GB) last night. Cause uncertain, but possibly a combination of a dead CMOS battery, systemd OnCalendar=daily, a (locking?) bug in syncmaildir, and generally, a system too exotic and complicated.

The crash So I somehow lost half my mail:

anarcat@angela:~(main)$ du -sh Maildir/
7,9G    Maildir/
anarcat@curie:~(main)$ du -sh Maildir
14G     Maildir
anarcat@marcos:~$ du -sh Maildir
8,0G    Maildir

Those are three different machines:

angela: my laptop, not always on
curie: my workstation, mostly always on
marcos: my mail server, always on

Those mails are synchronized using a rather exotic system based on SSH, syncmaildir and rsendmail. The anomaly started on curie:

-- Reboot --
mar 22 16:13:00 curie systemd[3199]: Starting pull emails with syncmaildir...
mar 22 16:13:00 curie smd-pull[4801]: rm: impossible de supprimer '/home/anarcat/.smd/workarea/Maildir': Le dossier n'est pas vide
mar 22 16:13:00 curie systemd[3199]: smd-pull.service: Main process exited, code=exited, status=1/FAILURE
mar 22 16:13:00 curie systemd[3199]: smd-pull.service: Failed with result 'exit-code'.
mar 22 16:13:00 curie systemd[3199]: Failed to start pull emails with syncmaildir.
mar 22 16:14:00 curie systemd[3199]: Starting pull emails with syncmaildir...
mar 22 16:14:00 curie smd-pull[7025]:  4091 ?        00:00:00 smd-push
mar 22 16:14:00 curie smd-pull[7025]: Already running.
mar 22 16:14:00 curie smd-pull[7025]: If this is not the case, remove /home/anarcat/.smd/lock by hand.
mar 22 16:14:00 curie smd-pull[7025]: any: smd-pushpull@localhost: TAGS: error::context(locking) probable-cause(another-instance-is-running) human-intervention(necessary) suggested-actions(run(kill 4091) run(rm /home/anarcat/.smd/lock))
mar 22 16:14:00 curie systemd[3199]: smd-pull.service: Main process exited, code=exited, status=1/FAILURE
mar 22 16:14:00 curie systemd[3199]: smd-pull.service: Failed with result 'exit-code'.
mar 22 16:14:00 curie systemd[3199]: Failed to start pull emails with syncmaildir.

Then it seems like smd-push (from curie) started destroying the universe for some reason:

mar 22 16:20:00 curie systemd[3199]: Starting pull emails with syncmaildir...
mar 22 16:20:00 curie smd-pull[9319]:  4091 ?        00:00:00 smd-push
mar 22 16:20:00 curie smd-pull[9319]: Already running.
mar 22 16:20:00 curie smd-pull[9319]: If this is not the case, remove /home/anarcat/.smd/lock by hand.
mar 22 16:20:00 curie smd-pull[9319]: any: smd-pushpull@localhost: TAGS: error::context(locking) probable-cause(another-instance-is-running) human-intervention(necessary) suggested-actions(ru
mar 22 16:20:00 curie systemd[3199]: smd-pull.service: Main process exited, code=exited, status=1/FAILURE
mar 22 16:20:00 curie systemd[3199]: smd-pull.service: Failed with result 'exit-code'.
mar 22 16:20:00 curie systemd[3199]: Failed to start pull emails with syncmaildir.
mar 22 16:21:34 curie smd-push[4091]: default: smd-client@smd-server-anarcat: TAGS: stats::new-mails(0), del-mails(293920), bytes-received(0), xdelta-received(26995)
mar 22 16:21:35 curie smd-push[9374]: register: smd-client@smd-server-register: TAGS: stats::new-mails(0), del-mails(0), bytes-received(0), xdelta-received(215)
mar 22 16:21:35 curie systemd[3199]: smd-push.service: Succeeded.

Notice the del-mails(293920) there: it is actively trying to destroy basically every email in my mail spool. Then somehow push and pull started both at once:

mar 22 16:21:35 curie systemd[3199]: Started push emails with syncmaildir.
mar 22 16:21:35 curie systemd[3199]: Starting push emails with syncmaildir...
mar 22 16:22:00 curie systemd[3199]: Starting pull emails with syncmaildir...
mar 22 16:22:00 curie smd-pull[10333]:  9455 ?        00:00:00 smd-push
mar 22 16:22:00 curie smd-pull[10333]: Already running.
mar 22 16:22:00 curie smd-pull[10333]: If this is not the case, remove /home/anarcat/.smd/lock by hand.
mar 22 16:22:00 curie smd-pull[10333]: any: smd-pushpull@localhost: TAGS: error::context(locking) probable-cause(another-instance-is-running) human-intervention(necessary) suggested-actions(r
mar 22 16:22:00 curie systemd[3199]: smd-pull.service: Main process exited, code=exited, status=1/FAILURE
mar 22 16:22:00 curie systemd[3199]: smd-pull.service: Failed with result 'exit-code'.
mar 22 16:22:00 curie systemd[3199]: Failed to start pull emails with syncmaildir.
mar 22 16:22:00 curie smd-push[9455]: smd-client: ERROR: Data transmission failed.
mar 22 16:22:00 curie smd-push[9455]: smd-client: ERROR: This problem is transient, please retry.
mar 22 16:22:00 curie smd-push[9455]: smd-client: ERROR: server sent ABORT or connection died
mar 22 16:22:00 curie smd-push[9455]: smd-server: ERROR: Unable to open Maildir/.kobo/cur/1498563708.M122624P22121.marcos,S=32234,W=32792:2,S: Maildir/.kobo/cur/1498563708.M122624P22121.marco
mar 22 16:22:00 curie smd-push[9455]: smd-server: ERROR: The problem should be transient, please retry.
mar 22 16:22:00 curie smd-push[9455]: smd-server: ERROR: Unable to open requested file.
mar 22 16:22:00 curie smd-push[9455]: default: smd-client@smd-server-anarcat: TAGS: stats::new-mails(0), del-mails(293920), bytes-received(0), xdelta-received(26995)
mar 22 16:22:00 curie smd-push[9455]: default: smd-client@smd-server-anarcat: TAGS: error::context(receive) probable-cause(network) human-intervention(avoidable) suggested-actions(retry)
mar 22 16:22:00 curie smd-push[9455]: default: smd-server@localhost: TAGS: error::context(transmit) probable-cause(simultaneous-mailbox-edit) human-intervention(avoidable) suggested-actions(r
mar 22 16:22:00 curie systemd[3199]: smd-push.service: Main process exited, code=exited, status=1/FAILURE
mar 22 16:22:00 curie systemd[3199]: smd-push.service: Failed with result 'exit-code'.
mar 22 16:22:00 curie systemd[3199]: Failed to start push emails with syncmaildir.

There it seems push tried to destroy the universe again: del-mails(293920). Interestingly, the push started again in parallel with the pull, right that minute:

mar 22 16:22:00 curie systemd[3199]: Starting push emails with syncmaildir...

... but didn't complete for a while, here's pull trying to start again:

mar 22 16:24:00 curie systemd[3199]: Starting pull emails with syncmaildir...
mar 22 16:24:00 curie smd-pull[12051]: 10466 ?        00:00:00 smd-push
mar 22 16:24:00 curie smd-pull[12051]: Already running.
mar 22 16:24:00 curie smd-pull[12051]: If this is not the case, remove /home/anarcat/.smd/lock by hand.
mar 22 16:24:00 curie smd-pull[12051]: any: smd-pushpull@localhost: TAGS: error::context(locking) probable-cause(another-instance-is-running) human-intervention(necessary) suggested-actions(run(kill 10466) run(rm /home/anarcat/.smd/lock))
mar 22 16:24:00 curie systemd[3199]: smd-pull.service: Main process exited, code=exited, status=1/FAILURE
mar 22 16:24:00 curie systemd[3199]: smd-pull.service: Failed with result 'exit-code'.
mar 22 16:24:00 curie systemd[3199]: Failed to start pull emails with syncmaildir.

... and the long push finally resolving:

mar 22 16:24:00 curie smd-push[10466]: smd-client: ERROR: Data transmission failed.
mar 22 16:24:00 curie smd-push[10466]: smd-client: ERROR: This problem is transient, please retry.
mar 22 16:24:00 curie smd-push[10466]: smd-client: ERROR: server sent ABORT or connection died
mar 22 16:24:00 curie smd-push[10466]: smd-client: ERROR: Data transmission failed.
mar 22 16:24:00 curie smd-push[10466]: smd-client: ERROR: This problem is transient, please retry.
mar 22 16:24:00 curie smd-push[10466]: smd-client: ERROR: server sent ABORT or connection died
mar 22 16:24:00 curie smd-push[10466]: smd-server: ERROR: Unable to open Maildir/.kobo/cur/1498563708.M122624P22121.marcos,S=32234,W=32792:2,S: Maildir/.kobo/cur/1498563708.M122624P22121.marcos,S=32234,W=32792:2,S: No such file or directory
mar 22 16:24:00 curie smd-push[10466]: smd-server: ERROR: The problem should be transient, please retry.
mar 22 16:24:00 curie smd-push[10466]: smd-server: ERROR: Unable to open requested file.
mar 22 16:24:00 curie smd-push[10466]: default: smd-client@smd-server-anarcat: TAGS: stats::new-mails(0), del-mails(293920), bytes-received(0), xdelta-received(26995)
mar 22 16:24:00 curie smd-push[10466]: default: smd-client@smd-server-anarcat: TAGS: error::context(receive) probable-cause(network) human-intervention(avoidable) suggested-actions(retry)
mar 22 16:24:00 curie smd-push[10466]: default: smd-server@localhost: TAGS: error::context(transmit) probable-cause(simultaneous-mailbox-edit) human-intervention(avoidable) suggested-actions(retry)
mar 22 16:24:00 curie systemd[3199]: smd-push.service: Main process exited, code=exited, status=1/FAILURE
mar 22 16:24:00 curie systemd[3199]: smd-push.service: Failed with result 'exit-code'.
mar 22 16:24:00 curie systemd[3199]: Failed to start push emails with syncmaildir.
mar 22 16:24:00 curie systemd[3199]: Starting push emails with syncmaildir...

This pattern repeats until 16:35, when that locking issue silently recovered somehow:

mar 22 16:35:03 curie systemd[3199]: Starting pull emails with syncmaildir...
mar 22 16:35:41 curie smd-pull[20788]: default: smd-client@localhost: TAGS: stats::new-mails(5), del-mails(1), bytes-received(21885), xdelta-received(6863398)
mar 22 16:35:42 curie smd-pull[21373]: register: smd-client@localhost: TAGS: stats::new-mails(0), del-mails(0), bytes-received(0), xdelta-received(215)
mar 22 16:35:42 curie systemd[3199]: smd-pull.service: Succeeded.
mar 22 16:35:42 curie systemd[3199]: Started pull emails with syncmaildir.
mar 22 16:36:35 curie systemd[3199]: Starting pull emails with syncmaildir...
mar 22 16:36:36 curie smd-pull[21738]: default: smd-client@localhost: TAGS: stats::new-mails(0), del-mails(0), bytes-received(0), xdelta-received(214)
mar 22 16:36:37 curie smd-pull[21816]: register: smd-client@localhost: TAGS: stats::new-mails(0), del-mails(0), bytes-received(0), xdelta-received(215)
mar 22 16:36:37 curie systemd[3199]: smd-pull.service: Succeeded.
mar 22 16:36:37 curie systemd[3199]: Started pull emails with syncmaildir.

... notice that huge xdelta-received there, that's 7GB right there. Mysteriously, the curie mail spool survived this, possibly because smd-pull started failing again:

mar 22 16:38:00 curie systemd[3199]: Starting pull emails with syncmaildir...
mar 22 16:38:00 curie smd-pull[23556]: 21887 ?        00:00:00 smd-push
mar 22 16:38:00 curie smd-pull[23556]: Already running.
mar 22 16:38:00 curie smd-pull[23556]: If this is not the case, remove /home/anarcat/.smd/lock by hand.
mar 22 16:38:00 curie smd-pull[23556]: any: smd-pushpull@localhost: TAGS: error::context(locking) probable-cause(another-instance-is-running) human-intervention(necessary) suggested-actions(run(kill 21887) run(rm /home/anarcat/.smd/lock))
mar 22 16:38:00 curie systemd[3199]: smd-pull.service: Main process exited, code=exited, status=1/FAILURE
mar 22 16:38:00 curie systemd[3199]: smd-pull.service: Failed with result 'exit-code'.
mar 22 16:38:00 curie systemd[3199]: Failed to start pull emails with syncmaildir.

That could have been when i got on angela to check my mail, and it was busy doing the nasty removal stuff... although the times don't match. Here is when angela came back online:

anarcat@angela:~(main)$ last
anarcat  :0           :0               Mon Mar 22 19:57   still logged in
reboot   system boot  5.10.0-0.bpo.3-a Mon Mar 22 19:57   still running
anarcat  :0           :0               Mon Mar 22 17:43 - 18:47  (01:03)
reboot   system boot  5.10.0-0.bpo.3-a Mon Mar 22 17:39   still running

Then finally the sync on curie started failing with:

mar 22 16:46:35 curie systemd[3199]: Starting pull emails with syncmaildir...
mar 22 16:46:42 curie smd-pull[27455]: smd-server: ERROR: Client aborted, removing /home/anarcat/.smd/curie-anarcat__Maildir.db.txt.new and /home/anarcat/.smd/curie-anarcat__Maildir.db.txt.mtime.new
mar 22 16:46:42 curie smd-pull[27455]: smd-client: ERROR: Failed to copy Maildir/.debian/cur/1613401668.M901837P27073.marcos,S=3740,W=3815:2,S to Maildir/.koumbit/cur/1613401640.M415457P27063.marcos,S=3790,W=3865:2,S
mar 22 16:46:42 curie smd-pull[27455]: smd-client: ERROR: The destination already exists but its content differs.
mar 22 16:46:42 curie smd-pull[27455]: smd-client: ERROR: To fix this problem you have two options:
mar 22 16:46:42 curie smd-pull[27455]: smd-client: ERROR: - rename Maildir/.koumbit/cur/1613401640.M415457P27063.marcos,S=3790,W=3865:2,S by hand so that Maildir/.debian/cur/1613401668.M901837P27073.marcos,S=3740,W=3815:2,S
mar 22 16:46:42 curie smd-pull[27455]: smd-client: ERROR:   can be copied without replacing it.
mar 22 16:46:42 curie smd-pull[27455]: smd-client: ERROR:   Executing  cd; mv -n "Maildir/.koumbit/cur/1613401640.M415457P27063.marcos,S=3790,W=3865:2,S" "Maildir/.koumbit/cur/1616446002.1.localhost"  should work.
mar 22 16:46:42 curie smd-pull[27455]: smd-client: ERROR: - run smd-push so that your changes to Maildir/.koumbit/cur/1613401640.M415457P27063.marcos,S=3790,W=3865:2,S
mar 22 16:46:42 curie smd-pull[27455]: smd-client: ERROR:   are propagated to the other mailbox
mar 22 16:46:42 curie smd-pull[27455]: default: smd-client@localhost: TAGS: error::context(copy-message) probable-cause(concurrent-mailbox-edit) human-intervention(necessary) suggested-actions(run(mv -n "/home/anarcat/.smd/workarea/Maildir/.koumbit/cur/1613401640.M415457P27063.marcos,S=3790,W=3865:2,S" "/home/anarcat/.smd/workarea/Maildir/.koumbit/tmp/1613401640.M415457P27063.marcos,S=3790,W=3865:2,S") run(smd-push default))
mar 22 16:46:42 curie systemd[3199]: smd-pull.service: Main process exited, code=exited, status=1/FAILURE
mar 22 16:46:42 curie systemd[3199]: smd-pull.service: Failed with result 'exit-code'.
mar 22 16:46:42 curie systemd[3199]: Failed to start pull emails with syncmaildir.

It went on like this until I found the problem. This is, presumably, a good thing because those emails were not being destroyed. On angela, things looked like this:

-- Reboot --
mar 22 17:39:29 angela systemd[1677]: Started run notmuch new at least once a day.
mar 22 17:39:29 angela systemd[1677]: Started run smd-pull regularly.
mar 22 17:40:46 angela systemd[1677]: Starting pull emails with syncmaildir...
mar 22 17:43:18 angela smd-pull[3916]: smd-server: ERROR: Unable to open Maildir/.tor/new/1616446842.M285912P26118.marcos,S=8860,W=8996: Maildir/.tor/new/1616446842.M285912P26118.marcos,S=886
0,W=8996: No such file or directory
mar 22 17:43:18 angela smd-pull[3916]: smd-server: ERROR: The problem should be transient, please retry.
mar 22 17:43:18 angela smd-pull[3916]: smd-server: ERROR: Unable to open requested file.
mar 22 17:43:18 angela smd-pull[3916]: smd-client: ERROR: Data transmission failed.
mar 22 17:43:18 angela smd-pull[3916]: smd-client: ERROR: This problem is transient, please retry.
mar 22 17:43:18 angela smd-pull[3916]: smd-client: ERROR: server sent ABORT or connection died
mar 22 17:43:18 angela smd-pull[3916]: default: smd-server@smd-server-anarcat: TAGS: error::context(transmit) probable-cause(simultaneous-mailbox-edit) human-intervention(avoidable) suggested
-actions(retry)
mar 22 17:43:18 angela smd-pull[3916]: default: smd-client@localhost: TAGS: error::context(receive) probable-cause(network) human-intervention(avoidable) suggested-actions(retry)
mar 22 17:43:18 angela systemd[1677]: smd-pull.service: Main process exited, code=exited, status=1/FAILURE
mar 22 17:43:18 angela systemd[1677]: smd-pull.service: Failed with result 'exit-code'.
mar 22 17:43:18 angela systemd[1677]: Failed to start pull emails with syncmaildir.
mar 22 17:43:18 angela systemd[1677]: Starting pull emails with syncmaildir...
mar 22 17:43:29 angela smd-pull[4847]: default: smd-client@localhost: TAGS: stats::new-mails(29), del-mails(0), bytes-received(401519), xdelta-received(38914)
mar 22 17:43:29 angela smd-pull[5600]: register: smd-client@localhost: TAGS: stats::new-mails(2), del-mails(0), bytes-received(92150), xdelta-received(471)
mar 22 17:43:29 angela systemd[1677]: smd-pull.service: Succeeded.
mar 22 17:43:29 angela systemd[1677]: Started pull emails with syncmaildir.
mar 22 17:43:29 angela systemd[1677]: Starting push emails with syncmaildir...
mar 22 17:43:32 angela smd-push[5693]: default: smd-client@smd-server-anarcat: TAGS: stats::new-mails(0), del-mails(0), bytes-received(0), xdelta-received(217)
mar 22 17:43:33 angela smd-push[6575]: register: smd-client@smd-server-register: TAGS: stats::new-mails(0), del-mails(0), bytes-received(0), xdelta-received(219)
mar 22 17:43:33 angela systemd[1677]: smd-push.service: Succeeded.
mar 22 17:43:33 angela systemd[1677]: Started push emails with syncmaildir.

Notice how long it took to get the first error, in that first failure: it failed after 3 minutes! Presumably that's when it started deleting all that mail. And this is during pull, not push, so the error didn't come from angela.

Affected data It seems 2GB of mail from my main INBOX was destroyed. Another 2.4GB of spam (kept for training purposes) was also destroyed, along with 700MB of Sent mail. The rest is hard to figure out, because the folders are actually still there, just smaller. So I relied on ncdu to figure out the size changes. (Note that I don't really archive (or delete much of) my mail since I use notmuch, which is why the INBOX is so large...) Concretely, according to the notmuch-new.service which still runs periodically on marcos, here are the changes that happened on the server:

mar 22 16:17:12 marcos notmuch[10729]: Added 7 new messages to the database. Removed 57985 messages. Detected 1372 file renames.
mar 22 16:22:43 marcos notmuch[12826]: No new mail. Removed 143842 messages. Detected 6072 file renames.
mar 22 16:27:02 marcos notmuch[13969]: No new mail. Removed 82071 messages. Detected 1783 file renames.
mar 22 16:29:45 marcos notmuch[15079]: Added 22743 new messages to the database. Detected 1 file rename.
mar 22 16:31:48 marcos notmuch[16196]: Added 22779 new messages to the database. Removed 5 messages.
mar 22 16:33:11 marcos notmuch[17192]: Added 3711 new messages to the database.
mar 22 16:40:41 marcos notmuch[19122]: Added 74558 new messages to the database. Detected 1 file rename.
mar 22 16:43:21 marcos notmuch[20325]: Added 9061 new messages to the database. Detected 4 file renames.
mar 22 17:43:08 marcos notmuch[7420]: Added 1793 new messages to the database. Detected 6 file renames.

That is basically the entire mail spool destroyed at first (283 898 messages), and then bits and pieces of it progressively re-added (134 645 messages), somehow, so 149 253 mails were lost, presumably.

Recovery I disabled the services all over the place:

systemctl --user --now disable smd-pull.service smd-pull.timer smd-push.service smd-push.timer notmuch-new.service notmuch-new.timer

(Well, technically, I did that only on angela, as I thought the problem was there. Luckily, curie kept going but it seems like it was harmless.) I made a backup of the mail spool on curie:

tar cf - Maildir/   pv -s 14G   gzip -c > Maildir.tgz

Then I crossed my fingers and ran smd-push -v -s, as that was suggested by smd error codes themselves. That thankfully started restoring mail. It failed a few times on weird cases of files being duplicates, but I resolved this by following the instructions. Or mostly: I actually deleted the files instead of moving them, which made smd even unhappier (if there ever was such a thing). I had to recreate some of those files, so, lesson learned: do follow the advice smd gives you, even if it seems useless or strange. But then smd-push was humming along, uploading tens of thousands of messages, saturating the upload in the office, refilling the mail spool on the server... yaay!... ? Except... well, of course that didn't quite work: the mail spool in the office eventually started to grow beyond the size of the mail spool on the workstation. That is what smd-push eventually settled on:

default: smd-client@smd-server-anarcat: TAGS: error::context(receive) probable-cause(network) human-intervention(avoidable) suggested-actions(retry)
default: smd-client@smd-server-anarcat: TAGS: error::context(receive) probable-cause(network) human-intervention(avoidable) suggested-actions(retry)
default: smd-client@smd-server-anarcat: TAGS: stats::new-mails(151697), del-mails(0), bytes-received(7539147811), xdelta-received(10881198)

It recreated 151 697 emails, adding about 2000 emails to the pool, kind of from nowhere at all. On marcos, before:

ncdu 1.13 ~ Use the arrow keys to navigate, press ? for help
--- /home/anarcat/Maildir ------------------------------------
    4,0 GiB [##########] /.notmuch
  717,3 MiB [#         ] /.Archives.2014
  498,2 MiB [#         ] /.feeds.debian-planet
  453,1 MiB [#         ] /.Archives.2012
  414,5 MiB [#         ] /.debian
  408,2 MiB [#         ] /.quoifaire
  389,8 MiB [          ] /.rapports
  356,6 MiB [          ] /.tor
  182,6 MiB [          ] /.koumbit
  179,8 MiB [          ] /tmp
   56,8 MiB [          ] /.nn
   43,0 MiB [          ] /.act-mtl
   32,6 MiB [          ] /.feeds.sysadvent
   31,7 MiB [          ] /.feeds.releases
   31,4 MiB [          ] /.Sent.2005
   26,3 MiB [          ] /.sage
   25,5 MiB [          ] /.freedombox
   24,0 MiB [          ] /.feeds.git-annex
   21,1 MiB [          ] /.Archives.2011
   19,1 MiB [          ] /.Sent.2003
   16,7 MiB [          ] /.bugtraq
   16,2 MiB [          ] /.mlug
 Total disk usage:   8,0 GiB  Apparent size:   7,6 GiB  Items: 184426

After:

ncdu 1.13 ~ Use the arrow keys to navigate, press ? for help
--- /home/anarcat/Maildir ------------------------------------
    4,7 GiB [##########] /.notmuch
    2,7 GiB [#####     ] /.junk
    1,9 GiB [###       ] /cur
  717,3 MiB [#         ] /.Archives.2014
  659,3 MiB [#         ] /.Sent
  513,9 MiB [#         ] /.Archives.2012
  498,2 MiB [#         ] /.feeds.debian-planet
  449,6 MiB [          ] /.Archives.2015
  414,5 MiB [          ] /.debian
  408,2 MiB [          ] /.quoifaire
  389,8 MiB [          ] /.rapports
  380,8 MiB [          ] /.Archives.2013
  356,6 MiB [          ] /.tor
  261,1 MiB [          ] /.Archives.2011
  240,9 MiB [          ] /.koumbit
  183,6 MiB [          ] /.Archives.2010
  179,8 MiB [          ] /tmp
  128,4 MiB [          ] /.lists
  106,1 MiB [          ] /.inso-interne
  103,0 MiB [          ] /.github
   75,0 MiB [          ] /.nanog
   69,8 MiB [          ] /.full-disclosure
 Total disk usage:  16,2 GiB  Apparent size:  15,5 GiB  Items: 341143

That is 156 717 files more. On curie:

ncdu 1.13 ~ Use the arrow keys to navigate, press ? for help
--- /home/anarcat/Maildir ------------------------------------------------------------------
    2,7 GiB [##########] /.junk
    2,3 GiB [########  ] /.notmuch
    1,9 GiB [######    ] /cur
  661,2 MiB [##        ] /.Archives.2014
  655,3 MiB [##        ] /.Sent
  512,0 MiB [#         ] /.Archives.2012
  447,3 MiB [#         ] /.Archives.2015
  438,5 MiB [#         ] /.feeds.debian-planet
  406,5 MiB [#         ] /.quoifaire
  383,6 MiB [#         ] /.debian
  378,6 MiB [#         ] /.Archives.2013
  303,3 MiB [#         ] /.tor
  296,0 MiB [#         ] /.rapports
  237,6 MiB [          ] /.koumbit
  233,2 MiB [          ] /.Archives.2011
  182,1 MiB [          ] /.Archives.2010
  127,0 MiB [          ] /.lists
  104,8 MiB [          ] /.inso-interne
  102,7 MiB [          ] /.register
   89,6 MiB [          ] /.github
   67,1 MiB [          ] /.full-disclosure
   66,5 MiB [          ] /.nanog
 Total disk usage:  13,3 GiB  Apparent size:  12,6 GiB  Items: 342465

Interestingly, there are more files, but less disk usage. It's possible the notmuch database there is more efficient. So maybe there's nothing to worry about. Last night's marcos backup has:

root@marcos:/home/anarcat# find /mnt/home/anarcat/Maildir   pv -l   wc -l
 341k 0:00:16 [20,4k/s] [                             <=>                                                                                                                                     ]
341040

... 341040 files, which seems about right, considering some mail was delivered during the day. An audit can be performed with hashdeep:

borg mount /media/sdb2/borg/::marcos-auto-2021-03-22 /mnt
hashdeep -c sha256 -r /mnt/home/anarcat/Maildir   pv -l -s 341k > Maildir-backup-manifest.txt

And then compared with:

hashdeep -c sha256 -k Maildir-backup-manifest.txt Maildir/

Some extra files should show up in the Maildir, and very few should actually be missing, because I shouldn't have deleted mail from the previous day the next day, or at least very few. The actual summary hashdeep gave me was:

hashdeep: Audit failed
   Input files examined: 0
  Known files expecting: 0
          Files matched: 339080
Files partially matched: 0
            Files moved: 782
        New files found: 107
  Known files not found: 106

So 106 files added, 107 deleted. Seems good enough for me... Postfix was stopped at Mar 22 21:12:59 to try and stop external events from confusing things even further. I reviewed the delivery log to see if mail that came in during the problem window disappeared:

grep 'dovecot:.*stored mail into mailbox' /var/log/mail.log  
  tail -20  
  sed 's/.*msgid=<//;s/>.*//'   
  while read msgid; do 
    notmuch count --exclude=false id:$msgid  
      grep 0 && echo $msgid missing;
  done

And things looked okay. Now of course if we go further back, we find mail I actually deleted (because I do do that sometimes), so it's hard to use this log as an audit trail. We can only hope that the curie spool is sufficiently coherent to be relied on. Worst case, we'll have to restore from last night's backup, but that's getting far away now: I get hundreds of mails a day in that mail spool, and reseting back to last night does not seem like a good idea. A dry run of smd-pull on angela seems to agree that it's missing some files:

default: smd-client@localhost: TAGS: stats::new-mails(154914), del-mails(0), bytes-received(0), xdelta-received(0)

... a number of mails somewhere in between the other two, go figure. A "wet" run of this was started, without deletion (-n), which gave us:

default: smd-client@localhost: TAGS: stats::new-mails(154911), del-mails(0), bytes-received(7658160107), xdelta-received(10837609)

Strange that it sync'd three less emails, but that's still better than nothing, and we have a mail spool on angela again:

anarcat@angela:~(main)$ notmuch new
purging with prefix '.': spam moved (0), ham moved (0), deleted (0), done
Note: Ignoring non-mail file: /home/anarcat/Maildir//.uidvalidity
Processed 1779 total files in 26s (66 files/sec.).
Added 1190 new messages to the database. Removed 3 messages. Detected 593 file renames.
tagging with prefix '.': spam, sent, feeds, koumbit, tor, lists, rapports, folders, done.

Notice how only 1190 messages were re-added, that is because I killed notmuch before it had time to remove all those mails from its database.

Possible causes I am totally at a loss as to why `smd` started destroying everything like it did. But a few things come to mind:

I rewired my office on that day.

This meant unplugging `curie`, the workstation.

It has a bad CMOS battery (known problem), so it jumped around the time continuum a few times, sometimes by years.

The `smd` services are ran from a systemd unit with `OnCalendar=*:0/2`. I have heard that it's possible that major time jumps "pile up" execution of jobs, and it seems this happened in this case.

It's possible that locking in `smd` is not as great as it could be, and that it corrupted its internal data structures on `curie`, which led it to command a destruction of the remote mail spool.

It's also possible that there was a disk failure on the server, `marcos`. But since it's running on a (software) RAID-1 array, and no errors have been found (according to `dmesg`), I don't think that's a plausible hypothesis.

Lessons learned

follow what `smd` says, even if it seems useless or strange.

trust but verify: just backup everything before you do anything, especially the largest data set.

daily backups are not great for email, unless you're ready to lose a day of email (which I'm not).

hashdeep is great. I keep finding new use cases for it. Last time it was to audit my camera SD card to make sure I didn't forget anything, and now this. it's fast and powerful.

borg is great too. the FUSE mount was especially useful, and it was pretty fast to explore the backup, even through that overhead: checksumming 15GB of mail took about 35 minutes, which gives a respectable 8MB/s, probably bottlenecked by the crap external USB drive I use for backups (!).

I really need to finish my backup system so that I have automated offsite backups, although in this case that would actually have been much slower (certainly not 8MB/s!).

Workarounds and solutions I setup fake-hwclock on curie, so that the next power failure will not upset my clock that badly. I am thinking of switching to ZFS or BTRFS for most of my filesystems, so that I can use filesystem snapshots (including remotely!) as a backup strategy. This seems so much more powerful than crawling the filesystem for changes, and allows for truly offsite backups protected from an attacker (hopefully). But it's a long way there. I'm also thinking of rebuilding my mail setup without `smd`. It's not the first time something like this happens with `smd`. It's the first time I am more confident it's the root cause of the problem, however, and it makes me really nervous for the future. I have used offlineimap in the past and it seems it was finally ported to Python 3 so that could be an option again. isync/mbsync is another option, which I tried before but do not remember why I didn't switch. A complete redesign with something like getmail and/or nncp could also be an option. But alas, I lack the time to go crazy with those experiments. Somehow, doing like everyone else and just going with Google still doesn't seem to be an option for me. Screw big tech. But I am afraid they will win, eventually. In any case, I'm just happy I got mail again, strangely.

pam-krb5 is a relatively simple Kerberos PAM module with no dependencies on larger infrastructure such as sssd. This is a small bug-fix release that fixes a possible double-free if krb5_cc_get_principal fails on the newly-acquired ticket cache during authentication. I am dubious this is exploitable because this temporary ticket cache should not be under the control of an attacker, but I'm putting out a release just in case. Thanks to Michael Muehle for the report. You can get the latest release from the pam-krb5 distribution page.

A recent vulnerability in GNU screen caused some people to reconsider their commitment to the venerable terminal multiplexing program. GNU screen is probably used by thousands of old sysadmins around the world to run long-standing processes and, particularly, IRC sessions, which are especially vulnerable to arbitrary garbage coming on ... screen, so to speak. So this vulnerability matters, and you should definitely pay attention to it. If you haven't switched to tmux yet, now might be a good time to get your fingers trained. But don't switch to it just yet for your IRC session, and read on for a better, more secure solution. After all, it's not because we found this flaw in screen that it doesn't exist in tmux (or your favorite terminal emulator, for that matter, a much scarier thought).

Hardening my bouncer Back in March 2019, I had already switched away from screen for IRC, but not to tmux like many did, but to dtach. I figured that I didn't actually need multiplexing to run my long-running IRC session: I just needed to be able to reattach to the terminal. That's what dtach does. No windows, no panes, and, especially, no way to start a new shell, which is exactly the kind of hardening I need. So I came up with this, to start irssi:

dtach -N /run/$USER/dtach-irssi.socket irssi

To attach:

dtach -a /run/$USER/dtach-irssi.socket

Fairly simple no? Already one attack vector gone: evil attacker can't get a new shell through my terminal multiplexer, yay.

Splitting into another user But why stop there! Why am I running irssi as my main user anyways! Let's take the lessons from good UNIX security, and run this as a separate user altogether. This requires me to create another user, say foo-irc:

adduser foo-irc

... and run it as a systemd service, because how else are you going to start this thing anyways, cron? I came up with something like this unit file:

[Unit]
Description=IRC screen session
After=network.target
[Service]
Type=simple
Environment="TERM=screen.xterm-256color"
User=%i
RuntimeDirectory=%i
ExecStart=-/usr/bin/dtach -N /run/%i/dtach-irssi.socket irssi
ExecStop=-/bin/sh -c 'echo /quit stopping service...   exec /usr/bin/dtach -p /run/%i/dtach-irssi.socket'
ExecReload=-/bin/sh -c 'echo /restart   exec /usr/bin/dtach -p /run/%i/dtach-irssi.socket'

Notice this is a service template, because of the %i stuff. I don't actually remember how to enable this thing, but let's say you drop this in /etc/systemd/system/irssi@.service, then you run:

systemctl daemon-reload

And then, not sure about this bit, instantiate that template:

systemctl enable irssi@foo-irc.service

And then this should start the irssi session:

systemctl start irssi@foo-irc.service

To access the session:

sudo -u foo-irc dtach -a /run/foo-irc/dtach-irssi.socket

Obviously, you will probably need to migrate your irssi configuration over, otherwise you'll end up with a blank, old-school irssi. Take a moment to savor the view though. Nostalgia. Ah.

Hardening irssi But this is still not enough. That pesky `foo-irc` user can still launch arbitrary commands, thanks to `irssi` `/exec` (and a generous Perl scripting environment). Let's throw the entire kitchen sink at it and see what sticks. At this point, the unit file becomes too long to just maintain in a blog post (which would be silly, but not unheard of), so just look at this git repository instead.

Password-less remote irssi The neat thing with this hardening is that I now feel comfortable enough with the setup to just add a password-less SSH key to that (basically throwaway) account: worst that can happen if someone gets a hold of that SSH key is they land in a heavily sandboxed irssi session. So yay, no password to jump on chat. Like a real client or something. Just make sure to secure the SSH key you'll deploy in authorized_keys with:

restrict,pty,command="dtach -a /run/foo-irc/dtach-irssi.socket" [...]

Obviously, make sure the keys are not writable by the user, by placing it somewhere outside their home, which might require hacking at your server's SSH configuration. Because otherwise a compromised user will be able to change his own authorized_keys, which could be bad.

Configuration management And at this point, you may have noticed that you shouldn't actually followed my instructions to the letter. Instead, just use this neat little Puppet module which does all of the above, but also include some little wrapper so that `mosh` still works. It also includes instructions on how to setup your SSH keys. Enjoy, and let me know if (or rather, how) I messed up.

Updates

it seems that dtach is not very active upstream: the last release (0.9) is from 2016, and the last commit (at the time of writing) is from 2017
dtach is not necessarily safer than screen or tmux from arbitrary input from the outside, in fact there was a vulnerability on dtach CVE-2012-3368 that led to an attacker accessing stack memory (but maybe not code execution)

after writing the Puppet module and publishing this article, I started to get weird behavior from dtach: i would leave the office at night and then return the next morning to find that I was timed out on servers. from my perspective, irssi noticed only when I re-attached the session:

09:39:51 -!- Irssi: warning Broken pipe
09:39:51 -!- Irssi: warning SSL write error: Broken pipe
09:39:51 -!- Irssi: warning SSL write error: Broken pipe
09:39:51 -!- Irssi: warning SSL write error: Broken pipe
09:39:51 -!- Irssi: warning SSL write error: Broken pipe
09:39:51 [bitlbee] -!- Irssi: Connection lost to localhost
09:39:51 -!- Irssi: warning SSL write error: Broken pipe
09:39:51 [gitter] -!- Irssi: Connection lost to irc.gitter.im
09:39:51 [OFTC] -!- Irssi: Connection lost to irc.oftc.net
09:42:34 [IMC] -!- Irssi: Connection lost to irc.indymedia.org
09:42:34 -!- Irssi: Connection lost to irc.hackint.org
09:42:34 -!- Irssi: Connection lost to chat.freenode.net
09:42:34 -!- dtach_away: Set away

from the outside I actually timed out a few minutes after I detached, which also makes for a weird asymmetry:

22:31:00 -!- anarcat [~anarcat@ocean] has quit [Ping timeout: 250 seconds]

that is eleven hours before the error I get.

the mosh wrapper script seems to not work as well as it did before. somehow just running mosh $server hangs with a blank screen instead of instantly rejoining the session. I'm not sure it is related to the timeout problem but I did rewrite the wrapper before publication. this is the old version:
```
#!/bin/sh
# inspired by https://serverfault.com/questions/749474/ssh-authorized-keys-command-option-multiple-commands
command="dtach -a /run/anarcat-irc/dtach-irssi.socket"
case "$SSH_ORIGINAL_COMMAND" in
    mosh-server*)
    exec mosh-server -- $command
    ;;
    *)
    exec $command
    ;;
esac
```
I'm thinking of trying that one out for a while to see if it's related. The weirdest thing is that mosh "un-hangs" if i reattach with plain ssh, so there's definitely something fishy going on here.

I got a dynamo hub for my bike and a fancy headlight. It's sweet, but I'm discovering that there're no standards for making tail lights work, so I just had to do some light reverse-engineering and soldering. And this is the findings. Different manufacturers do tail lights differently. Most tail lights are not connected directly to the dynamo, but to the headlight instead. Busch+M ller tail lights take an AC signal that looks very similar to what the hub is producing. You're not supposed to hook it up to the hub directly, but it does appear to work, and it's not clear how the headlight's tail-light output is different from the hub input. I haven't scoped it. Supernova tail lights work differently. Some guy on the internet reverse-engineered the headlight circuit showing an LM317 regulator producing 5.9V for the tail light. I have a Supernova E3 tail light (original one; model E161). The case says "6V", which is close to the 5.9V they give it. It wants its 6V, but I don't have a Supernova headlight, so I don't have 6V to give it. I do have a USB charger, so I have 5V instead. Giving it 5V does appear to work, but that results in less brightness than I would like. Presumably the voltage difference is to blame? I took apart the tail light. The circuitry is encased in hot glue. Scraping that off, we can see the action:

The circuit looks like this:

More or less, this is as expected. The circuit was designed to receive 6V, so the resistances (180 ohms) were selected to produce a certain amount of current. Given 5V, there's less current and less brightness. I can fix this by reducing the amount of resistance to bring the current levels up. I hooked up a power supply to produce 5V and 5.9V, and I measured the voltages in the circuit in those two states. Assume little accuracy in all of this

	5V	5.9V
V across input diode	0.8V	0.8V
V across the resistors	2.37V	3.25V
V across the LEDs	1.83V	1.85V

As expected, the voltages across the diodes are stable, and the resistors see the bulk of the voltage difference. The circuit designers wanted 3.25/180 ~ 18mA. With my reduced voltage I was getting 2.37/180 ~ 13mA instead. So I was 28% less bright than I should have been. That doesn't sound like a lot, but it's hard to tell by just looking at the thing. In any case, I can reduce the resistances to get the higher current at 5V: I want a resistance R of 180/3.25*2.37 ~ 131 ohms. In the interest of doing less work, I simply added more resistors in parallel instead of replacing the existing ones. So I need to add in parallel a resistance R such that 180R/(180+R) = 131. So I want a parallel R ~ 485 ohms. Looking through my box of parts, I don't have any nice surface-mount resistors with anywhere near the right resistance. But I do have through-hole ones at 470 ohms. Close-enough. I did some soldering gymnastics:

And I'm done. Haven't done any night rides outside yet with this setup, but in theory this should be bright-enough. And since the resistors are just burning off the excess voltage, and I'm giving it less voltage to burn off, I'm being more efficient than I would have been with 6V and the stock resistors.

This is the eleventh part in a series of blog posts. The previous post was FlashFloppy OLED display. The whole series is available here: Amiga.

It's still mostly all quiet on the Amiga project front. The last few months with both my kids at home have been too difficult and busy with no time for side projects. Primary Schools in the UK went back this week, and I've finally got around to installing the Gotek floppy emulator inside the Amiga A500's chassis. I followed A short "Internal Gotek Floppy" Installation Guide , which explains that you don't need to print (or buy) a special mount for it: Simply removing the Gotek's top plastic case is enough for it to fit inside, and you can support it with some nuts and bolts.

Gotek bolted into place

Easy in theory, but it still took me a while, with plenty of trial and error to line it up just right. I ended up using some M4 bolts and smaller (the kind that you use in electrical sockets) to support the bottom part of the Gotek case in three places: one, at the front, re-using an original vertical support for the old floppy drive; and two at the back, screwing through two holes in the A500's outer case and mainboard. If I'd chosen short enough bolts, the back of the Gotek case could have probably just rested on the ends of those bolts, but the ones I had were another few cm longer, so I drilled holes in the Gotek case to run the bolts through, and secured it with nuts.

Knob, display and mount, external

In that guide, the author has an OLED display for their Gotek that they wanted on the top of the Amiga case, so they fed the four tiny wires through the grill. I had a go at that, but I've got the OLED display and a rotary encoder, so that's 8 wires total, and damn fiddly. Instead for now I've just routed them all out through the hole left behind by the floppy drive's Button. I'm going to try and tidy it up with some heat shrink or similar. So finally, my Amiga is one "unit" again, without dangly bits hanging off it (if you ignore those 8 tiny wires for now) that I can move around to store or use much more easily.

The diffoscope maintainers are pleased to announce the release of diffoscope version 169. This version includes the following changes:

[ Chris Lamb ]
* Optimisations:
  - Use larger buffer/block sizes when extracting files from libarchive-
    based archives.
  - Use a much-shorter CSS class (instead of "diffponct") to dramatically
    reduce uncompressed HTML output.
* Logging improvements:
  - Don't emit "Unable to stat file" warning/debug messages; we have
    entirely-artificial directory entries such as ELF sections which, of
    course, never exist as filesystem files.
  - Don't emit a "Returning a FooContainer" logging message - we already emit
    "Instantiating a FooContainer" one and are unlikely to fail in the
    middle.
  - Format the report size logging messages when generating HTML reports.
  - Add the target directory when logging which directory we are extracting
    containers to.
* Miscellaneous:
  - Ignore "--debug" and similar arguments when creating a (hopefully useful)
    temporary directory.
  - Ensure all internal temporary directories have useful names.
  - Clarify a comment regarding diffoscope not extracting excluded files.
[ Vagrant Cascadian ]
* Skip a DEX-related test if the "procyon" tool is unavailable.

You find out more by visiting the project homepage.

tldr: Rust/coreutils ( https://github.com/uutils/coreutils/ ) is now available in Debian, good enough to boot a Debian with GNOME, install the top 1000 packages, build Firefox, the Linux Kernel and LLVM/Clang. Even if I wrote more than 100 patches to achieve that, it will probably be a bumpy ride for many other use cases.
It is also a terrific project to learn Rust. See the list of good first bugs. Even if I see Rust code every day at Mozilla, I was looking for an actual personal project (i.e. this isn't a Mozilla project) to learn Rust during the various COVID lockdowns. I started contributing to the alternative Coreutils developed in Rust. The project aims at proposing a drop-in replacement of the C-based GNU Coreutils, and I wanted to evaluate if this could be used to run a regular Debian. Similar to what I have done with clang.debian.net a few years ago (rebuilding the Debian archive using clang instead of gcc). I expect that most of the readers know what is the Coreutils. It is a set of programs performing simple operations (copy/move file, change permissions/ownership, etc). Even if some commands are from the 70s, they are at the base of Linux, Unix and macOS. While different implementations can be found, they are trying to remain compatible in terms of arguments, options, etc. This implementation of Coreutils isn t different! If you want to learn more about the history of Unix, I recommend this great Corecursive podcast with Brian Kernighan. While a lot of people contributed to this project, much was left to be done:

missing programs to be implemented. See https://github.com/uutils/coreutils#utilities
missing options in the various programs
code not following latest Rust best practices
lack of consistency in the code base (ex: functions with too many arguments)
lack of tests/low code coverage
Lots of failures when running the GNU Coreutils testsuite (141 tests pass on 613) - Some trivial to fix, some others harder A good way to start on a Rust project!

To start easy, I defined 4 goals for this work:

Package Coreutils in Debian/Ubuntu
Boot a Debian system with a Rust-based coreutils
Install the top 1000 packages in Debian - including GNOME
Build Firefox, the Linux Kernel and LLV/Clang

Packaging of Coreutils in Debian Packaging in Debian isn't a trivial or even simple task. It requires uploading independently all the dependencies in the archive. Rust, with its new ecosystem and small crates, is making this task significantly harder. The package is called rust-coreutils - https://tracker.debian.org/pkg/rust-coreutils For Debian/Ubuntu users, to have an idea of the complexity of packaging such applications, just run
`debtree --build-dep rust-coreutils dot -Tsvg > coreutils.svg` (should be around 1M). Since it isn't production ready, the rust-coreutils is installable in parallel with coreutils. This package does NOT replace the GNU/coreutils files (yet?), the new files are installed in `/usr/lib/cargo/bin/`. They can be used with: `export PATH=/usr/lib/cargo/bin/:$PATH` Or, uglier, overriding the files with the new ones.

Booting Debian with rust-coreutils To achieve this, because I knew I would likely break the image a few times, I created a new project to quickly install a full Debian with PXE and preseed. The project is available here: https://github.com/opencollab/qemu-debian-install-pxe-preseed/ A script to create the full qemu image: `build_qemu_debian_image.sh` A second script to boot on the newly created image: `boot.sh` Then, building and installing coreutils on the system (yeah, it is ugly - don t do that at home): `apt install rust-coreutils cd /usr/lib/cargo/bin/ for f in *; do cp -f $f /usr/bin/ done` First surprise, unlike the old init.d init system, as systemd is not relying on a series of scripts (it is mostly written in C), replacing the coreutils did not have an impact. Therefore, I didn't experience any issue during the boot process

Installing the most popular Debian packages Debian packages rely a lot on post-install scripts (stored in `/var/lib/dpkg/info/*`) to finalize and configure packages. They are (almost?) all using `/bin/sh` (or `/bin/bash)` to perform these actions. They intensively call coreutils applications. For example, in `/var/lib/dpkg/info/exim4-base.postinst`, we can find: `install -d -oDebian-exim -gadm -m2750 /var/log/exim4` With an ugly script, we can test the installations of the 1000 most popular packages one by one. Running this, some classes of issues in Rust/coreutils could be easily identified.

Implementing missing options A significant number of problems could be easily identified as a lack of support for some options. Here is a list of most of the fixes I had to implement to make this plan work:

Implement ln --relative: https://github.com/uutils/coreutils/pull/1540
`update-initramfs: Generating /boot/initrd.img-4.19.0-12-amd64 ln: error: Unrecognized option: 'r'`

Implement cp --archive: https://github.com/uutils/coreutils/pull/1580
`cp: error: Option 'archive' not yet implemented.`

Implement cp --no-dereference: https://github.com/uutils/coreutils/pull/1528
`cp: error: Option 'dereference' not yet implemented.`

Implement `sync -fs`: https://github.com/uutils/coreutils/pull/1639

Implement `install --owner & --group`: https://github.com/uutils/coreutils/pull/1641

Implement `mktemp -t` https://github.com/uutils/coreutils/pull/1677

...

Different behavior Most of the programs behaved as expected. Here is a list of differences:

install doesn't support using /dev/null as source file
`Setting up libreoffice-common (1:6.1.5-3+deb10u6) ... install: error: install: cannot install /dev/null to /etc/apparmor.d/local/usr.lib.libreoffice.program.oosplash : the source path is not an existing regular file` A limitation of rust itself https://github.com/rust-lang/rust/issues/79390

`mkdir --parent` is sometime used (instead of --parents) - https://github.com/uutils/coreutils/commit/dbc716546b352c8b81af1952915137049ed12239

"install foo.txt bar.txt" didn't work - https://github.com/uutils/coreutils/pull/1644
`install: error: target bar.txt is not a directory`

`install -d` won't fail if a directory already exists - https://github.com/uutils/coreutils/pull/1643

`ls non-existing-file` wasn't returning an error code. "if ls" was used by some scripts - https://github.com/uutils/coreutils/pull/1654

One of the issue was that `install -d /tmp/foo` could not be executed twice (while the second run with the gnu version wouldn't trigger an error if already existing) - https://github.com/uutils/coreutils/pull/1641

...

Compile Firefox, Clang and the Linux Kernel Build systems can vary significantly one from the other. To verify their usage of coreutils, I built these three major projects

Firefox As Firefox relies mostly on Python as a build system, it went smoothly. I didn t encounter any issue. The only unrelated issue that I noticed working on it was apt-key was broken because the script relied on a buggy option of mktemp.

Linux Kernel I identified only two issues compared to GNU Coreutils:

The chown command on a non-existing symlink target doesn t fail on the GNU version, the Rust one was triggering an error.
https://github.com/uutils/coreutils/pull/1694

Linux kernel
`ln -fsn ../../x86/boot/bzImage ./arch/x86_64/boot/bzImage ln: error: Unrecognized option: 'n'`

LLVM/Clang The llvm toolchain relies on Cmake. Just like for Firefox, I didn t face any issue.

Comparing with GNU coreutils using its testsuite Recently, James Robson added a new test to run the GNU testsuite on the Rust/coreutils.
`# TOTAL: 611 # PASS: 144 # SKIP: 86 # XFAIL: 0 # FAIL: 342 # XPASS: 0 # ERROR: 39`compared to 546 test passing with the GNU version. Even if a bunch of errors are just different outputs, it demonstrates that there is still a long road ahead.

Next steps & contribute First, we will need more motivated contributors to work on this project. Many features remain to be implemented, optimizations to be done (e.g. decreasing the memory usage), etc.
I started to create a list of good first bugs for newcomers: https://github.com/uutils/coreutils/issues?q=is%3Aissue+is%3Aopen+label%3A%22Good+first+bug%22
I will update this list of there is some interest for this project.
Helping improve the support of the GNU coreutils testsuite would be a huge step while being a great way to learn Rust! Then, once it is in a better state, we will be able to make it a reliable alternative in Debian/Ubuntu to the GNU/Coreutils. This might be also interesting for other folks who prefer a BSD license over a GPL.

XDG Just throwing this out for wider talk perhaps. I have been silently watching a list called xdg@lists.freedesktop.org. Now the list talks about freedesktop standards which basically is trying to have some sort of standards that all desktop environments can follow. One of the discussions on the specific list shared above is and was about New `MimeType` fields in .desktop . It is a fascinating thread with many people giving loads of interesting view points. If you are into desktops even casually, you would enjoy the discussions thoroughly. Seriously thinking, I shared the below on the list Hi all, I read quite a bit of the fascinating thread. As a user I have often had to fix the same at my end and at times had to re-fix it again. For me it s the mkv and related media file types. Now I have different applications which are for different things, For e.g. mpv is exclusively the player (followed at times by vlc) for playing the content. Mediainfo to have technical knowledge/bits about the content. ffmpeg for doing some basic stuff (extracting subtitles or adding subtitles etc.) to the content, re-encoding into smaller resolution etc etc. Sadly, none of the solutions as of date tell me if I hover or even go into properties of a mkv file which application does what. I do know this is perhaps a bit outside the discussion having, but does anybody have any ideas? At the very least, it should make the user more aware of what he can do with particular applications perhaps. One way is to perhaps have a database so if I ask in a way that the system understands which are the editors which can be used to manipulate or edit mkv files, it should give me a list available from debian database. I do know that apt, aptitude etc. do try to do something on those lines but it has many shortcomings. And as somebody else mentioned, don t think the solution would be done tomorrow but at least sometime in future. While I stopped at the above on the list, allow me to extrapolate or share a bit more where I am coming from. Somebody did share that this should be actually done by the desktop provider, and GNOME 1 did provide a solution in the early days, now then they have moved to being a more dumb desktop or in their words simplifying. In any case, while I haven t hadn t had the opportunity to see the new MAC desktop, Windows 10 desktop interface is nothing to write home about. All our DE s starting from GNOME, MATE, XFCE, could and should be giving us much more than it has, making users having richer and better experiences than they have today. And while I shared about .mkv (the Matroska Multimedia Container), the above applies and can apply to any file format which can be manipulated in different ways using different tools. I just shared the one I use most often as a use-case scenario. And this should be made easier now that we have processors have multiple cores (for e.g. Intel and AMD quad cores are now there at decent prices) . So would it be too much of an ask to have something which helps people be more productive. As of today, for anybody who is new to either Debian or any X desktop, there is still such a substantial learning curve and more so applications and how they fit in the puzzle and what can be done and what cannot be done. A much screwy thing to add to the mix is file format and which file format version can a particular app. read. The biggest and the most common culprit here is pdf. A good webpage which shares the versions and the comments which tell how people are still having issues with various versions of pdf. Most of the foss tools I use top out at pdf 1.4 which is quite an old format . I am open to testing and debugging if somebody does try decoding the new versions. For me, this is also an issue as most I.T. notices and what not we get, at times I have to try and figure out things, at times go to MS and use an acrobat reader and that is dumb.

Banking There are also lot of banking stuff that we cannot do on free software, especially in India as lot of powerful proprietary interests are there which make sure that no public API s are available, or even if there is, it would be something half-done or after back and forth, they say, this is just for show, as had shared last year. I would probably add another section later to talk about it. From what little I know, in Europe the law mandates that there are public API s not only for banking but wherever public money (read taxpayer money) is involved. Again, not all countries, but some more than others. At least, that is what I had seen over the years. Till later.

Freedom house puts India in partly free Just couple of days ago, freedom house published its 2020 rankings for all countries including India. While freedom house shared how democracy in the world has weakened, India chose to take offense about it being called partly free .
India, leader in Internet shutdowns Access Now (copyright)
The above illustration is shared by accessnow . The next big ones who have Internet shutdowns are Yemen 6 and Ethiopia 4. Such internet shutdowns have and will have sad repercussions as would share in another story as well.

Color-coding journalists A story was broken by caravan magazine yesterday and which was followed by newslaundry which shows how the Govt. is looking to just drive some narrative, does not matter whether it s true or false, it should just show that the Govt. is right and others are all wrong. As can be seen, almost all reporters barring a few have kept silent rather than refuting statements attributed to them or happenings which didn t happen. And this goes to a much larger narrative and disinformation route taken by the Govt. which doesn t have any semblance to the truth or reality as people know it. I would illustrate couple of examples below which shares that. In all my young and even adult-life I hadn t seen a Govt. this much against its own people.

Omega Seiki puts a manufacturing plant in Bangladesh Now Omega Seiki is an Indian vendor who chose or had to go to manufacture their electric vehicles in Bangladesh. Now while this is a slightly old story this was broken on social media recently. Everybody starting blaming both the vendor and saying we should break FTA (Free Trade Agreement) with Bangladesh, not knowing that despite the FTA, India has put tariff barriers between India and Bangladesh. I had to share research from Brookings to show where India has been losing. Of course, those who don t want to see, wouldn t see anything wrong in the picture.

Teen raped, asked to marry the accused when she turns 18 Now you may see the above headlines and feel it is ridiculous, but the fact is that these orders were put or given by Madras High Court couple of months back. This was then reported by both Livelaw and BarandBench respectively. Now to be truthful, this news didn t make much noise as it should have, probably as I had shared previously that the Govt. wants to lower the marriageable age to 15 or even less. And this is despite all the medical evidence on the contrary, because it assuages this Govt s masculinity. There is also the very recent case where the SC CJI asked the rapist if he is willing to marry a girl who was underage when she reaches maturity. Another one in which it seems martial rape is not a crime according to the CJI. So it seems these are the state of things in which India finds itself today. There are judges like Vrinda Grover who do question CJI but they are few and there are costs to them who ask questions. Although, as shared this news was overtaken by other news and would have remained so, if not one of the leaders of the present Govt. , a Ramesh Jarkiholi, who hails from Belagavi region of north Karnataka was caught in a sex CD scandal basically asking sexual favors for a permanent Govt. job. He had made statements after the Madras High Court case applauding the judgement given by the judge. While, due to public pressure he had to resign, but not before stating that he had everybody blue films including the Chief Minister of the State. And sad to report that six Karnataka Ministers rushed today or rather yesterday to put a petition in the civil court to restrain media from airing/printing/publishing any defamatory content against them. The court has granted a media gag against 68 media houses for the same. Sadly, the recent happening only reinforce what has been happening in Karnataka since a decade. Update 07/03/2021 Seems yesterday another 10 odd ministers rushed to get the same order. Seems different laws apply to politicians vis-a-vis others. A recent example of Rhea Chakravarthy, an actress and girlfriend of Sushant Singh who was hounded in his suicide case and many accusations made on TV but no evidence till date. From what we know as facts, Sushant committed suicide as he was not getting work due to cronyism in Bollywood. In fact, those who were behind it have white-washed themselves, deleted their tweets etc. and while the public knows, no accountability on them. In fact, there is and was so much that I wanted to share as to what has been happening to women, sadly and thankfully arre did the needful for me. They wrote an entire article which tells what the situation for women in India today is. And if you are wondering why I said, that is because when a site which was made exclusively for people to laugh and have a good time and get relief, when they start writing serious articles, you can be sure that things have gone horribly wrong

Asking Tesla to come to India and at the same time ambivalent on battery Recently, Mr. Nitin Gadkari, a prominent minister of the present Govt. invited Tesla and gave all sorts of incentives to start a manufacturing plant here in India. And while it seems that Tesla has accepted, looking at the Vodafone case, hopefully Tesla does make such contracts where if something goes wrong and they need to sue the Govt. they can do it in States or elsewhere. The way the Govt. acted in the Vodafone case had been a dampener to any MNC investments so far. Although to be fair to both Tesla and GOI, the basic models even if they are manufactured in India will go to less than 1% of the population. The cheapest Tesla Model Y which retails in the U.S. for USD 40k would be around INR 30 lakh. And this is their cheapest car to date. I do know there are rumors of the 25k but that is probably 2-3 years away as shared by Tesla China President Tom Zhu in an interview shared on YT.

https://www.youtube.com/watch?v=aH5leMWFBxI There are couple of interesting comments being made. The fact that China is going to fully open its automobile market to western companies shows how confident China feels about their own vendors. And I m not much impressed about Tesla as I am about the tiny car revolution happening in China. India, if it wanted to, could learn many lessons from China. Even the electric buses they had started in 2010 itself where people in our auto industry thought it was all a fad. Sadly, we are missing most of the technology and if and by the time Tesla starts a production line, dunno where we could get our lithium. India hasn t been as aggressive as other countries when it comes to securing raw natural resources in other countries, as some other countries have. Even besides that, it has been tough when you have so many people who still believe that ICE vehicles (Internal Combustion Engines) are better than EV s and even if they know they choose to believe the propaganda. Couple of months ago a young UK girl who had died due to asthma, an inquest found that air pollution was a factor. The girl s name was Ella Adoo-Kissi-Debrah. This is the first case where the doctor ruled air pollution as a chief factor in a person s death. Probably first of its kind of ruling anywhere in the world. I also shared TCO studies between BEV and ICE vehicles done by people who are considering an electric vehicle but those studies seem to fall on deaf ears.

Starlink This is another of Elon Musk s ventures and would be a money spinner for sure in around the globe. While Starlink has asked TRAI for permission, I don t think they will get it. There is also Bharti Global s Oneweb which probably has a better chance of getting permissions. The reason is censorship. As shared above, India is now a leader in Internet shutdowns and do see this trend only accelerate rather than go the other way around. For people who don t remember, remember how satellite phones were made illegal even though only businessmen could afford it. And this was just 5 years ago. As shared Oneweb would have better shot as they would accept all Government directives without a second thought. Unless Starlink gives a binding to the Govt. to be a willing partner when it wants to have internet shutdowns, it will not work. Now how Elon approaches that is to be seen and known. FWIW, you can t access Starlink webpage on BSNL broadband. My broadband provider gives at the most 300 kbps and sometimes, at late nights or early mornings, around 500 kbps.

Farmer Protests
Lastly, farmer protests have entered 100 days. In the interim, Vivek Kaul, an economist took stock of the Bihar APMC to see if things have really worked as the Govt. supporters were telling. The investigation and the results didn t inspire the confidence as the Govt. said. The sad part is though, that nowadays nobody, at least those in power as well as those who are supporters are keen to read, understand and even argue otherwise. They are all happy with whatsapp knowledge. Till date 200+ people have died in the farmer protests. All mainstream media houses have stopped talking about farmers in the hope that they will disappear. At the end of the day the Govt. wants that the corporates should win at whatever the cost.

Debian Code Search now offers an OpenAPI-based API! Various developers have created ad-hoc client libraries based on how the web interface works. The goal of offering an OpenAPI-based API is to provide developers with automatically generated client libraries for a large number of programming languages, that target a stable interface independent of the web interface s implementation details.

Getting started

Visit https://codesearch.debian.net/apikeys/ to download your personal API key. Login via Debian s GitLab instance salsa.debian.org; register there if you have no account yet.

Find the Debian Code Search client library for your programming language. If none exists yet, auto-generate a client library on editor.swagger.io: click Generate Client .

Search all code in Debian from your own analysis tool, migration tracking dashboard, etc.

curl example

curl \
  -H "x-dcs-apikey: $(cat dcs-apikey-stapelberg.txt)" \
  -X GET \
  "https://codesearch.debian.net/api/v1/search?query=i3Font&match_mode=regexp"

Web browser example You can try out the API in your web browser in the OpenAPI documentation.

Code example (Go) Here s an example program that demonstrates how to set up an auto-generated Go client for the Debian Code Search OpenAPI, run a query, and aggregate the results:

func burndown() error  
	cfg := openapiclient.NewConfiguration()
	cfg.AddDefaultHeader("x-dcs-apikey", apiKey)
	client := openapiclient.NewAPIClient(cfg)
	ctx := context.Background()
	// Search through the full Debian Code Search corpus, blocking until all
	// results are available:
	results, _, err := client.SearchApi.Search(ctx, "fmt.Sprint(err)", &openapiclient.SearchApiSearchOpts 
		// Literal searches are faster and do not require escaping special
		// characters, regular expression searches are more powerful.
		MatchMode: optional.NewString("literal"),
	 )
	if err != nil  
		return err
	 
	// Print to stdout a CSV file with the path and number of occurrences:
	wr := csv.NewWriter(os.Stdout)
	header := []string "path", "number of occurrences" 
	if err := wr.Write(header); err != nil  
		return err
	 
	occurrences := make(map[string]int)
	for _, result := range results  
		occurrences[result.Path]++
	 
	for _, result := range results  
		o, ok := occurrences[result.Path]
		if !ok  
			continue
		 
		// Print one CSV record per path:
		delete(occurrences, result.Path)
		record := []string result.Path, strconv.Itoa(o) 
		if err := wr.Write(record); err != nil  
			return err
		 
	 
	wr.Flush()
	return wr.Error()

The full example can be found under burndown.go.

Feedback? File a GitHub issue on `github.com/Debian/dcs` please!

Migration status I m aware of the following third-party projects using Debian Code Search:

Tool Migration status

Debian Code Search CLI tool Updated to OpenAPI

identify-incomplete-xs-go-import-path Update pending

gnome-codesearch makes no API queries

If you find any others, please point them to this post in case they are not using Debian Code Search s OpenAPI yet.

Tool	Migration status
Debian Code Search CLI tool	Updated to OpenAPI
identify-incomplete-xs-go-import-path	Update pending
gnome-codesearch	makes no API queries

Hi there. Long time no write! On Tuesday, February 23, 2021, I made an announcement at debian-devel-announce about a new service that I configured for Debian: a debuginfod server. This post serves two purposes: pay the promise I made to Jonathan Carter that I would write a blog post about the service, and go into a bit more detail about it. What's debuginfod? From the announcement above:

debuginfod is a new-ish project whose purpose is to serve
ELF/DWARF/source-code information over HTTP.  It is developed under the
elfutils umbrella.  You can find more information about it here:
  https://sourceware.org/elfutils/Debuginfod.html
In a nutshell, by using a debuginfod service you will not need to
install debuginfo (a.k.a. dbgsym) files anymore; the symbols will be
served to GDB (or any other debuginfo consumer that supports debuginfod)
over the network.  Ultimately, this makes the debugging experience much
smoother (I myself never remember the full URL of our debuginfo
repository when I need it).

Perhaps not everybody knows this, but until last year I was a Debugger Engineer (a.k.a. GDB hacker) at Red Hat. I was not involved with the creation of debuginfod directly, but I witnessed discussions about "having way to serve debug symbols over the internet" multiple times during my tenure at the company. So this is not a new idea, and it's not even the first implementation, but it's the first time that some engineers actually got their hands dirty enough to have something concrete to show. The idea to set up a debuginfod server for Debian started to brew after 2019's GNU Tools Cauldron, but as usual several things happened in $LIFE (including a global pandemic and leaving Red Hat and starting a completely different job at Canonical) which had the effect of shuffling my TODO list "a little". Benefits for Debian Debian unfortunately is lagging behind when it comes to offer its users a good debugging experience. Before the advent of our debuginfod server, if you wanted to debug a package in Debian you would need to:

Add the debian-debug apt repository to your /etc/apt/sources.list.
Install the dbgsym package that contains the debug symbols for the package you are debugging. Note that the version of the dbgsym package needs to be exactly the same as the version of the package you want to debug.
Figure out which shared libraries your package uses and install the dbgsym packages for all of them. Arguably, this step is optional but recommended if you would like to perform a more in-depth debugging.
Download the package source, possibly using apt source or some equivalent command.
Open GDB, and make sure you adjust the source paths properly (more below). This can be non-trivial.
Finally, debug the program.

Now, with the new service, you will be able to start from step 4, without having to mess with sources.list, dbgsym packages and version mismatches. The package source It is important to mention an existing (but perhaps not well-known) limitation of our debugging experience in Debian: the need to manually download the source packages and adjust GDB to properly find them (see step 4 above). debuginfod is able to serve source code as well, but our Debian instance is not doing that at the moment. Debian does not provide a patched source tree that is ready to be consumed by GDB or debuginfod (for a good example of a distribution that does this, see Fedora's debugsource packages). Let me show you an example of debugging GDB itself (using debuginfod) on Debian:

$ HOME=/tmp DEBUGINFOD_URLS=https://debuginfod.debian.net gdb -q gdb
Reading symbols from gdb...
Downloading separate debug info for /tmp/gdb...
Reading symbols from /tmp/.cache/debuginfod_client/02046bac4352940d19d9164bab73b2f5cefc8c73/debuginfo...
(gdb) start
Temporary breakpoint 1 at 0xd18e0: file /build/gdb-Nav6Es/gdb-10.1/gdb/gdb.c, line 28.
Starting program: /usr/bin/gdb 
Downloading separate debug info for /lib/x86_64-linux-gnu/libreadline.so.8...
Downloading separate debug info for /lib/x86_64-linux-gnu/libz.so.1...
Downloading separate debug info for /lib/x86_64-linux-gnu/libncursesw.so.6...
Downloading separate debug info for /lib/x86_64-linux-gnu/libtinfo.so.6...
Downloading separate debug info for /tmp/.cache/debuginfod_client/d6920dbdd057f44edaf4c1fbce191b5854dfd9e6/debuginfo...
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Downloading separate debug info for /lib/x86_64-linux-gnu/libexpat.so.1...
Downloading separate debug info for /lib/x86_64-linux-gnu/liblzma.so.5...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libbabeltrace.so.1...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libbabeltrace-ctf.so.1...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libipt.so.2...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libmpfr.so.6...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libsource-highlight.so.4...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libxxhash.so.0...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libdebuginfod.so.1...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libstdc++.so.6...
Downloading separate debug info for /lib/x86_64-linux-gnu/libgcc_s.so.1...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0...
Downloading separate debug info for /tmp/.cache/debuginfod_client/dbfea245d26065975b4084f4e9cd2d83c65973ee/debuginfo...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libdw.so.1...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libelf.so.1...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libuuid.so.1...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libgmp.so.10...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libboost_regex.so.1.74.0...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4...
Downloading separate debug info for /lib/x86_64-linux-gnu/libbz2.so.1.0...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libicui18n.so.67...
Downloading separate debug info for /tmp/.cache/debuginfod_client/acaa831dbbc8aa70bb2131134e0c83206a0701f9/debuginfo...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libicuuc.so.67...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libnghttp2.so.14...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libidn2.so.0...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/librtmp.so.1...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libssh2.so.1...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libpsl.so.5...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libnettle.so.8...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libgnutls.so.30...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libbrotlidec.so.1...
Downloading separate debug info for /tmp/.cache/debuginfod_client/39739740c2f8a033de95c1c0b1eb8be445610b31/debuginfo...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libunistring.so.2...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libhogweed.so.6...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libgcrypt.so.20...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libp11-kit.so.0...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libtasn1.so.6...
Downloading separate debug info for /lib/x86_64-linux-gnu/libcom_err.so.2...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libsasl2.so.2...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libbrotlicommon.so.1...
Downloading separate debug info for /lib/x86_64-linux-gnu/libgpg-error.so.0...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libffi.so.7...
Downloading separate debug info for /lib/x86_64-linux-gnu/libkeyutils.so.1...
Temporary breakpoint 1, main (argc=1, argv=0x7fffffffebf8) at /build/gdb-Nav6Es/gdb-10.1/gdb/gdb.c:28
28      /build/gdb-Nav6Es/gdb-10.1/gdb/gdb.c: Directory not empty.
(gdb) list
23      in /build/gdb-Nav6Es/gdb-10.1/gdb/gdb.c
(gdb)

(See all those Downloading separate debug info for... lines? Nice!) As you can see, when we try to list the contents of the file we're in, nothing shows up. This happens because GDB doesn't know where the file is. So you have to tell it. In this case, it's relatively easy: you see that the GDB package's build directory is /build/gdb-Nav6Es/gdb-10.1/. When you apt source gdb, you will end up with a directory called $PWD/gdb-10.1/ containing the full source of the package. Notice that the last directory's name in both paths is the same, so in this case we can use GDB's

set
substitute-path

command do the job for us (in this example $PWD is /tmp/):

$ HOME=/tmp DEBUGINFOD_URLS=https://debuginfod.debian.net gdb -q gdb
Reading symbols from gdb...
Reading symbols from /tmp/.cache/debuginfod_client/02046bac4352940d19d9164bab73b2f5cefc8c73/debuginfo...
(gdb) set substitute-path /build/gdb-Nav6Es/ /tmp/
(gdb) start
Temporary breakpoint 1 at 0xd18e0: file /build/gdb-Nav6Es/gdb-10.1/gdb/gdb.c, line 28.
Starting program: /usr/bin/gdb 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Temporary breakpoint 1, main (argc=1, argv=0x7fffffffebf8) at /build/gdb-Nav6Es/gdb-10.1/gdb/gdb.c:28
warning: Source file is more recent than executable.
28        memset (&args, 0, sizeof args);
(gdb) list
23      int
24      main (int argc, char **argv)
25       
26        struct captured_main_args args;
27
28        memset (&args, 0, sizeof args);
29        args.argc = argc;
30        args.argv = argv;
31        args.interpreter_p = INTERP_CONSOLE;
32        return gdb_main (&args);
(gdb)

Much better, huh? The problem is that this process is manual, and changes depending on how the package you're debugging was built. What can we do to improve this? What I personally would like to see is something similar to what the Fedora project already does: create a new debug package which will contain the full, patched source package. This would mean changing our building infrastructure and possibly other somewhat complex things. Using the service (by default) At the time of this writing, I am working on an elfutils Merge Request whose purpose is to implement a debconf question to ask the user whether she wants to use our service by default. If you would like to start using the service right now, all you have to do is set the following environment variable in your shell:

DEBUGINFOD_URLS="https://debuginfod.debian.net"

More information You can find more information about our debuginfod service here. Try to keep an eye on the page as it's being constantly updated. If you'd like to get in touch with me, my email is my domain at debian dot org. I sincerely believe that this service is a step in the right direction, and hope that it can be useful to you :-).

Search Results: "cas"

30 March 2021

Above all, just be nice. Be nice. It s basically the best thing you can do for free software.

28 March 2021

27 March 2021

24 March 2021

23 March 2021

20 March 2021

19 March 2021

13 March 2021

12 March 2021

9 March 2021

Compile Firefox, Clang and the Linux Kernel Build systems can vary significantly one from the other. To verify their usage of coreutils, I built these three major projects

Firefox As Firefox relies mostly on Python as a build system, it went smoothly. I didn t encounter any issue. The only unrelated issue that I noticed working on it was apt-key was broken because the script relied on a buggy option of mktemp.

LLVM/Clang The llvm toolchain relies on Cmake. Just like for Firefox, I didn t face any issue.

7 March 2021

6 March 2021

curl example curl \ -H "x-dcs-apikey: $(cat dcs-apikey-stapelberg.txt)" \ -X GET \ "https://codesearch.debian.net/api/v1/search?query=i3Font&match_mode=regexp"

Web browser example You can try out the API in your web browser in the OpenAPI documentation.

Feedback? File a GitHub issue on github.com/Debian/dcs please!

5 March 2021

curl example
`curl \ -H "x-dcs-apikey: $(cat dcs-apikey-stapelberg.txt)" \ -X GET \ "https://codesearch.debian.net/api/v1/search?query=i3Font&match_mode=regexp"`

Feedback? File a GitHub issue on `github.com/Debian/dcs` please!