Last month, I traveled to Kenya to attend a conference called State of the Map 2024 ( SotM for short), which is an annual meetup of OpenStreetMap contributors from all over the world. It was held at the University of Nairobi Towers in Nairobi, from the 6th to the 8th of September.
University of Nairobi.
I have been contributing to OpenStreetMap for the last three years, and this conference seemed like a great opportunity to network with others in the community. As soon as I came across the travel grant announcement, I jumped in and filled the form immediately. I was elated when I was selected for the grant and couldn t wait to attend. The grant had an upper limit of 1200 and covered food, accommodation, travel and miscellaneous expenses such as visa fee.
Pre-travel tasks included obtaining Kenya s eTA and getting a yellow fever vaccine. Before the conference, Mikko from the Humanitarian OpenStreetMap Team introduced me to Rabina and Pragya from Nepal, Ibtehal from Bangladesh, and Sajeevini from Sri Lanka. We all booked the Nairobi Transit Hotel, which was within walking distance of the conference venue. Pragya, Rabina, and I traveled together from Delhi to Nairobi, while Ibtehal was my roommate in the hotel.
Our group at the conference.
The venue, University of Nairobi Towers, was a tall building and the conference was held on the fourth, fifth and sixth floors. The open area on the fifth floor of the building had a nice view of Nairobi s skyline and was a perfect spot for taking pictures. Interestingly, the university had a wing dedicated to Mahatma Gandhi, who is regarded in India as the Father of the Nation.
View of Nairobi's skyline from the open area on the fifth floor.
A library in Mahatma Gandhi wing of the University of Nairobi.
The diversity of the participants was mind-blowing, with people coming from a whopping 54 countries. I was surprised to notice that I was the only participant traveling from India, despite India having a large OpenStreetMap community. That said, there were two other Indian participants who traveled from other countries. I finally got to meet Arnalie (from the Phillipines) and Letwin (from Zimbabwe), both of whom I had only met online before. I had met Anisa (from Albania) earlier during DebConf 2023. But I missed Mikko and Honey from the Humanitarian OpenStreetMap Team, whom I knew from the Open Mapping Guru program.
I learned about the extent of OSM use through Pragya and Rabina s talk; about the logistics of running the OSM Board, in the OSMF (OpenStreetMap Foundation) session; about the Youth Mappers from Sajeevini, about the OSM activities in Malawi from Priscilla Kapolo, and about mapping in Zimbabwe from Letwin. However, I missed Ibtehal s lightning session. The ratio of women speakers and participants at the conference was impressive, and I hope we can get such gender representation in our Delhi/NCR mapping parties.
One of the conference halls where talks took place.
Outside of talks, the conference also had lunch and snack breaks, giving ample time for networking with others. In the food department, there were many options for a lacto-ovo vegetarian like myself, including potatoes, rice, beans, chips etc. I found out that the milk tea in Kenya (referred to as white tea ) is usually not as strong compared to India, so I switched to coffee (which is also called white coffee when taken with milk). The food wasn t spicy, but I can t complain :) Fruit juices served as a nice addition to lunch.
One of the lunch meals served during the conference.
At the end of the second day of the conference, there was a surprise in store for us a bus ride to the Bao Box restaurant. The ride gave us the experience of a typical Kenyan matatu (privately-owned minibuses used as share taxis), complete with loud rap music. I remember one of the songs being Kraff s Nursery Rhymes. That day, I was wearing an original Kenyan cricket jersey - one that belonged to Dominic Wesonga, who represented Kenya in four ODIs. This confused Priscilla Kapolo, who asked if I was from Kenya! Anyway, while it served as a good conversation starter, it didn t attract as much attention as I expected :) I had some pizza and chips there, and later some drinks with Ibtehal. After the party, Piyush went with us to our hotel and we played a few games of UNO.
Minibus which took us from the university to Bao Box restaurant.
This minibus in the picture gave a sense of a real matatu.
I am grateful to the organizers Laura and Dorothea for introducing me to Nikhil when I was searching for a companion for my post-conference trip. Nikhil was one of the aforementioned Indian participants, and a wildlife lover. We had some nice conversations; he wanted to go to the Masai Maara Natural Reserve, but it was too expensive for me. In addition, all the safaris were multi-day affairs, and I wasn t keen on being around wildlife for that long. Eventually I chose to go my own way, exploring the coastal side and visiting Mombasa.
While most of the work regarding the conference was done using free software (including the reimbursement form and Mastodon announcements), I was disappointed by the use of WhatsApp for coordination with the participants. I don t use WhatsApp and so was left out. WhatsApp is proprietary software (they do not provide the source code) and users don t control it. It is common to highlight that OpenStreetMap is controlled by users and the community, rather than a company - this should apply to WhatsApp as well.
My suggestion is to use XMPP, which shares similar principles with OpenStreetMap, as it is privacy-respecting, controlled by users, and powered by free software. I understand the concern that there might not be many participants using XMPP already. Although it is a good idea to onboard people to free software like XMPP, we can also create a Matrix group, and bridge it with both the XMPP group and the Telegram group. In fact, using Matrix and bridging it with Telegram is how I communicated with the South Asian participants. While it s not ideal - as Telegram s servers are proprietary and centralized - but it s certainly much better than creating a WhatsApp-only group. The setup can be bridged with IRC as well. On the other hand, self-hosted mailing lists for participants is also a good idea.
Finally, I would like to thank SotM for the generous grant, enabling me to attend this conference, meet the diverse community behind OSM and visit the beautiful country of Kenya. Stay tuned for the blog post on Kenya trip.
Thanks to Sahilister, Contrapunctus, Snehal and Badri for reviewing the draft of this blog post before publishing.
Another short status update of what happened on my side last
month. Besides the usual amount of housekeeping last month was a lot
about getting old issues resolved by finishing some stale merge
requests and work in pogress MRs. I also pushed out the Phosh 0.42.0
Releasephosh
Mark mobile-data quick setting as insensitive when modem is off (MR)
Sanitize versions as this otherwise breaks the libphosh-rs build (MR)
lockscreen: Swap deck and carousel to avoid triggering the plugins page when entering pin and let the lockscreen shrink to smaller sizes (MR) (two more year old usability issues out of the way)
Ensure we send enough feedback when phone is blanked/locked (MR). This should be way easier now for apps as they
don't need to do anything and we can avoid duplicate feedback sent from e.g. Chatty.
Fix possible use after free when activating notifications on the lock screen (MR)
Don't lose preedit when switching applications, opening menus, etc (MR). This fixes the case (e.g. with word completion in phosh-osk-stub enabled) where it looks to the user as if the last typed word would get lost when switching from a text editor to another app or when opening a menu
Fix word salad with presage completer when entering cursor navigation mode (and in some other cases)
(MR 1). Presage
has the best completion but was marked experimental due to that.
Submit preedit on changes to terminal and emoji layout (MR)
Unbreak and modernize CI a bit (MR). A passing CI is so much more motivating for contributers and reviewers.
Fotema
Fix app-id and hence the icon shown in Phosh's overview (MR)
Help Development
If you want to support my work see donations. This includes
a list of hardware we want to improve support for. Thanks a lot to all current and past donors.
Vagrant Cascadian: Could you tell me a bit about yourself? What sort
of things do you work on?
Kees Cook: I m a Free Software junkie living in Portland, Oregon, USA.
I have been focusing on the upstream Linux kernel s protection
of itself. There is a lot of support that the kernel provides
userspace to defend itself, but when I first started focusing on this
there was not as much attention given to the kernel protecting
itself. As userspace got more hardened the kernel itself became a
bigger target. Almost 9 years ago I formally announced the Kernel Self-Protection Project
because the work necessary was way more than my time and expertise could do
alone. So I just try to get people to help as much as possible; people who
understand the ARM architecture, people who understand the memory management
subsystem to help, people who understand how to make the kernel less buggy.
Vagrant: Could you describe the path that lead you to working on this
sort of thing?
Kees: I have always been interested in security through the aspect of
exploitable flaws. I always thought it was like a magic trick to make a
computer do something that it was very much not designed to do and seeing how
easy it is to subvert bugs. I wanted to improve that fragility. In 2006, I
started working at Canonical on Ubuntu and was mainly focusing on bringing
Debian and Ubuntu up to what was the state of the art for Fedora and Gentoo s
security hardening efforts. Both had really pioneered a lot of userspace
hardening with compiler flags and ELF
stuff and many other things for hardened
binaries. On the whole, Debian had not really paid attention to it. Debian s
packaging building process at the time was sort of a chaotic free-for-all as
there wasn t centralized build methodology for defining things. Luckily that
did slowly change over the years. In Ubuntu we had the opportunity to apply top
down build rules for hardening all the packages. In 2011 Chrome OS was
following along and took advantage of a bunch of the security hardening work as
they were based on ebuild out of Gentoo
and when they looked for someone to
help out they reached out to me. We recognized the Linux kernel was pretty much
the weakest link in the Chrome OS security posture and I joined them to help
solve that. Their userspace was pretty well handled but the kernel had a lot
of weaknesses, so focusing on hardening was the next place to go. When I
compared notes with other users of the Linux kernel within Google there were a
number of common concerns and desires. Chrome OS already had an upstream
first requirement, so I tried to consolidate the concerns and solve them
upstream. It was challenging to land anything in other kernel team repos at
Google, as they (correctly) wanted to minimize their delta from upstream, so I
needed to work on any major improvements entirely in upstream and had a lot of
support from Google to do that. As such, my focus shifted further from working
directly on Chrome OS into being entirely upstream and being more of a
consultant to internal teams, helping with integration or sometimes
backporting. Since the volume of needed work was so gigantic I needed to find
ways to inspire other developers (both inside and outside of Google) to help.
Once I had a budget I tried to get folks paid (or hired) to work on these areas
when it wasn t already their job.
Vagrant: So my understanding of some of your recent work is basically
defining undefined behavior in the language or compiler?
Kees: I ve found the term undefined behavior to have a really strict
meaning within the compiler community, so I have tried to redefine my goal as
eliminating unexpected behavior or ambiguous language constructs . At the
end of the day ambiguity leads to bugs, and bugs lead to exploitable security
flaws. I ve been taking a four-pronged approach: supporting the work people are
doing to get rid of ambiguity, identify new areas where ambiguity needs to be
removed, actually removing that ambiguity from the C language, and then dealing
with any needed refactoring in the Linux kernel source to adapt to the new
constraints.
None of this is particularly novel; people have recognized how dangerous some
of these language constructs are for decades and decades but I think it is a
combination of hard problems and a lot of refactoring that nobody has the
interest/resources to do. So, we have been incrementally going after the lowest
hanging fruit. One clear example in recent years was the elimination of C s
implicit fall-through in switch statements. The language would just fall
through between adjacent cases if a break (or other code flow directive)
wasn t present. But this is ambiguous: is the code meant to fall-through, or
did the author just forget a break statement? By defining the [[fallthrough]] statement,
and requiring its use in
Linux,
all switch statements now have explicit code flow, and the entire class of
bugs disappeared. During our refactoring we actually found that 1 in 10 added
[[fallthrough]] statements were actually missing break statements. This
was an extraordinarily common bug!
So getting rid of that ambiguity is where we have been. Another area I ve been
spending a bit of time on lately is looking at how defensive security work has
challenges associated with metrics. How do you measure your defensive security
impact? You can t say because we installed locks on the doors, 20% fewer
break-ins have happened. Much of our signal is always secondary or
retrospective, which is frustrating: This class of flaw was used X much over
the last decade so, and if we have eliminated that class of flaw and will never
see it again, what is the impact? Is the impact infinity? Attackers will just
move to the next easiest thing. But it means that exploitation gets
incrementally more difficult. As attack surfaces are reduced, the expense of
exploitation goes up.
Vagrant: So it is hard to identify how effective this is how bad would it be
if people just gave up?
Kees: I think it would be pretty bad, because as we have seen, using
secondary factors, the work we have done in the industry at large, not just the
Linux kernel, has had an impact. What we, Microsoft, Apple, and everyone else
is doing for their respective software ecosystems, has shown that the price of
functional exploits in the black market has gone up. Especially for really
egregious stuff like a zero-click remote code execution.
If those were cheap then obviously we are not doing something right, and it
becomes clear that it s trivial for anyone to attack the infrastructure that
our lives depend on. But thankfully we have seen over the last two decades that
prices for exploits keep going up and up into millions of dollars. I think it
is important to keep working on that because, as a central piece of modern
computer infrastructure, the Linux kernel has a giant target painted on it. If
we give up, we have to accept that our computers are not doing what they were
designed to do, which I can t accept. The safety of my grandparents shouldn t
be any different from the safety of journalists, and political activists, and
anyone else who might be the target of attacks. We need to be able to trust our
devices otherwise why use them at all?
Vagrant: What has been your biggest success in recent years?
Kees: I think with all these things I am not the only actor. Almost
everything that we have been successful at has been because of a lot
of people s work, and one of the big ones that has been coordinated
across the ecosystem and across compilers was initializing stack variables to 0 by default.
This feature was added in
Clang,
GCC,
and MSVC
across the board even though there were a lot of fears about forking the C language.
The worry was that developers would come to depend on zero-initialized stack
variables, but this hasn t been the case because we still warn about
uninitialized variables when the compiler can figure that out. So you still
still get the warnings at compile time but now you can count on the contents of
your stack at run-time and we drop an entire class of uninitialized variable flaws.
While the exploitation of this class has mostly been around memory content
exposure, it has also been used for control flow attacks.
So that was politically and technically a large challenge: convincing people it
was necessary, showing its utility, and implementing it in a way that everyone
would be happy with, resulting in the elimination of a large and persistent
class of flaws in C.
Vagrant: In a world where things are generally Reproducible do you see ways
in which that might affect your work?
Kees: One of the questions I frequently get is, What version of the Linux
kernel has feature $foo? If I know how things are built, I can answer with
just a version number. In a Reproducible Builds scenario I can count on the
compiler version, compiler flags, kernel configuration, etc. all those things
are known, so I can actually answer definitively that a certain feature exists.
So that is an area where Reproducible Builds affects me most directly.
Indirectly, it is just being able to trust the binaries you are running are
going to behave the same for the same build environment is critical for sane
testing.
Vagrant: Have you used diffoscope?
Kees: I have! One subset of tree-wide refactoring that we do when getting
rid of ambiguous language usage in the kernel is when we have to make source
level changes to satisfy some new compiler requirement but where the binary
output is not expected to change at all. It is mostly about getting the
compiler to understand what is happening, what is intended in the cases where
the old ambiguity does actually match the new unambiguous description of what
is intended. The binary shouldn t change. We have
used diffoscope to compare
the before and after binaries to confirm that yep, there is no change in
binary .
Vagrant: You cannot just use checksums for that?
Kees: For the most part, we need to only compare the text segments. We try
to hold as much stable as we can, following the
Reproducible Builds documentation for the kernel,
but there are macros in the kernel that are sensitive to source line numbers
and as a result those will change the layout of the data segment (and sometimes
the text segment too). With diffoscope there s flexibility where I can exclude
or include different comparisons. Sometimes I just go look at what diffoscope
is doing and do that manually, because I can tweak that a little harder, but
diffoscope is definitely the default. Diffoscope is awesome!
Vagrant: Where has reproducible builds affected you?
Kees: One of the notable wins of reproducible builds lately was
dealing with the fallout of the XZ backdoor and just being able to ask
the question is my build environment running the expected
code? and to be able to compare the output generated from one
install that never had a vulnerable XZ and one that did have a
vulnerable XZ and compare the results of what you get. That was
important for kernel builds because the XZ threat actor was working to
expand their influence and capabilities to include Linux kernel
builds, but they didn t finish their work before they were noticed. I
think what happened with Debian proving the build infrastructure was not affected is an
important example of how people would have needed to verify the kernel
builds too.
Vagrant: What do you want to see for the near or distant future in security work?
Kees: For reproducible builds in the kernel, in the work that has been
going on in the ClangBuiltLinux project, one of the driving forces of
code and usability quality has been the
continuous integration work. As soon as something breaks, on the
kernel side, the Clang side, or something in between the two, we get a
fast signal and can chase it and fix the bugs quickly. I would like to
see someone with funding to maintain a reproducible kernel build
CI. There have been places where there are certain
architecture configurations or certain build configuration where we lose
reproducibility and right now we have sort of a standard open source
development feedback loop where those things get fixed but the time
in between introduction and fix can be large. Getting a CI for
reproducible kernels would give us the opportunity to shorten that
time.
Vagrant: Well, thanks for that! Any last closing thoughts?
Kees: I am a big fan of reproducible builds, thank you for all your work.
The world is a safer place because of it.
Vagrant: Likewise for your work!
For more information about the Reproducible Builds project, please see our website at
reproducible-builds.org. If you are interested in
ensuring the ongoing security of the software that underpins our civilisation
and wish to sponsor the Reproducible Builds project, please reach out to the
project by emailing
contact@reproducible-builds.org.
Hey everyone!
The 2024 Linux Display Next
hackfest concluded
in May, and its outcomes continue to shape the Linux Display stack. Igalia
hosted this year s event in A Coru a, Spain, bringing together leading experts
in the field. Samuel Iglesias and I
organized this year s edition and this blog post summarizes the experience and
its fruits.
One of the highlights of this year s hackfest was the wide range of backgrounds
represented by our 40 participants (both on-site and remotely). Developers and
experts from various companies and open-source projects came together to
advance the Linux Display ecosystem. You can find the list of participants
here.
The event covered a broad spectrum of topics affecting the development of Linux
projects, user experiences, and the future of display technologies on Linux.
From cutting-edge topics to long-term discussions, you can check the event
agenda
here.
Organization Highlights
The hackfest was marked by in-depth discussions and knowledge sharing among
Linux contributors, making everyone inspired, informed, and connected to the
community. Building on feedback from the previous year, we refined the
unconference format to enhance participant preparation and engagement.
Structured Agenda and Timeboxes: Each session had a defined scope, time
limit (1h20 or 2h10), and began with an introductory talk on the topic.
Participant-Led Discussions: We pre-selected in-person participants to
lead discussions, allowing them to prepare introductions, resources, and
scope.
Transparent Scheduling: The schedule was shared in advance as GitHub
issues, encouraging participants to review and prepare for sessions of
interest.
Engaging Sessions: The hackfest featured a variety of topics, including
presentations and discussions on how participants were addressing specific
subjects within their companies.
No Breakout Rooms, No Overlaps: All participants chose to attend all
sessions, eliminating the need for separate breakout rooms. We also adapted
run-time schedule to keep everybody involved in the same topics.
Real-time Updates: We provided notifications and updates through
dedicated emails and the event matrix room.
Strengthening Community Connections: The hackfest offered ample
opportunities for networking among attendees.
Social Events: Igalia sponsored coffee breaks, lunches, and a dinner at
a local restaurant.
Museum Visit: Participants enjoyed a sponsored visit to the Museum of
Estrela Galicia Beer (MEGA).
Fruitful Discussions and Follow-up
The structured agenda and breaks allowed us to cover multiple topics during the
hackfest. These discussions have led to new display feature development and
improvements, as evidenced by patches, merge requests, and implementations in
project repositories and mailing lists.
With the KMS color management API taking shape, we discussed refinements and
best approaches to cover the variety of color pipeline from different
hardware-vendors. We are also investigating techniques for a performant
SDR<->HDR content reproduction and reducing latency and power consumption when
using the color blocks of the hardware.
Color Management/HDR
Color Management and HDR continued to be the hottest topic of the hackfest. We
had three sessions dedicated to discuss Color and HDR across Linux Display
stack layers.
Color/HDR (Kernel-Level)
Harry Wentland (AMD) led this session.
Here, kernel Developers shared the Color Management pipeline of AMD, Intel and
NVidia. We counted with diagrams and explanations from HW-vendors developers
that discussed differences, constraints and paths to fit them into the KMS
generic color management properties such as advertising modeset needs,
IN\_FORMAT, segmented LUTs,
interpolation types, etc. Developers from Qualcomm and ARM also added
information regarding their hardware.
Upstream work related to this session:
Color/HDR (Compositor-Level)
Sebastian Wick (RedHat) led this session.
It started with Sebastian s presentation covering Wayland color protocols and
compositor implementation. Also, an explanation of APIs provided by Wayland and
how they can be used to achieve better color management for applications and
discussions around ICC profiles and color representation metadata. There was
also an intensive Q&A about LittleCMS with Marti Maria.
Upstream work related to this session:
Color/HDR (Use Cases and Testing)
Christopher Cameron (Google) and Melissa Wen (Igalia) led this session.
In contrast to the other sessions, here we focused less on implementation and
more on brainstorming and reflections of real-world SDR and HDR transformations
(use and validation) and gainmaps. Christopher gave a nice presentation
explaining HDR gainmap images and how we should think of HDR. This presentation
and Q&A were important to put participants at the same page of how to
transition between SDR and HDR and somehow emulating HDR.
We also discussed on the usage of a kernel background color property.
Finally, we discussed a bit about
Chamelium
and the future of VKMS (future work and maintainership).
Power Savings vs Color/Latency
Mario Limonciello (AMD) led this session.
Mario gave an introductory presentation about AMD ABM (adaptive backlight
management) that is similar to Intel DPST. After some discussions, we agreed on
exposing a kernel property for power saving policy. This work was already
merged on kernel and the userspace support is under development.
Upstream work related to this session:
Strategy for video and gaming use-cases
Leo Li (AMD) led this session.
Miguel Casas (Google) started this session with a presentation of Overlays in
Chrome/OS Video, explaining the main goal of power saving by switching off GPU
for accelerated compositing and the challenges of different colorspace/HDR for
video on Linux.
Then Leo Li presented different strategies for video and gaming and we
discussed the userspace need of more detailed feedback mechanisms to understand
failures when offloading. Also, creating a debugFS interface came up as a tool
for debugging and analysis.
Real-time scheduling and async KMS API
Xaver Hugl (KDE/BlueSystems) led this session.
Compositor developers have exposed some issues with doing real-time scheduling
and async page flips. One is that the Kernel limits the lifetime of realtime
threads and if a modeset takes too long, the thread will be killed and thus the
compositor as well. Also, simple page flips take longer than expected and
drivers should optimize them.
Another issue is the lack of feedback to compositors about hardware programming
time and commit deadlines (the lastest possible time to commit). This is
difficult to predict from drivers, since it varies greatly with the type of
properties. For example, color management updates take much longer.
In this regard, we discusssed implementing a hw_done callback to timestamp
when the hardware programming of the last atomic commit is complete. Also an
API to pre-program color pipeline in a kind of A/B scheme. It may not be
supported by all drivers, but might be useful in different ways.
VRR/Frame Limit, Display Mux, Display Control, and more and beer
We also had sessions to discuss a new KMS API to mitigate headaches on VRR
and Frame Limit as different brightness level at different refresh rates,
abrupt changes of refresh rates, low frame rate compensation (LFC) and precise
timing in VRR more.
On Display Control we discussed features missing in the current KMS
interface for HDR mode, atomic backlight settings, source-based tone mapping,
etc. We also discussed the need of a place where compositor developers can post
TODOs to be developed by KMS people.
The Content-adaptive Scaling and Sharpening session focused on sharpening
and scaling filters. In the Display Mux session, we discussed proposals to
expose the capability of dynamic mux switching display signal between discrete
and integrated GPUs.
In the last session of the 2024 Display Next Hackfest, participants
representing different compositors summarized current and future work and built
a Linux Display wish list , which includes: improvements to VTTY and HDR
switching, better dmabuf API for multi-GPU support, definition of tone mapping,
blending and scaling sematics, and wayland protocols for advertising to clients
which colorspaces are supported.
We closed this session with a status update on feature development by
compositors, including but not limited to: plane offloading (from libcamera to
output) / HDR video offloading (dma-heaps) / plane-based scrolling for web
pages, color management / HDR / ICC profiles support, addressing issues such as
flickering when color primaries don t match, etc.
After three days of intensive discussions, all in-person participants went to a
guided tour at the Museum of Extrela Galicia beer (MEGA), pouring and tasting
the most famous local beer.
Feedback and Future Directions
Participants provided valuable feedback on the hackfest, including suggestions
for future improvements.
Schedule and Break-time Setup: Having a pre-defined agenda and schedule
provided a better balance between long discussions and mental refreshments,
preventing the fatigue caused by endless discussions.
Action Points: Some participants recommended explicitly asking for action
points at the end of each session and assigning people to follow-up tasks.
Remote Participation: Remote attendees appreciated the inclusive setup
and opportunities to actively participate in discussions.
Technical Challenges: There were bandwidth and video streaming issues
during some sessions due to the large number of participants.
Thank you for joining the 2024 Display Next Hackfest
We can t help but thank the 40 participants, who engaged in-person or virtually
on relevant discussions, for a collaborative evolution of the Linux display
stack and for building an insightful agenda.
A big thank you to the leaders and presenters of the nine sessions: Christopher
Cameron (Google), Harry Wentland (AMD), Leo Li (AMD), Mario Limoncello (AMD),
Sebastian Wick (RedHat) and Xaver Hugl (KDE/BlueSystems) for the effort in
preparing the sessions, explaining the topic and guiding discussions. My
acknowledge to the others in-person participants that made such an effort to
travel to A Coru a: Alex Goins (NVIDIA), David Turner (Raspberry Pi), Georges
Stavracas (Igalia), Joan Torres (SUSE), Liviu Dudau (Arm), Louis Chauvet
(Bootlin), Robert Mader (Collabora), Tian Mengge (GravityXR), Victor Jaquez
(Igalia) and Victoria Brekenfeld (System76). It was and awesome opportunity to
meet you and chat face-to-face.
Finally, thanks virtual participants who couldn t make it in person but
organized their days to actively participate in each discussion, adding
different perspectives and valuable inputs even remotely: Abhinav Kumar
(Qualcomm), Chaitanya Borah (Intel), Christopher Braga (Qualcomm), Dor Askayo
(Red Hat), Jiri Koten (RedHat), Jonas dahl (Red Hat), Leandro Ribeiro
(Collabora), Marti Maria (Little CMS), Marijn Suijten, Mario Kleiner, Martin
Stransky (Red Hat), Michel D nzer (Red Hat), Miguel Casas-Sanchez (Google),
Mitulkumar Golani (Intel), Naveen Kumar (Intel), Niels De Graef (Red Hat),
Pekka Paalanen (Collabora), Pichika Uday Kiran (AMD), Shashank Sharma (AMD),
Sriharsha PV (AMD), Simon Ser, Uma Shankar (Intel) and Vikas Korjani (AMD).
We look forward to another successful Display Next hackfest, continuing to
drive innovation and improvement in the Linux display ecosystem!
Hey everyone!
The 2024 Linux Display Next
hackfest concluded
in May, and its outcomes continue to shape the Linux Display stack. Igalia
hosted this year s event in A Coru a, Spain, bringing together leading experts
in the field. Samuel Iglesias and I
organized this year s edition and this blog post summarizes the experience and
its fruits.
One of the highlights of this year s hackfest was the wide range of backgrounds
represented by our 40 participants (both on-site and remotely). Developers and
experts from various companies and open-source projects came together to
advance the Linux Display ecosystem. You can find the list of participants
here.
The event covered a broad spectrum of topics affecting the development of Linux
projects, user experiences, and the future of display technologies on Linux.
From cutting-edge topics to long-term discussions, you can check the event
agenda
here.
Organization Highlights
The hackfest was marked by in-depth discussions and knowledge sharing among
Linux contributors, making everyone inspired, informed, and connected to the
community. Building on feedback from the previous year, we refined the
unconference format to enhance participant preparation and engagement.
Structured Agenda and Timeboxes: Each session had a defined scope, time
limit (1h20 or 2h10), and began with an introductory talk on the topic.
Participant-Led Discussions: We pre-selected in-person participants to
lead discussions, allowing them to prepare introductions, resources, and
scope.
Transparent Scheduling: The schedule was shared in advance as GitHub
issues, encouraging participants to review and prepare for sessions of
interest.
Engaging Sessions: The hackfest featured a variety of topics, including
presentations and discussions on how participants were addressing specific
subjects within their companies.
No Breakout Rooms, No Overlaps: All participants chose to attend all
sessions, eliminating the need for separate breakout rooms. We also adapted
run-time schedule to keep everybody involved in the same topics.
Real-time Updates: We provided notifications and updates through
dedicated emails and the event matrix room.
Strengthening Community Connections: The hackfest offered ample
opportunities for networking among attendees.
Social Events: Igalia sponsored coffee breaks, lunches, and a dinner at
a local restaurant.
Museum Visit: Participants enjoyed a sponsored visit to the Museum of
Estrela Galicia Beer (MEGA).
Fruitful Discussions and Follow-up
The structured agenda and breaks allowed us to cover multiple topics during the
hackfest. These discussions have led to new display feature development and
improvements, as evidenced by patches, merge requests, and implementations in
project repositories and mailing lists.
With the KMS color management API taking shape, we discussed refinements and
best approaches to cover the variety of color pipeline from different
hardware-vendors. We are also investigating techniques for a performant
SDR<->HDR content reproduction and reducing latency and power consumption when
using the color blocks of the hardware.
Color Management/HDR
Color Management and HDR continued to be the hottest topic of the hackfest. We
had three sessions dedicated to discuss Color and HDR across Linux Display
stack layers.
Color/HDR (Kernel-Level)
Harry Wentland (AMD) led this session.
Here, kernel Developers shared the Color Management pipeline of AMD, Intel and
NVidia. We counted with diagrams and explanations from HW-vendors developers
that discussed differences, constraints and paths to fit them into the KMS
generic color management properties such as advertising modeset needs,
IN\_FORMAT, segmented LUTs,
interpolation types, etc. Developers from Qualcomm and ARM also added
information regarding their hardware.
Upstream work related to this session:
Color/HDR (Compositor-Level)
Sebastian Wick (RedHat) led this session.
It started with Sebastian s presentation covering Wayland color protocols and
compositor implementation. Also, an explanation of APIs provided by Wayland and
how they can be used to achieve better color management for applications and
discussions around ICC profiles and color representation metadata. There was
also an intensive Q&A about LittleCMS with Marti Maria.
Upstream work related to this session:
Color/HDR (Use Cases and Testing)
Christopher Cameron (Google) and Melissa Wen (Igalia) led this session.
In contrast to the other sessions, here we focused less on implementation and
more on brainstorming and reflections of real-world SDR and HDR transformations
(use and validation) and gainmaps. Christopher gave a nice presentation
explaining HDR gainmap images and how we should think of HDR. This presentation
and Q&A were important to put participants at the same page of how to
transition between SDR and HDR and somehow emulating HDR.
We also discussed on the usage of a kernel background color property.
Finally, we discussed a bit about
Chamelium
and the future of VKMS (future work and maintainership).
Power Savings vs Color/Latency
Mario Limonciello (AMD) led this session.
Mario gave an introductory presentation about AMD ABM (adaptive backlight
management) that is similar to Intel DPST. After some discussions, we agreed on
exposing a kernel property for power saving policy. This work was already
merged on kernel and the userspace support is under development.
Upstream work related to this session:
Strategy for video and gaming use-cases
Leo Li (AMD) led this session.
Miguel Casas (Google) started this session with a presentation of Overlays in
Chrome/OS Video, explaining the main goal of power saving by switching off GPU
for accelerated compositing and the challenges of different colorspace/HDR for
video on Linux.
Then Leo Li presented different strategies for video and gaming and we
discussed the userspace need of more detailed feedback mechanisms to understand
failures when offloading. Also, creating a debugFS interface came up as a tool
for debugging and analysis.
Real-time scheduling and async KMS API
Xaver Hugl (KDE/BlueSystems) led this session.
Compositor developers have exposed some issues with doing real-time scheduling
and async page flips. One is that the Kernel limits the lifetime of realtime
threads and if a modeset takes too long, the thread will be killed and thus the
compositor as well. Also, simple page flips take longer than expected and
drivers should optimize them.
Another issue is the lack of feedback to compositors about hardware programming
time and commit deadlines (the lastest possible time to commit). This is
difficult to predict from drivers, since it varies greatly with the type of
properties. For example, color management updates take much longer.
In this regard, we discusssed implementing a hw_done callback to timestamp
when the hardware programming of the last atomic commit is complete. Also an
API to pre-program color pipeline in a kind of A/B scheme. It may not be
supported by all drivers, but might be useful in different ways.
VRR/Frame Limit, Display Mux, Display Control, and more and beer
We also had sessions to discuss a new KMS API to mitigate headaches on VRR
and Frame Limit as different brightness level at different refresh rates,
abrupt changes of refresh rates, low frame rate compensation (LFC) and precise
timing in VRR more.
On Display Control we discussed features missing in the current KMS
interface for HDR mode, atomic backlight settings, source-based tone mapping,
etc. We also discussed the need of a place where compositor developers can post
TODOs to be developed by KMS people.
The Content-adaptive Scaling and Sharpening session focused on sharpening
and scaling filters. In the Display Mux session, we discussed proposals to
expose the capability of dynamic mux switching display signal between discrete
and integrated GPUs.
In the last session of the 2024 Display Next Hackfest, participants
representing different compositors summarized current and future work and built
a Linux Display wish list , which includes: improvements to VTTY and HDR
switching, better dmabuf API for multi-GPU support, definition of tone mapping,
blending and scaling sematics, and wayland protocols for advertising to clients
which colorspaces are supported.
We closed this session with a status update on feature development by
compositors, including but not limited to: plane offloading (from libcamera to
output) / HDR video offloading (dma-heaps) / plane-based scrolling for web
pages, color management / HDR / ICC profiles support, addressing issues such as
flickering when color primaries don t match, etc.
After three days of intensive discussions, all in-person participants went to a
guided tour at the Museum of Extrela Galicia beer (MEGA), pouring and tasting
the most famous local beer.
Feedback and Future Directions
Participants provided valuable feedback on the hackfest, including suggestions
for future improvements.
Schedule and Break-time Setup: Having a pre-defined agenda and schedule
provided a better balance between long discussions and mental refreshments,
preventing the fatigue caused by endless discussions.
Action Points: Some participants recommended explicitly asking for action
points at the end of each session and assigning people to follow-up tasks.
Remote Participation: Remote attendees appreciated the inclusive setup
and opportunities to actively participate in discussions.
Technical Challenges: There were bandwidth and video streaming issues
during some sessions due to the large number of participants.
Thank you for joining the 2024 Display Next Hackfest
We can t help but thank the 40 participants, who engaged in-person or virtually
on relevant discussions, for a collaborative evolution of the Linux display
stack and for building an insightful agenda.
A big thank you to the leaders and presenters of the nine sessions: Christopher
Cameron (Google), Harry Wentland (AMD), Leo Li (AMD), Mario Limoncello (AMD),
Sebastian Wick (RedHat) and Xaver Hugl (KDE/BlueSystems) for the effort in
preparing the sessions, explaining the topic and guiding discussions. My
acknowledge to the others in-person participants that made such an effort to
travel to A Coru a: Alex Goins (NVIDIA), David Turner (Raspberry Pi), Georges
Stavracas (Igalia), Joan Torres (SUSE), Liviu Dudau (Arm), Louis Chauvet
(Bootlin), Robert Mader (Collabora), Tian Mengge (GravityXR), Victor Jaquez
(Igalia) and Victoria Brekenfeld (System76). It was and awesome opportunity to
meet you and chat face-to-face.
Finally, thanks virtual participants who couldn t make it in person but
organized their days to actively participate in each discussion, adding
different perspectives and valuable inputs even remotely: Abhinav Kumar
(Qualcomm), Chaitanya Borah (Intel), Christopher Braga (Qualcomm), Dor Askayo,
Jiri Koten (RedHat), Jonas dahl (Red Hat), Leandro Ribeiro
(Collabora), Marti Maria (Little CMS), Marijn Suijten, Mario Kleiner, Martin
Stransky (Red Hat), Michel D nzer (Red Hat), Miguel Casas-Sanchez (Google),
Mitulkumar Golani (Intel), Naveen Kumar (Intel), Niels De Graef (Red Hat),
Pekka Paalanen (Collabora), Pichika Uday Kiran (AMD), Shashank Sharma (AMD),
Sriharsha PV (AMD), Simon Ser, Uma Shankar (Intel) and Vikas Korjani (AMD).
We look forward to another successful Display Next hackfest, continuing to
drive innovation and improvement in the Linux display ecosystem!
In this guide, we'll implement Secure Boot by taking full control of the device,
removing preinstalled keys, and installing our own. For a comprehensive overview
of the benefits and process, refer to this excellent post from rodsbooks.
Key Components
To implement Secure Boot, we need three essential keys:
Platform Key (PK): The top-level key in Secure Boot, typically provided by
the motherboard manufacturer. We'll replace the vendor-supplied PK with our
own for complete control.
Key Exchange Key (KEK): Used to sign updates for the Signatures Database and
Forbidden Signatures Database.
Database Key (DB): Used to sign or verify binaries (bootloaders, boot
managers, shells, drivers, etc.).
There's also a Forbidden Signature Key (dbx), which is the opposite of the DB
key. We won't be generating this key in this guide.
Preparing for Key Enrollment
Before enrolling our keys, we need to put the device in Secure Boot Setup
Mode. Verify the status using the bootctl status command. You should see
output similar to the following image:
Generating Keys
Follow these instructions from the Arch Wiki
to generate the keys manually. You'll need the efitools and openssl
packages. I recommend using rsa:2048 as the key size for better
compatibility with older firmware.
After generating the keys, copy all .auth files to the
/efi/loader/keys/<hostname>/ folder. For example:
Enrolling Keys in Firmware
Use systemd-boot to enroll your keys:
systemctlreboot--boot-loader-menu=0
Select the enroll option with your hostname in the systemd-boot menu.
After key enrollment, the system will reboot into the newly signed kernel.
Verify with bootctl:
Dealing with Lockdown Mode
Secure Boot enables lockdown mode on distro-shipped kernels, which restricts
certain features like kprobes/BPF and DKMS drivers. To avoid this, consider
compiling the upstream kernel directly, which doesn't enable lockdown mode by
default.
As Linus Torvalds has stated, "there is no reason to tie Secure Boot to lockdown
LSM." You can read more about Torvalds' opinion on UEFI tied with lockdown.
Next Steps
One thing that remains is automating the signing of systemd-boot on upgrade,
which is currently a manual process. I'm exploring dpkg triggers for achieving
this, and if I succeed, I will write a new post with details.
Acknowledgments
Special thanks to my anonymous colleague who provided invaluable assistance
throughout this process.
After years on the waiting list, May First was just
given a /24 block of IP addresses. Excellent.
Now we want to start using them for, among other things, sending email.
I haven t added a new IP address to our mail relays in a while and things seems
to change regularly in the world of email so I m curious: what s the best 2024
way to warm up IP addresses, particularly using postfix?
Sendergrid has a nice page on the
topic. It
establishes the number of messages to send per day. But I m not entirely sure
how to fit messages per day into our setup.
We use round robin DNS to direct email to one of several dozen email relay
servers using postfix. And unfortunately our DNS software
(knot) doesn t
have a way to add weights to ensure some IPs show up more often than others
(much less limit the specific number of messages a given relay should get).
Postfix has some nice knobs for rate limiting, particularly:
default_destination_recipient_limit
and
default_destination_rate_delay
If default_destination_recipient_limit is over 1, then
default_destination_rate_delay is equal to the minimum delay between sending
email to the same domain.
So, I m staring our IP addresses out at 30m - which prevents any single domain
from receiving more than 2 messages per hour. Sadly, there are a lot of
different domain names that deliver to the same set of popular corporate MX
servers, so I am not sure I can accurately control how many messages a given
provider sees coming from a given IP address. But it s a start.
A bigger problem is that messages that exceed the limit hang out in the
active queue until they can be sent without violating the rate limit. Since I
can t fully control the number of messages a given queue receives (due to my
inability to control the DNS round robin weights), a lot of messages are going
to be severely delayed, especially ones with an @gmail.com domain name.
I know I can temporarily set relayhost to a different queue and flush
deferred messages, however, as far as I can tell, it doesn t work with
active messages.
To help mitigate the problem I m only using our bulk mail queue to warm up IPs,
but really, this is not ideal.
Suggestions welcome!
Update #1
If you are running postfix in a multi-instance setup and you have instances
that are already warmed up, you can move active messages between queues with
these steps:
# Put the message on hold in the warming up instance
postsuper -c /etc/postfix-warmingup -h $queueid
# Copy to a warmed up instance
cp --preserve=mode,ownership,timestamp /var/spool/postfix-warmingup/hold/$queueid /var/spool/postfix-warmedup/incoming/
# Queue the message
postqueue -c /etc/postfix-warmedup -i $queueid
# Delete from the original queue.
postsuper -c /etc/postfix-warmingup -d $queueid
After just 12 hours we had thousands of messages piling up. This warm up method
was never going to work without the ability to move them to a faster queue.
[Additional update: be sure to reload the postfix instance after flushing the queue so
messages are drained from the active queue on the correct schedule. See update #4.]
Update #2
After 24 hours, most email is being accepted as far as I can tell. I am still
getting a small percentage of email deferred by Yahoo with:
421 4.7.0 [TSS04] Messages from 204.19.241.9 temporarily deferred due to unexpected volume or user complaints - 4.16.55.1; see https://postmaster.yahooinc.com/error-codes (in reply
So I will keep it as 30m for another 24 hours or so and then move to 15m. Now
that I can flush the backlog of active messages I am in less of a hurry.
Update #3
Well, this doesn t seem to be working the way I want it to.
When a message arrives faster than the designated rate limit, it remains in the active queue.
I m entirely sure how the timing is supposed to work, but at this point I m
down to a 5m rate delay, and the active messages are just hanging out for a lot
longer than 5m. I tried flushing the queue, but that only seems to affect the
deferred messages. I finally got them re-tried with systemctl reload. I
wonder if there is a setting to control this retry? Or better yet, why can t
these messages that exceed the rate delayed be deferred instead?
Update #4
I think I see why I was confused in Update #3 about the timing. I suspect that
when I move messages out of the active queue it screws up the timer. Reloading
the instance resets the timer. Every time you muck with active messages, you
should reload.
The relational model is probably the one innovation that brought computers to the mainstream for business users. This article by Donald Chamberlin, creator of one of the first query languages (that evolved into the ubiquitous SQL), presents its history as a commemoration of the 50th anniversary of his publication of said query language.
The article begins by giving background on information processing before the advent of today s database management systems: with systems storing and processing information based on sequential-only magnetic tapes in the 1950s, adopting a record-based, fixed-format filing system was far from natural. The late 1960s and early 1970s saw many fundamental advances, among which one of the best known is E. F. Codd s relational model. The first five pages (out of 12) present the evolution of the data management community up to the 1974 SIGFIDET conference. This conference was so important in the eyes of the author that, in his words, it is the event that starts the clock on 50 years of relational databases.
The second part of the article tells about the growth of the structured English query language (SEQUEL) eventually renamed SQL including the importance of its standardization and its presence in commercial products as the dominant database language since the late 1970s. Chamberlin presents short histories of the various implementations, many of which remain dominant names today, that is, Oracle, Informix, and DB2. Entering the 1990s, open-source communities introduced MySQL, PostgreSQL, and SQLite.
The final part of the article presents controversies and criticisms related to SQL and the relational database model as a whole. Chamberlin presents the main points of controversy throughout the years: 1) the SQL language lacks orthogonality; 2) SQL tables, unlike formal relations, might contain null values; and 3) SQL tables, unlike formal relations, may contain duplicate rows. He explains the issues and tradeoffs that guided the language design as it unfolded. Finally, a section presents several points that explain how SQL and the relational model have remained, for 50 years, a winning concept, as well as some thoughts regarding the NoSQL movement that gained traction in the 2010s.
This article is written with clear language and structure, making it easy and pleasant to read. It does not drive a technical point, but instead is a recap on half a century of developments in one of the fields most important to the commercial development of computing, written by one of the greatest authorities on the topic.
A legit email went to spam. Here are the redacted, relevant headers:
[redacted]
X-Spam-Flag: YES
X-Spam-Level: ******
X-Spam-Status: Yes, score=6.3 required=5.0 tests=DKIM_SIGNED,DKIM_VALID,
[redacted]
* 1.0 RCVD_IN_XBL RBL: Received via a relay in Spamhaus XBL
* [185.220.101.64 listed in xxxxxxxxxxxxx.zen.dq.spamhaus.net]
* 3.0 RCVD_IN_SBL_CSS Received via a relay in Spamhaus SBL-CSS
* 2.5 RCVD_IN_AUTHBL Received via a relay in Spamhaus AuthBL
* 0.0 RCVD_IN_PBL Received via a relay in Spamhaus PBL
[redacted]
[very first received line follows...]
Received: from [10.137.0.13] ([185.220.101.64])
by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-378956d2ee6sm12487760f8f.83.2024.09.11.15.05.52
for <xxxxx@mayfirst.org>
(version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
Wed, 11 Sep 2024 15:05:53 -0700 (PDT)
At first I though a Gmail IP address was listed in spamhaus - I even opened a
ticket. But then I realized it wasn t the last hop that Spamaus is complaining
about, it s the first hop, specifically the ip 185.220.101.64 which appears
to be a Tor exit node.
The sender is using their own client to relay email directly to Gmail. Like any
sane person, they don t trust Gmail to protect their privacy, so they are
sending via Tor. But WTF, Gmail is not stripping the sending IP address from
the header.
I m a big fan of harm reduction and have always considered using your own
client to relay email with Gmail as a nice way to avoid some of the
surveillance tax Google imposes.
However, it seems that if you pursue this option you have two unpleasant
choices:
Embed your IP address in every email message or
Use Tor and have your email messages go to spam
I supposed you could also use a VPN, but I doubt the IP reputation of most VPN
exit nodes are going to be more reliable than Tor.
The debridement operation was a success: nothing bad grew afterwards. I was
discharged after a couple of nights with crutches, instructions not to
weight-bear, a remarkable, portable negative-pressure "Vac" pump that lived by
my side, and some strong painkillers.
About two weeks later, I had a skin graft. The surgeon took some skin from my
thigh and stitched it over the debridement wound. I was discharged same-day,
again with the Vac pump, and again with instructions not to weight-bear, at
least for a few days.
This time I only kept the Vac pump for a week, and after a dressing change
(the first time I saw the graft), I was allowed to walk again. Doing so is
strangely awkward, and sometimes a little painful. I have physio exercises
to help me regain strength and understanding about what I can do.
The donor site remained bandaged for another week before I saw it. I was
expecting a stitched cut, but the surgeons have removed the top few layers
only, leaving what looks more like a graze or sun-burn. There are four
smaller, tentative-looking marks adjacent, suggesting they got it right on
the fifth attempt. I'm not sure but I think these will all fade away to
near-invisibility with time, and they don't hurt at all.
I've now been off work for roughly 12 weeks, but I think I am returning
very soon. I am looking forward to returning to some sense of normality.
It's been an interesting experience. I thought about writing more
about what I've gone through, in particular my experiences in Hospital,
dealing with the bureaucracy and things falling "between the gaps".
Hanif Kureishi has done a better job than I could.
It's clear that the NHS is staffed by incredibly passionate people, but
there are a lot of structural problems that interfere with care.
Review: The Book That Broke the World, by Mark Lawrence
Series:
Library Trilogy #2
Publisher:
Ace
Copyright:
2024
ISBN:
0-593-43796-9
Format:
Kindle
Pages:
366
The Book That Broke the World is high fantasy and a direct sequel
to The Book That Wouldn't Burn. You
should not start here. In a delightful break from normal practice, the
author provides a useful summary of the previous volume at the start of
this book to jog your memory.
At the end of The Book That Wouldn't Burn, the characters were
scattered and in various states of corporeality after some major
revelations about the nature of the Library and the first appearance of
the insectile Skeer. The Book That Wouldn't Burn picks up where it
left off, and there is a lot more contact with the Skeer, but my guess
that they would be the next viewpoint characters does not pan out.
Instead, we get a new group and a new protagonist: Celcha, whose sees
angels who come to visit her brother.
I have complaints, but before I launch into those, I should say that I
liked this book apart from the totally unnecessary cannibalism. (I'll get
to that.) Livira is a bit sidelined, which is regrettable, but Celcha and
her brother are interesting new characters, and both Arpix and Clovis,
supporting characters in the first book, get some excellent character
development. Similar to the first book, this is a puzzle box story full
of world-building tidbits with intellectually-satisfying interactions.
Lawrence elaborates and complicates his setting in ways that don't
contradict earlier parts of the story but create more room and depth for
the characters to be creative. I came away still invested in this world
and eager to find out how Lawrence pulls the world-building and narrative
threads together.
The biggest drawback of this book is that it's not new. My thought after
finishing the first book of the series was that if Lawrence had enough
world-building ideas to fill three books to that same level of density,
this had the potential of being one of my favorite fantasy series of all
time. By the end of the second book, I concluded that this is not the
case. Instead of showing us new twists and complications the way the
first book did throughout, The Book That Broke the World mostly
covers the same thematic ground from some new angles. It felt like
Lawrence was worried the reader of the first book may not have understood
the theme or the world-building, so he spent most of the second book
nailing down anything that moved.
I found that frustrating. One of the best parts of The Book That
Wouldn't Burn was that Lawrence trusted the reader to keep up, which for
me hit the glorious but rare sweet spot of pacing where I was figuring out
the world at roughly the same pace as the characters. It surprised me in
some very enjoyable ways. The Book That Broke the World did not
surprise me. There are a few new things, which I enjoyed, and a few
elaborations and developments of ideas, which I mostly enjoyed, but I saw
the big plot twist coming at least fifty pages before it happened and
found the aftermath more annoying than revelatory. It doesn't help that
the plot rests on character misunderstandings, one of my least favorite
tropes.
One of the other disappointments of this book is that the characters stop
using the Library as a library. The Library at the center of this series
is a truly marvelous piece of world-building with numerous fascinating
features that are unrelated to its contents, but Livira used it first and
foremost as a repository of books. The first book was full of characters
solving problems by finding a relevant book and reading it.
In The Book That Broke the World, sadly, this is mostly gone. The
Library is mostly reduced to a complicated Big Dumb Object setting. It's
still a delightful bit of world-building, and we learn about a few new
features, but I only remember two places where the actual books are
important to the story. Even the book referenced in the title is mostly
important as an artifact with properties unrelated to the words that it
contains or to the act of reading it. I think this is a huge lost
opportunity and something I hope Lawrence fixes in the last book of the
trilogy.
This book instead focuses on the politics around the existence of the
Library itself. Here I'm cautiously optimistic, although a lot is going
to depend on the third book. Lawrence has set up a three-sided argument
between groups that I will uncharitably describe as the libertarian
techbros, the "burn it all down" reactionaries, and the neoliberal
centrist technocrats. All three of those positions suck, and Lawrence had
better be setting the stage for Livira to find a different path. Her
unwillingness to commit to any of those sides gives me hope, but bringing
this plot to a satisfying conclusion is going to be tricky. I hope I like
what Lawrence comes up with, but it feels far from certain.
It doesn't help that he's started delivering some points with a
sledgehammer, and that's where we get to the unnecessary cannibalism.
Thankfully this is a fairly small part of the tail end of the book, but it
was an unpleasant surprise that I did not want in this novel and that I
don't think made the story any better.
It's tempting to call the cannibalism gratuitous, but it does fit one of
the main themes of this story, namely that humans are depressingly good at
using any rule-based object in unexpected and nasty ways that are contrary
to the best intentions of the designer. This is the fundamental challenge
of the Library as a whole and the question that I suspect the third book
will be devoted to addressing, so I understand why Lawrence wanted to
emphasize his point. The reason why there is cannibalism here is directly
related to a profound misunderstanding of the properties of the library,
and I detected an echo of one of C.S. Lewis's arguments in
The Last Battle about the nature of
Hell.
The problem, though, is that this is Satanic baby-killerism, to borrow a
term from
Fred Clark. There are numerous ways to show this type of perversion of
well-intended systems, which I know because Lawrence used other ones in
the first book that were more subtle but equally effective. One of the
best parts of The Book That Wouldn't Burn is that there were few
real villains. The conflict was structural, all sides had valid
perspectives, and the ethical points of that story were made with some
care and nuance.
The problem with cannibalism as it's used here is not merely that it's
gross and disgusting and off-putting to the reader, although it is all of
those things. If I wanted to read horror, I would read horror novels. I
don't appreciate surprise horror used for shock value in regular fantasy.
But worse, it's an abandonment of moral nuance. The function of
cannibalism in this story is like the function of Satanic baby-killers:
it's to signal that these people are wholly and irredeemably evil. They
are the Villains, they are Wrong, and they cease to be characters and
become symbols of what the protagonists are fighting. This is destructive
to the story because it's designed to provoke a visceral short-circuit in
the reader and let the author get away with sloppy story-telling. If the
author needs to use tactics like this to point out who is the villain,
they have failed to set up their moral quandary properly.
The worst part is that this was entirely unnecessary because Lawrence's
story-telling wasn't sloppy and he set up his moral quandary just fine.
No one was confused about the ethical point here. I as the reader was
following without difficulty, and had appreciated the subtlety with which
Lawrence posed the question. But apparently he thought he was too subtle
and decided to come back to the point with a pile-driver. I think that
seriously injured the story. The ethical argument here is much more
engaging and thought-provoking when it's more finely balanced.
That's a lot of complaints, mostly because this is a good book that I
badly wanted to be a great book but which kept tripping over its own feet.
A lot of trilogies have weak second books. Hopefully this is another
example of the mid-story sag, and the finale will be worthy of the start
of the story. But I have to admit the moral short-circuiting and the
de-emphasis of the actual books in the library has me a bit nervous. I
want a lot out of the third book, and I hope I'm not asking this author
for too much.
If you liked the first book, I think you'll like this one too, with the
caveat that it's quite a bit darker and more violent in places, even apart
from the surprise cannibalism. But if you've not started this series, you
may want to wait for the third book to see if Lawrence can pull off the
ending.
Followed by The Book That Held Her Heart, currently scheduled for
publication in April of 2025.
Rating: 7 out of 10
Review: The Wings Upon Her Back, by Samantha Mills
Publisher:
Tachyon
Copyright:
2024
ISBN:
1-61696-415-4
Format:
Kindle
Pages:
394
The Wings Upon Her Back is a political steampunk science fantasy
novel. If the author's name sounds familiar, it may be because Samantha
Mills's short story "Rabbit Test" won Nebula, Locus, Hugo, and Sturgeon
awards. This is her first novel.
Winged Zemolai is a soldier of the mecha god and the protege of Mecha
Vodaya, the Voice. She has served the city-state of Radezhda by defending
it against all enemies, foreign and domestic, for twenty-six years.
Despite that, it takes only a moment of errant mercy for her entire life
to come crashing down. On a whim, she spares a kitchen worker who was
concealing a statue of the scholar god, meaning that he was only
pretending to worship the worker god like all workers should. Vodaya is
unforgiving and uncompromising, as is the sleeping mecha god. Zemolai's
wings are ripped from her back and crushed in the hand of the god, and
she's left on the ground to die of mechalin withdrawal.
The Wings Upon Her Back is told in two alternating timelines. The
main one follows Zemolai after her exile as she is rescued by a young
group of revolutionaries who think she may be useful in their plans. The
other thread starts with Zemolai's childhood and shows the reader how she
became Winged Zemolai: her scholar family, her obsession with flying, her
true devotion to the mecha god, and the critical early years when she
became Vodaya's protege. Mills maintains the separate timelines through
the book and wraps them up in a rather neat piece of symbolic parallelism
in the epilogue.
I picked up this book on a recommendation from C.L. Clark, and yes, indeed, I can see why she liked this book. It's a
story about a political awakening, in which Zemolai slowly realizes that
she has been manipulated and lied to and that she may, in fact, be one of
the baddies. The Wings Upon Her Back is more personal than some
other books with that theme, since Zemolai was specifically (and
abusively) groomed for her role by Vodaya. Much of the book is Zemolai
trying to pull out the hooks that Vodaya put in her or, in the flashback
timeline, the reader watching Vodaya install those hooks.
The flashback timeline is difficult reading. I don't think Mills could
have left it out, but she says in the afterword that it was the hardest
part of the book to write and it was also the hardest part of the book to
read. It fills in some interesting bits of world-building and backstory,
and Mills does a great job pacing the story revelations so that both
threads contribute equally, but mostly it's a story of manipulative abuse.
We know from the main storyline that Vodaya's tactics work, which gives
those scenes the feel of a slow-motion train wreck. You know what's going
to happen, you know it will be bad, and yet you can't look away.
It occurred to me while reading this that Emily Tesh's
Some Desperate Glory told a similar type
of story without the flashback structure, which eliminates the stifling
feeling of inevitability. I don't think that would not have worked for
this story. If you simply rearranged the chapters of The Wings Upon
Her Back into a linear narrative, I would have bailed on the book.
Watching Zemolai being manipulated would have been too depressing and
awful for me to make it to the payoff without the forward-looking hope of
the main timeline. It gave me new appreciation for the difficulty of what
Tesh pulled off.
Mills uses this interwoven structure well, though. At about 90% through
this book I had no idea how it could end in the space remaining, but it
reaches a surprising and satisfying conclusion. Mills uses a type of
ending that normally bothers me, but she does it by handling the
psychological impact so well that I couldn't help but admire it. I'm
avoiding specifics because I think it worked better when I wasn't
expecting it, but it ties beautifully into the thematic point of the book.
I do have one structural objection, though. It's one of those problems I
didn't notice while reading, but that started bothering me when I thought
back through the story from a political lens. The Wings Upon Her
Back is Zemolai's story, her redemption arc, and that means she drives
the plot. The band of revolutionaries are great characters (particularly
Galiana), but they're supporting characters. Zemolai is older, more
experienced, and knows critical information they don't have, and she uses
it to effectively take over. As setup for her character arc, I see why
Mills did this. As political praxis, I have issues.
There is a tendency in politics to believe that political skill is
portable and repurposable. Converting opposing operatives to the cause is
welcomed not only because they indicate added support, but also because
they can use their political skill to help you win instead. To an extent
this is not wrong, and is probably the most true of combat skills (which
Zemolai has in abundance). But there's an underlying assumption that
politics is symmetric, and a critical reason why I hold many of the
political positions that I do hold is that I don't think politics is
symmetric.
If someone has been successfully stoking resentment and xenophobia in
support of authoritarians, converts to an anti-authoritarian cause, and
then produces propaganda stoking resentment and xenophobia against
authoritarians, this is in some sense an improvement. But if one believes
that resentment and xenophobia are inherently wrong, if one's politics are
aimed at reducing the resentment and xenophobia in the world, then in a
way this person has not truly converted. Worse, because this is an
effective manipulation tactic, there is a strong tendency to put this type
of political convert into a leadership position, where they will,
intentionally or not, start turning the anti-authoritarian movement into a
copy of the authoritarian movement they left. They haven't actually
changed their politics because they haven't understood (or simply don't
believe in) the fundamental asymmetry in the positions. It's the same
criticism that I have of realpolitik: the ends do not justify the
means because the means corrupt the ends.
Nothing that happens in this book is as egregious as my example, but the
more I thought about the plot structure, the more it bothered me that
Zemolai never listens to the revolutionaries she joins long enough to
wrestle with why she became an agent of an authoritarian state and they
didn't. They got something fundamentally right that she got wrong, and
perhaps that should have been reflected in who got to make future
decisions. Zemolai made very poor choices and yet continues to be the
sole main character of the story, the one whose decisions and actions
truly matter. Maybe being wrong about everything should be disqualifying
for being the main character, at least for a while, even if you think
you've understood why you were wrong.
That problem aside, I enjoyed this. Both timelines were compelling and
quite difficult to put down, even when they got rather dark. I could have
done with less body horror and a few fewer fight scenes, but I'm glad I
read it.
Science fiction readers should be warned that the world-building, despite
having an intricate and fascinating surface, is mostly vibes. I started
the book wondering how people with giant metal wings on their back can
literally fly, and thought the mentions of neural ports, high-tech
materials, and immune-suppressing drugs might mean that we'd get some sort
of explanation. We do not: heavier-than-air flight works because it looks
really cool and serves some thematic purposes. There are enough hints of
technology indistinguishable from magic that you could make up your own
explanations if you wanted to, but that's not something this book is
interested in. There's not a thing wrong with that, but don't get caught
by surprise if you were in the mood for a neat scientific explanation of
apparent magic.
Recommended if you like somewhat-harrowing character development with a
heavy political lens and steampunk vibes, although it's not the sort of
book that I'd press into the hands of everyone I know. The Wings
Upon Her Back is a complete story in a single novel.
Content warning: the main character is a victim of physical and emotional
abuse, so some of that is a lot. Also surgical gore, some torture, and
genocide.
Rating: 7 out of 10
Tobias Frost
did 2.5h (out of 7.75h assigned and 4.25h from previous period), thus carrying over 9.5h to the next month.
Evolution of the situation
In August, we have released 1 DLAs.
During the month of August Debian 11 "bullseye" officially transitioned to the responsibility of the LTS team (on 2024-08-15). However, because the final point release (11.11) was not made until 2024-08-31, LTS contributors were prevented from uploading packages to bullseye until after the point release had been made. That said, the team was not at all idle, and was busy at work on a variety of tasks which impacted both LTS and the broader Debian community, as well as preparing uploads which will be released during the month of September.
Of particular note, LTS contributor Bastien Roucari s prepared updates of the putty and cacti packages for bookworm (12) and bullseye (12), which were accepted by the old-stable release managers for the August point releases. He also analysed several security regressions in the apache2 package. LTS contributor Emilio Pozuelo Monfort worked on the Rust toolchain in bookworm and bullseye, which will be needed to support the upcoming Firefox ESR and Thunderbird ESR releases from the Mozilla project. Additionally, LTS contributor Thorsten Alteholz prepared bookworm and bullseye updates of the cups package (12), which were accepted by the old-stable release managers for the August point releases.
LTS contributor Markus Koschany collaborated with Emmanuel Bourg, co-maintainer of the tomcat packages in Debian. Regressions in a proposed security fix necessitated the updating of the tomcat10 package in Debian to the latest upstream release.
LTS contributors Bastien and Santiago Ruano Rinc n collaborated with the upstream developers and the Debian maintainer (Bernhard Schmidt) of the FreeRADIUS project towards addressing the BlastRADIUS vulnerability in the bookworm and bullseye versions of the freeradius package. If you use FreeRADIUS in Debian bookworm or bullseye, we encourage you to test the packages following the instructions found in the call for testers to help identifying any possible regression that could be introduced with these updates.
Testing is an important part of the work the LTS Team does, and in that vein LTS contributor Sean Whitton worked on improving the documentation and tooling around creating test filesystems which can be used for testing a variety of package update scenarios.
Thanks to our sponsors
Sponsors that joined recently are in bold.
Debian Python 3 patch review, by Stefano Rivera
Last month, at DebConf, Stefano reviewed the current patch set of Debian s cPython packages with Matthias Klose, the primary maintainer until now. As a result of that review, Stefano re-reviewed the patchset, updating descriptions, etc. A few patches were able to be dropped, and a few others were forwarded upstream.
One finds all sorts of skeletons doing reviews like this. One of the patches had been inactive (fortunately, because it was buggy) since the day it was applied, 13 years ago. One is a cleanup that probably only fixes a bug on HPUX, and is a result of copying code from xfree86 into Python 25 years ago. It was fixed in xfree86 a year later. Others support just Debian-specific functionality and probably never seemed worth forwarding. Or good cleanup that only really applies to Debian.
A trivial new patch would allow Debian to multiarch co-install Python stable ABI dynamic extensions (like we can with regular dynamic extensions). Performance concerns are stalling it in review, at the moment.
DebConf 24 Organization, by Stefano Rivera
Stefano helped organize DebConf 24, which concluded in early August. The event is run by a large entirely volunteer team. The work involved in making this happen is far too varied to describe here. While Freexian provides funding for 20% of collaborator time to spend on Debian-related work, it only covers a small fraction of contributions to time-intensive tasks like this.
Since the end of the event, Stefano has been doing some work on the conference finances, and initiated the reimbursement process for travel bursaries.
Archive rebuilds on Debusine, by Stefano Rivera
The recent setuptools 73 upload to Debian unstable removed the test subcommand, breaking many packages that were using python3 setup.py test in their Debian packaging. Stefano did a partial archive-rebuild using debusine.debian.net to find the regressions and file bugs.
Debusine will be a powerful tool to do QA work like this for Debian in the future, but it doesn t have all the features needed to coordinate rebuild-testing, yet. They are planned to be fleshed out in the next year. In the meantime, Debusine has the building blocks to work through a queue of package building tasks and store the results, it just needs to be driven from outside the system.
So, Stefano started working on a set of tools using the Debusine client API to perform archive rebuilds, found and tagged existing bugs, and filed many more.
OpenSSL vs. cryptography, by Colin Watson
Colin facilitated a discussion between Debian s OpenSSL team and the upstream maintainers of Python cryptography about a new incompatibility between Debian s OpenSSL packaging and cryptography s handling of OpenSSL s legacy provider, which was causing a number of build and test failures. While the issue remains open, the Debian OpenSSL maintainers have effectively reverted the change now, so it s no longer a pressing problem.
/usr-move, by Helmut Grohne
There are less than 40 source packages left to move files to /usr, so what we re left with is the long tail of the transition. Rather than fix all of them, Helmut started a discussion on removing packages from unstable and filed a first batch. As libvirt is being restructured in experimental, we re handling the fallout in collaboration with its maintainer Andrea Bolognani. Since base-files validates the aliasing symlinks before upgrading, it was discovered that systemd has its own ideas with no solution as of yet. Helmut also proposed that dashchecks for ineffective diversions of /bin/sh and that lintianwarns about aliased files.
rebootstrap by Helmut Grohne
Bootstrapping Debian for a new or existing CPU architecture still is a quite manual process. The rebootstrap project attempts to automate part of the early stage, but it still is very sensitive to changes in unstable. We had a number of fairly intrusive changes this year already. August included a little more fallout from the earlier gcc-for-host work where the C++ include search path would end up being wrong in the generated cross toolchain. A number of packages such as util-linux (twice), libxml2, libcap-ng or systemd had their stage profiles broken. e2fsprogs gained a cycle with libarchive-dev due to having gained support for creating an ext4 filesystem from a tar archive. The restructuring of glib2.0 remains an unsolved problem for now, but libxt and cdebconf should be buildable without glib2.0.
Salsa CI, by Santiago Ruano Rinc n
Santiago completed the initial RISC-V support (!523) in the Salsa CI s pipeline. The main work started in July, but it was required to take into account some comments in the review (thanks to Ahmed!) and some final details in [!534]. riscv64 is the most recently supported port in Debian, which will be part of trixie. As its name suggests, the new build-riscv64 job makes it possible to test that a package successfully builds in the riscv64 architecture. The RISC-V runner (salsaci riscv64 runner 01) runs in a couple of machines generously provided by lab.rvperf.org. Debian Developers interested in running this job in their projects should enable the runner (salsaci riscv64 runner 01) in Settings / CI / Runners, and follow the instructions available at https://salsa.debian.org/salsa-ci-team/pipeline/#build-job-on-risc-v.
Santiago also took part in discussions about how to optimize the build jobs and reviewed !537 to make the build-source job to only satisfy the Build-Depends and Build-Conflicts fields by Andrea Pappacoda. Thanks a lot to him!
Miscellaneous contributions
Stefano submitted patches for BeautifulSoup to support the latest soupsieve and lxml.
Stefano uploaded pypy3 7.3.17, upgrading the cPython compatibility from 3.9 to 3.10. Then ran into a GCC-14-related regression, which had to be ignored for now as it s proving hard to fix.
Colin released libpipeline 1.5.8 and man-db 2.13.0; the latter included foundations allowing adding an autopkgtest for man-db.
Colin tracked down test failures in python-asyncssh and Ruby resulting from certain odd /etc/hosts configurations.
Carles upgraded the packages python-ring-doorbell and simplemonitor to new upstream versions.
Carles started discussions and implementation of a tool (still in early days) named po-debconf-manager : a way for translators and reviewers to collaborate using git as a backend instead of mailing list; and submit the translations using salsa MR. More information next month.
Carles (dog-fooding po-debconf-manager ) reviewed debconf templates translated by a collaborator.
Carles reviewed and submitted the translation of apt .
Helmut sent 19 patches for improving cross building.
Helmut implemented the cross-exe-wrapper proposed by Simon McVittie for use with glib2.0.
Helmut detailed what it takes to make Perl s ExtUtils::PkgConfig suitable for cross building.
Helmut made the deletion of the root password work in debvm in all situations and implemented a test case using expect.
Anupa attended Debian Publicity team meeting and is moderating and posting on Debian Administrators LinkedIn group.
Thorsten uploaded package gutenprint to fix a FTBFS with gcc14 and package ipp-usb to fix a /usr-merge issue.
Santiago updated bzip2 to fix a long-standing bug that requested to include a pkg-config file. An important impact of this change is that it makes it possible to use Rust bindings for libbz2 by Sequoia, an implementation of OpenPGP.
a: b
touch $@b:
touch $@# A common chain of build steps%-GENERATED.c: %-generate
touch $@%.o: %.c
touch $@%.so: %-GENERATED.o
touch $@xxx-GENERATED.o: CFLAGS += adsf
# Imitates .d files created with "gcc -MMD". Does not exist on the initial build
ifneq ($(wildcard xxx.so),)
xxx-GENERATED.o: xxx-GENERATED.c
endif
This is all very simple build-system stuff. Let's see how it works:
$ rm -rf a b xxx-GENERATED.c xxx-GENERATED.o xxx.so
[start from a clean slate]
$ touch xxx-generate xxx.h
[Files that would be available in a project exist; xxx-generate is some tool]
[that would generate xxx-GENERATED.c ]
$ touch a
["a" exists but the file "b" it depends on does not]
$ make a xxx.so
touch b
touch a
touch xxx-GENERATED.c
touch xxx-GENERATED.o
touch xxx.so
rm xxx-GENERATED.c
[It built everything, but then deleted xxx-GENERATED.c]
$ make a xxx.so
remake: 'a' is up to date.
touch xxx-GENERATED.c
touch xxx-GENERATED.o
touch xxx.so
[It knew to not rebuild "a", but the missing xxx-GENERATED.c caused it to]
[re-build stuff ]
Well that's not good. What if we add .SECONDARY: to the end of the Makefile
to mark everything as a secondary file?
$ rm -rf a b xxx-GENERATED.c xxx-GENERATED.o xxx.so
$ touch xxx-generate xxx.h
$ touch a
$ make a xxx.so
remake: 'a' is up to date.
touch xxx-GENERATED.c
touch xxx-GENERATED.o
touch xxx.so
[It didn't bother rebuilding "a" even though its prerequisites "b" doesn't]
[exist. But it didn't delete the xxx-GENERATED.c at least ]
$ make a xxx.so
remake: 'a' is up to date.
remake: 'xxx.so' is up to date.
[It knew to not rebuild anything. Great.]
So it doesn't work right with or without .SECONDARY:, but it's much closer
with it. The solution is to mark everything as not an intermediate file.
mrbuild cannot do this without a bleeding-edge version of GNU Make, but users
of mrbuild can do this by explicitly mentioning specific files in rules. This
would suffice:
How does Linux move from an awake machine to a hibernating one?
How does it then manage to restore all state?
These questions led me to read way too much C in trying to figure out
how this particular hardware/software boundary is navigated.
This investigation will be split into a few parts, with the first one going
from invocation of hibernation to synchronizing all filesystems to disk.
This article has been written using Linux version 6.9.9,
the source of which can be found in many places, but can be navigated
easily through the Bootlin Elixir Cross-Referencer:
https://elixir.bootlin.com/linux/v6.9.9/source
Each code snippet will begin with a link to the above giving
the file path and the line number of the beginning of the snippet.
A Starting Point for Investigation: /sys/power/state and /sys/power/disk
These two system files exist to allow debugging of hibernation,
and thus control the exact state used directly.
Writing specific values to the state file controls the exact sleep mode used
and disk controls the specific hibernation mode1.
This is extremely handy as an entry point to understand how these systems work,
since we can just follow what happens when they are written to.
Show and Store Functions
These two files are defined using the power_attr macro:
kernel/power/power.h:80
show is called on reads and store on writes.
state_show is a little boring for our purposes, as it just prints all the
available sleep states.
kernel/power/main.c:657
/*
* state - control system sleep states.
*
* show() returns available sleep state labels, which may be "mem", "standby",
* "freeze" and "disk" (hibernation).
* See Documentation/admin-guide/pm/sleep-states.rst for a description of
* what they mean.
*
* store() accepts one of those strings, translates it into the proper
* enumerated value, and initiates a suspend transition.
*/
static ssize_t state_show(struct kobject *kobj, struct kobj_attribute *attr,
char *buf)
char *s = buf;
#ifdef CONFIG_SUSPEND
suspend_state_t i;
for (i = PM_SUSPEND_MIN; i < PM_SUSPEND_MAX; i++)
if (pm_states[i])
s += sprintf(s,"%s ", pm_states[i]);
#endif
if (hibernation_available())
s += sprintf(s, "disk ");
if (s != buf)
/* convert the last space to a newline */
*(s-1) = '\n';
return (s - buf);
state_store, however, provides our entry point.
If the string disk is written to the state file, it calls hibernate().
This is our entry point.
kernel/power/main.c:715
static ssize_t state_store(struct kobject *kobj, struct kobj_attribute *attr,
const char *buf, size_t n)
suspend_state_t state;
int error;
error = pm_autosleep_lock();
if (error)
return error;
if (pm_autosleep_state() > PM_SUSPEND_ON)
error = -EBUSY;
goto out;
state = decode_state(buf, n);
if (state < PM_SUSPEND_MAX)
if (state == PM_SUSPEND_MEM)
state = mem_sleep_current;
error = pm_suspend(state);
else if (state == PM_SUSPEND_MAX)
error = hibernate();
else
error = -EINVAL;
out:
pm_autosleep_unlock();
return error ? error : n;
static suspend_state_t decode_state(const char *buf, size_t n)
#ifdef CONFIG_SUSPEND
suspend_state_t state;
#endif
char *p;
int len;
p = memchr(buf, '\n', n);
len = p ? p - buf : n;
/* Check hibernation first. */
if (len == 4 && str_has_prefix(buf, "disk"))
return PM_SUSPEND_MAX;
#ifdef CONFIG_SUSPEND
for (state = PM_SUSPEND_MIN; state < PM_SUSPEND_MAX; state++)
const char *label = pm_states[state];
if (label && len == strlen(label) && !strncmp(buf, label, len))
return state;
#endif
return PM_SUSPEND_ON;
Could we have figured this out just via function names?
Sure, but this way we know for sure that nothing else is happening before this
function is called.
Autosleep
Our first detour is into the autosleep system.
When checking the state above, you may notice that
the kernel grabs the pm_autosleep_lock before checking the current
state.
autosleep is a mechanism originally from Android
that sends the entire system to either suspend or hibernate whenever it is
not actively working on anything.
This is not enabled for most desktop configurations, since it s primarily
for mobile systems and inverts the standard suspend and hibernate interactions.
This system is implemented as a workqueue2
that checks the current number of wakeup events, processes and drivers that
need to run3, and if there aren t any, then the system is put
into the autosleep state, typically suspend. However, it could be hibernate if
configured that way via /sys/power/autosleep in a similar manner to
using /sys/power/state to manually enable hibernation.
kernel/power/main.c:841
static DEFINE_MUTEX(autosleep_lock);
static struct wakeup_source *autosleep_ws;
static void try_to_suspend(struct work_struct *work)
unsigned int initial_count, final_count;
if (!pm_get_wakeup_count(&initial_count, true))
goto out;
mutex_lock(&autosleep_lock);
if (!pm_save_wakeup_count(initial_count)
system_state != SYSTEM_RUNNING)
mutex_unlock(&autosleep_lock);
goto out;
if (autosleep_state == PM_SUSPEND_ON)
mutex_unlock(&autosleep_lock);
return;
if (autosleep_state >= PM_SUSPEND_MAX)
hibernate();
else
pm_suspend(autosleep_state);
mutex_unlock(&autosleep_lock);
if (!pm_get_wakeup_count(&final_count, false))
goto out;
/*
* If the wakeup occurred for an unknown reason, wait to prevent the
* system from trying to suspend and waking up in a tight loop.
*/
if (final_count == initial_count)
schedule_timeout_uninterruptible(HZ / 2);
out:
queue_up_suspend_work();
static DECLARE_WORK(suspend_work, try_to_suspend);
void queue_up_suspend_work(void)
if (autosleep_state > PM_SUSPEND_ON)
queue_work(autosleep_wq, &suspend_work);
The Steps of Hibernation
Hibernation Kernel Config
It s important to note that most of the hibernate-specific functions below
do nothing unless you ve defined CONFIG_HIBERNATION in your Kconfig4.
As an example, hibernate itself is defined as the following if
CONFIG_HIBERNATE is not set.
include/linux/suspend.h:407
static inline int hibernate(void) return -ENOSYS;
Check if Hibernation is Available
We begin by confirming that we actually can perform hibernation,
via the hibernation_available function.
kernel/power/hibernate.c:742
if (!hibernation_available())
pm_pr_dbg("Hibernation not available.\n");
return -EPERM;
nohibernate is controlled by the kernel command line, it s set via
either nohibernate or hibernate=no.
security_locked_down is a hook for Linux Security Modules to prevent
hibernation. This is used to prevent hibernating to an unencrypted storage
device, as specified in the manual page
kernel_lockdown(7).
Interestingly, either level of lockdown, integrity or confidentiality,
locks down hibernation because with the ability to hibernate you can extract
bascially anything from memory and even reboot into a modified kernel image.
secretmem_active checks whether there is any active use of
memfd_secret, and if so it prevents hibernation.
memfd_secret returns a file descriptor that can be mapped into a process
but is specifically unmapped from the kernel s memory space.
Hibernating with memory that not even the kernel is supposed to
access would expose that memory to whoever could access the hibernation image.
This particular feature of secret memory was apparently
controversial, though not as
controversial as performance concerns around fragmentation when unmapping
kernel memory
(which did not end up being a real problem).
cxl_mem_active just checks whether any CXL memory is active.
A full explanation is provided in the
commit introducing this check
but there s also a shortened explanation from cxl_mem_probe that
sets the relevant flag when initializing a CXL memory device.
drivers/cxl/mem.c:186
* The kernel may be operating out of CXL memory on this device,
* there is no spec defined way to determine whether this device
* preserves contents over suspend, and there is no simple way
* to arrange for the suspend image to avoid CXL memory which
* would setup a circular dependency between PCI resume and save
* state restoration.
Check Compression
The next check is for whether compression support is enabled, and if so
whether the requested algorithm is enabled.
kernel/power/hibernate.c:747
/*
* Query for the compression algorithm support if compression is enabled.
*/
if (!nocompress)
strscpy(hib_comp_algo, hibernate_compressor, sizeof(hib_comp_algo));
if (crypto_has_comp(hib_comp_algo, 0, 0) != 1)
pr_err("%s compression is not available\n", hib_comp_algo);
return -EOPNOTSUPP;
The nocompress flag is set via the hibernate command line parameter,
setting hibernate=nocompress.
If compression is enabled, then hibernate_compressor is copied to
hib_comp_algo. This synchronizes the current requested compression
setting (hibernate_compressor) with the current compression setting
(hib_comp_algo).
Both values are character arrays of size CRYPTO_MAX_ALG_NAME
(128 in this kernel).
kernel/power/hibernate.c:50
static char hibernate_compressor[CRYPTO_MAX_ALG_NAME] = CONFIG_HIBERNATION_DEF_COMP;
/*
* Compression/decompression algorithm to be used while saving/loading
* image to/from disk. This would later be used in 'kernel/power/swap.c'
* to allocate comp streams.
*/
char hib_comp_algo[CRYPTO_MAX_ALG_NAME];
hibernate_compressor defaults to lzo if that algorithm is enabled, otherwise to lz4 if
enabled5. It can be overwritten using the hibernate.compressor setting to
either lzo or lz4.
kernel/power/Kconfig:95
choice
prompt "Default compressor"
default HIBERNATION_COMP_LZO
depends on HIBERNATION
config HIBERNATION_COMP_LZO
bool "lzo"
depends on CRYPTO_LZO
config HIBERNATION_COMP_LZ4
bool "lz4"
depends on CRYPTO_LZ4
endchoice
config HIBERNATION_DEF_COMP
string
default "lzo" if HIBERNATION_COMP_LZO
default "lz4" if HIBERNATION_COMP_LZ4
help
Default compressor to be used for hibernation.
static const char * const comp_alg_enabled[] =
#if IS_ENABLED(CONFIG_CRYPTO_LZO)
COMPRESSION_ALGO_LZO,
#endif
#if IS_ENABLED(CONFIG_CRYPTO_LZ4)
COMPRESSION_ALGO_LZ4,
#endif
;
static int hibernate_compressor_param_set(const char *compressor,
const struct kernel_param *kp)
unsigned int sleep_flags;
int index, ret;
sleep_flags = lock_system_sleep();
index = sysfs_match_string(comp_alg_enabled, compressor);
if (index >= 0)
ret = param_set_copystring(comp_alg_enabled[index], kp);
if (!ret)
strscpy(hib_comp_algo, comp_alg_enabled[index],
sizeof(hib_comp_algo));
else
ret = index;
unlock_system_sleep(sleep_flags);
if (ret)
pr_debug("Cannot set specified compressor %s\n",
compressor);
return ret;
static const struct kernel_param_ops hibernate_compressor_param_ops =
.set = hibernate_compressor_param_set,
.get = param_get_string,
;
static struct kparam_string hibernate_compressor_param_string =
.maxlen = sizeof(hibernate_compressor),
.string = hibernate_compressor,
;
We then check whether the requested algorithm is supported via crypto_has_comp.
If not, we bail out of the whole operation with EOPNOTSUPP.
As part of crypto_has_comp we perform any needed initialization of the
algorithm, loading kernel modules and running initialization code as needed6.
Grab Locks
The next step is to grab the sleep and hibernation locks via
lock_system_sleep and hibernate_acquire.
kernel/power/hibernate.c:758
sleep_flags = lock_system_sleep();
/* The snapshot device should not be opened while we're running */
if (!hibernate_acquire())
error = -EBUSY;
goto Unlock;
First, lock_system_sleep marks the current thread as not freezable, which
will be important later7. It then grabs the system_transistion_mutex,
which locks taking snapshots or modifying how they are taken,
resuming from a hibernation image, entering any suspend state, or rebooting.
The GFP Mask
The kernel also issues a warning if the gfp mask is changed via either
pm_restore_gfp_mask or pm_restrict_gfp_mask
without holding the system_transistion_mutex.
GFP flags tell the kernel how it is permitted to handle a request for memory.
include/linux/gfp_types.h:12
* GFP flags are commonly used throughout Linux to indicate how memory
* should be allocated. The GFP acronym stands for get_free_pages(),
* the underlying memory allocation function. Not every GFP flag is
* supported by every function which may allocate memory.
In the case of hibernation specifically we care about the IO and FS flags,
which are reclaim operators, ways the system is permitted to attempt to free
up memory in order to satisfy a specific request for memory.
include/linux/gfp_types.h:176
* Reclaim modifiers
* -----------------
* Please note that all the following flags are only applicable to sleepable
* allocations (e.g. %GFP_NOWAIT and %GFP_ATOMIC will ignore them).
*
* %__GFP_IO can start physical IO.
*
* %__GFP_FS can call down to the low-level FS. Clearing the flag avoids the
* allocator recursing into the filesystem which might already be holding
* locks.
gfp_allowed_mask sets which flags are permitted to be set at the current time.
As the comment below outlines, preventing these flags from being set
avoids situations where the kernel needs to do I/O to allocate memory
(e.g. read/writing swap8) but the
devices it needs to read/write to/from are not currently available.
kernel/power/main.c:24
/*
* The following functions are used by the suspend/hibernate code to temporarily
* change gfp_allowed_mask in order to avoid using I/O during memory allocations
* while devices are suspended. To avoid races with the suspend/hibernate code,
* they should always be called with system_transition_mutex held
* (gfp_allowed_mask also should only be modified with system_transition_mutex
* held, unless the suspend/hibernate code is guaranteed not to run in parallel
* with that modification).
*/
static gfp_t saved_gfp_mask;
void pm_restore_gfp_mask(void)
WARN_ON(!mutex_is_locked(&system_transition_mutex));
if (saved_gfp_mask)
gfp_allowed_mask = saved_gfp_mask;
saved_gfp_mask = 0;
void pm_restrict_gfp_mask(void)
WARN_ON(!mutex_is_locked(&system_transition_mutex));
WARN_ON(saved_gfp_mask);
saved_gfp_mask = gfp_allowed_mask;
gfp_allowed_mask &= ~(__GFP_IO __GFP_FS);
Sleep Flags
After grabbing the system_transition_mutex the kernel then returns and
captures the previous state of the threads flags in sleep_flags.
This is used later to remove PF_NOFREEZE if it wasn t previously set on the
current thread.
kernel/power/main.c:52
unsigned int lock_system_sleep(void)
unsigned int flags = current->flags;
current->flags = PF_NOFREEZE;
mutex_lock(&system_transition_mutex);
return flags;
EXPORT_SYMBOL_GPL(lock_system_sleep);
#define PF_NOFREEZE 0x00008000 /* This thread should not be frozen */
Then we grab the hibernate-specific semaphore to ensure no one can open a
snapshot or resume from it while we perform hibernation.
Additionally this lock is used to prevent hibernate_quiet_exec,
which is used by the nvdimm driver to active its firmware with all
processes and devices frozen, ensuring it is the only thing running at that
time9.
kernel/power/hibernate.c:82
Prepare Console
The kernel next calls pm_prepare_console.
This function only does anything if CONFIG_VT_CONSOLE_SLEEP has been set.
This prepares the virtual terminal for a suspend state, switching away to
a console used only for the suspend state if needed.
kernel/power/console.c:130
void pm_prepare_console(void)
if (!pm_vt_switch())
return;
orig_fgconsole = vt_move_to_console(SUSPEND_CONSOLE, 1);
if (orig_fgconsole < 0)
return;
orig_kmsg = vt_kmsg_redirect(SUSPEND_CONSOLE);
return;
/*
* There are three cases when a VT switch on suspend/resume are required:
* 1) no driver has indicated a requirement one way or another, so preserve
* the old behavior
* 2) console suspend is disabled, we want to see debug messages across
* suspend/resume
* 3) any registered driver indicates it needs a VT switch
*
* If none of these conditions is present, meaning we have at least one driver
* that doesn't need the switch, and none that do, we can avoid it to make
* resume look a little prettier (and suspend too, but that's usually hidden,
* e.g. when closing the lid on a laptop).
*/
static bool pm_vt_switch(void)
struct pm_vt_switch *entry;
bool ret = true;
mutex_lock(&vt_switch_mutex);
if (list_empty(&pm_vt_switch_list))
goto out;
if (!console_suspend_enabled)
goto out;
list_for_each_entry(entry, &pm_vt_switch_list, head)
if (entry->required)
goto out;
ret = false;
out:
mutex_unlock(&vt_switch_mutex);
return ret;
There is an explanation of the conditions under which a switch is performed
in the comment above the function, but we ll also walk through the steps here.
Firstly we grab the vt_switch_mutex to ensure nothing will modify the list
while we re looking at it.
We then examine the pm_vt_switch_list.
This list is used to indicate the drivers that require a switch during suspend.
They register this requirement, or the lack thereof, via
pm_vt_switch_required.
kernel/power/console.c:31
/**
* pm_vt_switch_required - indicate VT switch at suspend requirements
* @dev: device
* @required: if true, caller needs VT switch at suspend/resume time
*
* The different console drivers may or may not require VT switches across
* suspend/resume, depending on how they handle restoring video state and
* what may be running.
*
* Drivers can indicate support for switchless suspend/resume, which can
* save time and flicker, by using this routine and passing 'false' as
* the argument. If any loaded driver needs VT switching, or the
* no_console_suspend argument has been passed on the command line, VT
* switches will occur.
*/
void pm_vt_switch_required(struct device *dev, bool required)
Next, we check console_suspend_enabled. This is set to false
by the kernel parameter no_console_suspend, but defaults to true.
Finally, if there are any entries in the pm_vt_switch_list, then we
check to see if any of them require a VT switch.
Only if none of these conditions apply, then we return false.
If a VT switch is in fact required, then we move first the currently active
virtual terminal/console10 (vt_move_to_console)
and then the current location of kernel messages (vt_kmsg_redirect)
to the SUSPEND_CONSOLE.
The SUSPEND_CONSOLE is the last entry in the list of possible
consoles, and appears to just be a black hole to throw away messages.
kernel/power/console.c:16
#define SUSPEND_CONSOLE (MAX_NR_CONSOLES-1)
Interestingly, these are separate functions because you can use
TIOCL_SETKMSGREDIRECT (an ioctl11) to send kernel messages to a specific virtual terminal,
but by default its the same as the currently active console.
The locations of the previously active console and the previous kernel
messages location are stored in orig_fgconsole and orig_kmsg, to
restore the state of the console and kernel messages after the machine wakes
up again.
Interestingly, this means orig_fgconsole also ends up storing any errors,
so has to be checked to ensure it s not less than zero before we try to do
anything with the kernel messages on both suspend and resume.
drivers/tty/vt/vt_ioctl.c:1268
/* Perform a kernel triggered VT switch for suspend/resume */
static int disable_vt_switch;
int vt_move_to_console(unsigned int vt, int alloc)
int prev;
console_lock();
/* Graphics mode - up to X */
if (disable_vt_switch)
console_unlock();
return 0;
prev = fg_console;
if (alloc && vc_allocate(vt))
/* we can't have a free VC for now. Too bad,
* we don't want to mess the screen for now. */
console_unlock();
return -ENOSPC;
if (set_console(vt))
/*
* We're unable to switch to the SUSPEND_CONSOLE.
* Let the calling function know so it can decide
* what to do.
*/
console_unlock();
return -EIO;
console_unlock();
if (vt_waitactive(vt + 1))
pr_debug("Suspend: Can't switch VCs.");
return -EINTR;
return prev;
Unlike most other locking functions we ve seen so far, console_lock
needs to be careful to ensure nothing else is panicking and needs to
dump to the console before grabbing the semaphore for the console
and setting a couple flags.
Panics
Panics are tracked via an atomic integer set to the id of the processor
currently panicking.
kernel/printk/printk.c:2649
/**
* console_lock - block the console subsystem from printing
*
* Acquires a lock which guarantees that no consoles will
* be in or enter their write() callback.
*
* Can sleep, returns nothing.
*/
void console_lock(void)
might_sleep();
/* On panic, the console_lock must be left to the panic cpu. */
while (other_cpu_in_panic())
msleep(1000);
down_console_sem();
console_locked = 1;
console_may_schedule = 1;
EXPORT_SYMBOL(console_lock);
/*
* Return true if a panic is in progress on a remote CPU.
*
* On true, the local CPU should immediately release any printing resources
* that may be needed by the panic CPU.
*/
bool other_cpu_in_panic(void)
return (panic_in_progress() && !this_cpu_in_panic());
/* Return true if a panic is in progress on the current CPU. */
bool this_cpu_in_panic(void)
/*
* We can use raw_smp_processor_id() here because it is impossible for
* the task to be migrated to the panic_cpu, or away from it. If
* panic_cpu has already been set, and we're not currently executing on
* that CPU, then we never will be.
*/
return unlikely(atomic_read(&panic_cpu) == raw_smp_processor_id());
console_locked is a debug value, used to indicate that the lock should be
held, and our first indication that this whole virtual terminal system is
more complex than might initially be expected.
kernel/printk/printk.c:373
/*
* This is used for debugging the mess that is the VT code by
* keeping track if we have the console semaphore held. It's
* definitely not the perfect debug tool (we don't know if _WE_
* hold it and are racing, but it helps tracking those weird code
* paths in the console code where we end up in places I want
* locked without the console semaphore held).
*/
static int console_locked;
console_may_schedule is used to see if we are permitted to sleep
and schedule other work while we hold this lock.
As we ll see later, the virtual terminal subsystem is not re-entrant,
so there s all sorts of hacks in here to ensure we don t leave important
code sections that can t be safely resumed.
Disable VT Switch
As the comment below lays out, when another program is handling graphical
display anyway, there s no need to do any of this, so the kernel provides
a switch to turn the whole thing off.
Interestingly, this appears to only be used by three drivers,
so the specific hardware support required must not be particularly common.
/*
* Normally during a suspend, we allocate a new console and switch to it.
* When we resume, we switch back to the original console. This switch
* can be slow, so on systems where the framebuffer can handle restoration
* of video registers anyways, there's little point in doing the console
* switch. This function allows you to disable it by passing it '0'.
*/
void pm_set_vt_switch(int do_switch)
console_lock();
disable_vt_switch = !do_switch;
console_unlock();
EXPORT_SYMBOL(pm_set_vt_switch);
The rest of the vt_switch_console function is pretty normal,
however, simply allocating space if needed to create the requested
virtual terminal
and then setting the current virtual terminal via set_console.
Virtual Terminal Set Console
With set_console, we begin (as if we haven t been already) to enter the
madness that is the virtual terminal subsystem.
As mentioned previously, modifications to its state must be made very
carefully, as other stuff happening at the same time could create complete
messes.
All this to say, calling set_console does not actually perform any
work to change the state of the current console.
Instead it indicates what changes it wants and then schedules that work.
drivers/tty/vt/vt.c:3153
int set_console(int nr)
struct vc_data *vc = vc_cons[fg_console].d;
if (!vc_cons_allocated(nr) vt_dont_switch
(vc->vt_mode.mode == VT_AUTO && vc->vc_mode == KD_GRAPHICS))
/*
* Console switch will fail in console_callback() or
* change_console() so there is no point scheduling
* the callback
*
* Existing set_console() users don't check the return
* value so this shouldn't break anything
*/
return -EINVAL;
want_console = nr;
schedule_console_callback();
return 0;
The check for vc->vc_mode == KD_GRAPHICS is where most end-user graphical
desktops will bail out of this change, as they re in graphics mode and don t
need to switch away to the suspend console.
vt_dont_switch is a flag used by the ioctls11VT_LOCKSWITCH and
VT_UNLOCKSWITCH to prevent the system from switching virtual terminal
devices when the user has explicitly locked it.
VT_AUTO is a flag indicating that automatic virtual terminal switching is enabled12,
and thus deliberate switching to a suspend terminal is not required.
However, if you do run your machine from a virtual terminal, then we
indicate to the system that we want to change to the requested virtual terminal
via the want_console variable
and schedule a callback via schedule_console_callback.
drivers/tty/vt/vt.c:315
/*
* This is the console switching callback.
*
* Doing console switching in a process context allows
* us to do the switches asynchronously (needed when we want
* to switch due to a keyboard interrupt). Synchronization
* with other console code and prevention of re-entrancy is
* ensured with console_lock.
*/
static void console_callback(struct work_struct *ignored)
console_lock();
if (want_console >= 0)
if (want_console != fg_console &&
vc_cons_allocated(want_console))
hide_cursor(vc_cons[fg_console].d);
change_console(vc_cons[want_console].d);
/* we only changed when the console had already
been allocated - a new console is not created
in an interrupt routine */
want_console = -1;
...
console_callback first looks to see if there is a console change wanted
via want_console and then changes to it if it s not the current console and
has been allocated already.
We do first remove any cursor state with hide_cursor.
drivers/tty/vt/vt.c:841
This will call a chain of power management callbacks, passing first
PM_HIBERNATION_PREPARE and then PM_POST_HIBERNATION on startup or
on error with another callback.
kernel/power/main.c:98
int pm_notifier_call_chain_robust(unsigned long val_up, unsigned long val_down)
int ret;
ret = blocking_notifier_call_chain_robust(&pm_chain_head, val_up, val_down, NULL);
return notifier_to_errno(ret);
The power management notifier is a blocking notifier chain, which means it
has the following properties.
include/linux/notifier.h:23
* Blocking notifier chains: Chain callbacks run in process context.
* Callouts are allowed to block.
The callback chain is a linked list with each entry containing a priority
and a function to call. The function technically takes in a data value,
but it is always NULL for the power management chain.
include/linux/notifier.h:49
struct notifier_block;
typedef int (*notifier_fn_t)(struct notifier_block *nb,
unsigned long action, void *data);
struct notifier_block
notifier_fn_t notifier_call;
struct notifier_block __rcu *next;
int priority;
;
Because it is prioritized, appending to the list requires walking it until
an item with lower13 priority is found to insert the current item before.
kernel/notifier.c:252
/*
* Blocking notifier chain routines. All access to the chain is
* synchronized by an rwsem.
*/
static int __blocking_notifier_chain_register(struct blocking_notifier_head *nh,
struct notifier_block *n,
bool unique_priority)
int ret;
/*
* This code gets used during boot-up, when task switching is
* not yet working and interrupts must remain disabled. At
* such times we must not call down_write().
*/
if (unlikely(system_state == SYSTEM_BOOTING))
return notifier_chain_register(&nh->head, n, unique_priority);
down_write(&nh->rwsem);
ret = notifier_chain_register(&nh->head, n, unique_priority);
up_write(&nh->rwsem);
return ret;
#define NOTIFY_DONE 0x0000 /* Don't care */
#define NOTIFY_OK 0x0001 /* Suits me */
#define NOTIFY_STOP_MASK 0x8000 /* Don't call further */
#define NOTIFY_BAD (NOTIFY_STOP_MASK 0x0002)
/* Bad/Veto action */
When notifying the chain, if a function returns STOP or BAD then
the previous parts of the chain are called again with PM_POST_HIBERNATION14
and an error is returned.
kernel/notifier.c:107
/**
* notifier_call_chain_robust - Inform the registered notifiers about an event
* and rollback on error.
* @nl: Pointer to head of the blocking notifier chain
* @val_up: Value passed unmodified to the notifier function
* @val_down: Value passed unmodified to the notifier function when recovering
* from an error on @val_up
* @v: Pointer passed unmodified to the notifier function
*
* NOTE: It is important the @nl chain doesn't change between the two
* invocations of notifier_call_chain() such that we visit the
* exact same notifier callbacks; this rules out any RCU usage.
*
* Return: the return value of the @val_up call.
*/
static int notifier_call_chain_robust(struct notifier_block **nl,
unsigned long val_up, unsigned long val_down,
void *v)
int ret, nr = 0;
ret = notifier_call_chain(nl, val_up, v, -1, &nr);
if (ret & NOTIFY_STOP_MASK)
notifier_call_chain(nl, val_down, v, nr-1, NULL);
return ret;
Each of these callbacks tends to be quite driver-specific, so we ll cease
discussion of this here.
Sync Filesystems
The next step is to ensure all filesystems have been synchronized to disk.
This is performed via a simple helper function that times how long the full
synchronize operation, ksys_sync takes.
kernel/power/main.c:69
ksys_sync wakes and instructs a set of flusher threads to write out
every filesystem, first their inodes15, then the full filesystem, and
then finally all block devices, to ensure all pages are written out
to disk.
fs/sync.c:87
/*
* Sync everything. We start by waking flusher threads so that most of
* writeback runs on all devices in parallel. Then we sync all inodes reliably
* which effectively also waits for all flusher threads to finish doing
* writeback. At this point all data is on disk so metadata should be stable
* and we tell filesystems to sync their metadata via ->sync_fs() calls.
* Finally, we writeout all block devices because some filesystems (e.g. ext2)
* just write metadata (such as inodes or bitmaps) to block device page cache
* and do not sync it on their own in ->sync_fs().
*/
void ksys_sync(void)
int nowait = 0, wait = 1;
wakeup_flusher_threads(WB_REASON_SYNC);
iterate_supers(sync_inodes_one_sb, NULL);
iterate_supers(sync_fs_one_sb, &nowait);
iterate_supers(sync_fs_one_sb, &wait);
sync_bdevs(false);
sync_bdevs(true);
if (unlikely(laptop_mode))
laptop_sync_completion();
It follows an interesting pattern of using iterate_supers to run both
sync_inodes_one_sb and then sync_fs_one_sb on each known filesystem16.
It also calls both sync_fs_one_sb and sync_bdevs twice, first without waiting
for any operations to complete and then again waiting for completion17.
When laptop_mode is enabled the system runs additional filesystem synchronization
operations after the specified delay without any writes.
mm/page-writeback.c:111
/*
* Flag that puts the machine in "laptop mode". Doubles as a timeout in jiffies:
* a full sync is triggered after this time elapses without any disk activity.
*/
int laptop_mode;
EXPORT_SYMBOL(laptop_mode);
However, when running a filesystem synchronization operation, the system will
add an additional timer to schedule more writes after the laptop_mode delay.
We don t want the state of the system to change at all while performing
hibernation, so we cancel those timers.
mm/page-writeback.c:2198
/*
* We're in laptop mode and we've just synced. The sync's writes will have
* caused another writeback to be scheduled by laptop_io_completion.
* Nothing needs to be written back anymore, so we unschedule the writeback.
*/
void laptop_sync_completion(void)
struct backing_dev_info *bdi;
rcu_read_lock();
list_for_each_entry_rcu(bdi, &bdi_list, bdi_list)
del_timer(&bdi->laptop_mode_wb_timer);
rcu_read_unlock();
As a side note, the ksys_sync function is simply called when the
system call sync is used.
fs/sync.c:111
SYSCALL_DEFINE0(sync)
ksys_sync();
return 0;
The End of Preparation
With that the system has finished preparations for hibernation.
This is a somewhat arbitrary cutoff, but next the system will begin
a full freeze of userspace to then dump memory out to an image and finally
to perform hibernation. All this will be covered in future articles!
Hibernation modes are outside of scope for this article, see the previous article for a high-level description of the different types of hibernation.
Including checking whether the algorithm is larval? Which appears to indicate that it requires additional setup, but is an interesting choice of name for such a state.
Specifically when we get to process freezing, which we ll get to in the next article in this series.
Swap space is outside the scope of this article, but in short it is a buffer on disk that the kernel uses to store memory not current in use to free up space for other things. See Swap Management for more details.
The code for this is lengthy and tangential, thus it has not been included here. If you re curious about the details of this, see kernel/power/hibernate.c:858 for the details of hibernate_quiet_exec, and drivers/nvdimm/core.c:451 for how it is used in nvdimm.
Annoyingly this code appears to use the terms console and virtual terminal interchangeably.
ioctls are special device-specific I/O operations that permit performing actions outside of the standard file interactions of read/write/seek/etc. 2
I m not entirely clear on how this flag works, this subsystem is particularly complex.
In this case a higher number is higher priority.
Or whatever the caller passes as val_down, but in this case we re specifically looking at how this is used in hibernation.
An inode refers to a particular file or directory within the filesystem. See Wikipedia for more details.
Each active filesystem is registed with the kernel through a structure known as a superblock, which contains references to all the inodes contained within the filesystem, as well as function pointers to perform the various required operations, like sync.
I m including minimal code in this section, as I m not looking to deep dive into the filesystem code at this time.
Several people have been asking me to explain and/or write about my
solution for chatting nowadays. I realize that the current scenario
is much more complex than, say, 10 or 20 years ago. Back then, this
post would probably be more about the IRC client I used than about
different chatting technologies.
I have also spent a non trivial amount of time setting things up the
way I want, so I understand that it s about time to write about my
setup not only because I think it can be helpful to others, but also
because I would like to document things for myself.
The backbone: Matrix
I chose to use Matrix as the place where I integrate everything.
Despite there being some heavy (and justified) criticism on the
protocol itself, it serves me well for what I need right now.
Obviously, I don t like the fact that I have to provide Matrix and all
of its accompanying bridges a VPS with 4GB of RAM and 3 vCPUs, but I
think that that ship has sailed, unfortunately.
In an ideal world, I would be using XMPP and dedicating only a
fraction of the resources I m using today to have a full chat system.
And since I have been running my personal XMPP server for more than a
decade now, I did try to find a solution that would allow me to keep
using it, but unfortunately the protocol became almost a hobbyist
thing, so there s that.
A few disclaimers
I self-host everything, including my Matrix server. Much of what I
did won t work if you don t self-host Matrix, so keep that in mind.
This won t be a post teaching you how to deploy the services. My
intention is to describe what I use and for what purpose.
Also, as much as I try to use Debian packages for everything I do, I
opted to deploy all services using a community-maintained Ansible
playbook which is very well written and organized:
matrix-docker-ansible-deploy.
Last but not least, as I said above, you will likely need a machine
with a good amount of RAM, CPU and storage, especially if you deploy
Synapse as your Matrix homeserver (which is what I recommend if you
plan to use the bridges I ll mention). My current VPS has 4GB of RAM,
3 vCPUs and 80GB of storage (of which I m currently using
approximately 55GB).
Problem #1: my Matrix client(s)
There are a lot of clients that can talk the Matrix protocol, but most
of them are either web clients or GUI programs. I live on the
terminal, more specifically inside Emacs, so I settled for the amazing
ement.el Emacs mode. It works surprisingly well, but unfortunately
doesn t support end-to-end encryption out of the box; for that, you
have to hook it up with pantalaimon. Unfortunately, the project seems
abandoned and therefore I don t recommend you to use it. I don t use
it myself.
When I have to reply some E2E encrypted message from another user, I
go to my web browser and use my self-hosted Element client. It s a
nuisance, but one that I m willing to accept because of security
concerns.
If you re into web clients and don t want to use Element (because it
is heavy), you can try Cinny. It s lightweight and supports a decent
set of features.
If you re a terminal lover but don t use Emacs, you may want to try
gomuks or iamb.
Problem #2: IRC bridging
There are basically two types of IRC bridges for Matrix:
The regular and most used matrix-appservice-irc. This bridge takes
Matrix to IRC (think of IRC users with the [m] suffix appended to
their nicknames), and is what the matrix.org and other big
homeservers (including matrix.debian.social) use. It s a complex
service which allows thousands of Matrix users to connect to IRC
networks, but that unfortunately has complex problems and is only
worth using if you intend to host a community server.
A bouncer-like bridge called Heisenbridge. This is what I use
personally. It takes IRC to Matrix, which means that people on
IRC will not know that you re using Matrix. This bridge is much
simpler, and because it acts like a bouncer it s pretty much
impossible for it to cause problems with the IRC network.
Due to the fact that I sometimes like to use other IRC clients, I
still run a regular ZNC bouncer, and I use Heisenbridge to connect to
my ZNC. This means that I can use, e.g., ERC inside Emacs and my
Matrix bridge at the same time. But you don t necessarily need to run
another bouncer; you can simply use Heisenbridge and connect directly
to the IRC network(s) you want.
A word of caution, though: unlike ZNC, Heisenbridge doesn t support
per-user configuration when you use it in bouncer mode. This is the
reason why you need to self-host it, and why it s not possible to
offer the service to other users (they would have access to your IRC
network configuration otherwise).
It s also worth talking about logs. I find that keeping logs of
everything that goes on IRC has saved me a bunch of times, and so I
find it really important to continue doing that. Unfortunately,
neither ement.el nor Element support logging things out of the box
(at least not that I know). This is also one of the reasons why I
still keep my ZNC around: I configure it to log everything.
Problem #3: Telegram
I don t use Telegram myself, but unfortunately several people from the
Debian community do, especially in Brazil. There is a whole Debian
community on Telegram, and I wanted to be able to bridge our Debian
Matrix channels to their Telegram counterparts.
I am currently using mautrix-telegram for that, and it s working
great. You need someone with a Telegram account to configure their
credentials so that the bridge can connect to it, but afterwards it s
really easy to bridge channels together.
Problem #4: GitLab webhooks
Something else I wanted to be able to do was to receive notifications
regarding new issues, merge requests and other activities from Salsa.
For this, I m using maubot, which is awesome and has a
huge list of plugins. I m using the gitlab one.
Final thoughts
Overall, I m satisfied with the setup I have now. It has certainly
taken some time and effort to find the right tool for each problem I
needed to solve, and I still feel like there are some rough edges to
soften (like the fact that my Emacs client doesn t support E2E
encryption out of the box, or the whole logging situation), but
otherwise things are working fine and I haven t had any big problems
with the deployment. You do have to be much more careful about stuff
(for example, when I installed an unrelated service that hijacked my
Apache configuration and made Matrix s federation silently stop
working), though.
If you have more specific questions about any part of my setup, shoot
me an email and I ll do my best to help.
Happy chatting!
Welcome to the August 2024 report from the Reproducible Builds project!
Our reports attempt to outline what we ve been up to over the past month, highlighting news items from elsewhere in tech where they are related. As ever, if you are interested in contributing to the project, please visit our Contribute page on our website.
Table of contents:
Intermediate Autotools build artifacts removed from PostgreSQL distribution tarballs
Peter Eisentraut wrote a detailed blog post on the subject of The new PostgreSQL 17 make dist . Like many projects, the PostgreSQL database has previously pre-built parts of its GNU Autotools build system: the reason for this is a mix of convenience and traditional practice . Peter astutely notes that this arrangement in the build system is quite tricky as:
You need to carefully maintain the different states of clean source code , partially built source code , and fully built source code , and the commands to transition between them.
However, Peter goes on to mention that:
a lot more attention is nowadays paid to the software supply chain. There are security and legal reasons for this. When users install software, they want to know where it came from, and they want to be sure that they got the right thing, not some fake version or some version of dubious legal provenance.
And cites the XZ Utils backdoor as a reason to care about transparent and reproducible ways of distributing and communicating a source tarball and provenance. Because of this, intermediate build artifacts are now henceforth essentially disallowed from PostgreSQL distribution tarballs.
Distribution news
In Debian this month, 30 reviews of Debian packages were added, 17 were updated and 10 were removed this month adding to our knowledge about identified issues. One issue type was added by Chris Lamb, too. []
In addition, an issue was filed to update the Salsa CI pipeline (used by 1,000s of Debian packages) to no longer test for reproducibility with reprotest s build_path variation. Holger Levsen provided a rationale for this change in the issue, which has already been made to the tests being performed by tests.reproducible-builds.org.
In Arch Linux this month, Jelle van der Waa published a short blog post on the topic of Investigating creating reproducible images with mkosi, motivated by the desire to make it possible for anyone to re-recreate the official Arch cloud image bit-by-bit identical on their own machine as per [the] reproducible builds definition. In addition, Jelle filed a patch for pacman, the Arch Linux package manager, to respect the SOURCE_DATE_EPOCH environment variable when installing a package.
In openSUSE news, Bernhard M. Wiedemann published another report for that distribution.
In Android news, the IzzyOnDroid project added 49 new rebuilder recipes and now features 256 total reproducible applications representing 21% of the total offerings in the repository. IzzyOnDroid is an F-Droid style repository for Android apps[:] applications in this repository are official binaries built by the original application developers, taken from their resp. repositories (mostly GitHub).
Bernhard M. Wiedemann posted a brief message to the list with some helpful information regarding nondeterminism within Rust binaries, positing the use of the codegen-units = 16 default and resulting in a bug being filed in the Rust issue tracker. []
Bernhard also wrote to the list, following up to a thread in November 2023, on attempts to make the LibreOffice suite of office applications build reproducibly. In the thread from this month, Bernhard could announce that the four patches previously mentioned have landed in LibreOffice upstream.
Fay Stegerman linked the mailing list to a thread she made on the Signal issue tracker regarding whether device-specific binaries [can] ever be considered meaningfully reproducible . In particular: the whole part about allow[ing] multiple third parties to come to a consensus on a correct result breaks down completely when correct is device-specific and not something everyone can agree on. []
Developer kpcyrdposted an update for source code indexing project, whatsrc.org. Announcing that it now importing packages from live-bootstrap ( a usable Linux system [that is] created with only human-auditable, and wherever possible, human-written, source code ) into its database of provenance data.
Lastly, Mechtilde Stehmann posted an update to an earlier thread about how Java builds are not reproducible on the armhf architecture, enquiring how they might gain temporary access to such a machine in order to perform some deeper testing. []
diffoscopediffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb released versions 274, 275, 276 and 277, uploaded these to Debian, and made the following changes as well:
New features:
Strip ANSI escapes usually colour codes from the output of the Procyon Java decompiler. []
Factor out a method for stripping ANSI escapes. []
Append output from dumppdf(1) in more cases, avoiding situations where we fallback to a binary diff. []
Add support for versions of Perl s IO::Compress::Zip version 2.212. []
Bug fixes:
Also catch RuntimeError exceptions when importing the PyPDF library so that it, or, crucially, its transitive dependencies, cannot not cause diffoscope to traceback at runtime and build time. []
Do not call marshal.load( ) of precompiled Python bytecode as it, alas, inherently unsafe. Replace for now with a brief summary of the code section of .pyc. [][]
Don t include excessive debug output when calling dumppdf(1). []
Testsuite-related changes:
Don t bother to check version number in test_python.py: the fixture for this test is fixed. [][]
Update test_zip text fixtures and definitions to support new changes to the Perl IO::Compress library. []
In addition, Mattia Rizzolo updated the available architectures for a number of test dependencies [] and Sergei Trofimovich fixed an issue to avoid diffoscope crashing when hashing directory symlinks [] and Vagrant Cascadian proposed GNU Guix updates for diffoscope versions 275 and 276 and 277.
Website updates
There were a rather substantial number of improvements made to our website this month, including:
Alba Herrerias:
Substantially extend the guidance on the Contribute page. []
Chris Lamb:
Set the future: true configuration value so we render all files and documents in the website, regardless of whether they have a date property in the future. After all, we don t re-generate the website on a timer, and have other ways of making unpublished, draft posts. [][]
Considerably overhaul the History page in the documentation, linking strip-nondeterminism and SOURCE_DATE_EPOCH [], fixing the test statistics link [], adjusting the Google Summer of Code application link [], a link to a Debian bug [], and removed a dead link to the debhelper utility [].
Use the jekyll-sitemap plugin to create a sitemap for the website. []
Use raw HTML to avoid a literal .lead directive appearing in the page. []
Upstream patches
The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:
Reproducibility testing framework
The Reproducible Builds project operates a comprehensive testing framework running primarily at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In August, a number of changes were made by Holger Levsen, including:
Temporarily install the openssl-provider-legacy package for the Debian unstable environments for running diffoscope due to Debian bug #1078944. [][][][]
Mark Debian armhf architecture nodes as being down due to proxy down. [][]
Detect proxy failures. [][][]
Run the index-buildinfo for the builtin-pho script with the -q switch. []
In addition, Mattia Rizzolo updated the website configuration to install the ruby-jekyll-sitemap package as it is now used in the website [], Roland Clobus updated the script to build Debian live images to treat openQA issues as warnings [], and Vagrant Cascadian marked the cbxi4b node as down [].
If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:
University of Nairobi.
Our group at the conference.
View of Nairobi's skyline from the open area on the fifth floor.
A library in Mahatma Gandhi wing of the University of Nairobi.
One of the conference halls where talks took place.
One of the lunch meals served during the conference.
Minibus which took us from the university to Bao Box restaurant.
This minibus in the picture gave a sense of a real matatu.
Another short status update of what happened on my side last
month. Besides the usual amount of housekeeping last month was a lot
about getting old issues resolved by finishing some stale merge
requests and work in pogress MRs. I also pushed out the 
The Reproducible Builds project 
Vagrant: So my understanding of some of your recent work is basically
defining 
Like each month, have a look at the work funded by 
