Search Results: "andy"

5 April 2021

Kees Cook: security things in Linux v5.9

Previously: v5.8 Linux v5.9 was released in October, 2020. Here s my summary of various security things that I found interesting: seccomp user_notif file descriptor injection
Sargun Dhillon added the ability for SECCOMP_RET_USER_NOTIF filters to inject file descriptors into the target process using SECCOMP_IOCTL_NOTIF_ADDFD. This lets container managers fully emulate syscalls like open() and connect(), where an actual file descriptor is expected to be available after a successful syscall. In the process I fixed a couple bugs and refactored the file descriptor receiving code. zero-initialize stack variables with Clang
When Alexander Potapenko landed support for Clang s automatic variable initialization, it did so with a byte pattern designed to really stand out in kernel crashes. Now he s added support for doing zero initialization via CONFIG_INIT_STACK_ALL_ZERO, which besides actually being faster, has a few behavior benefits as well. Unlike pattern initialization, which has a higher chance of triggering existing bugs, zero initialization provides safe defaults for strings, pointers, indexes, and sizes. Like the pattern initialization, this feature stops entire classes of uninitialized stack variable flaws. common syscall entry/exit routines
Thomas Gleixner created architecture-independent code to do syscall entry/exit, since much of the kernel s work during a syscall entry and exit is the same. There was no need to repeat this in each architecture, and having it implemented separately meant bugs (or features) might only get fixed (or implemented) in a handful of architectures. It means that features like seccomp become much easier to build since it wouldn t need per-architecture implementations any more. Presently only x86 has switched over to the common routines. SLAB kfree() hardening
To reach CONFIG_SLAB_FREELIST_HARDENED feature-parity with the SLUB heap allocator, I added naive double-free detection and the ability to detect cross-cache freeing in the SLAB allocator. This should keep a class of type-confusion bugs from biting kernels using SLAB. (Most distro kernels use SLUB, but some smaller devices prefer the slightly more compact SLAB, so this hardening is mostly aimed at those systems.) new CAP_CHECKPOINT_RESTORE capability
Adrian Reber added the new CAP_CHECKPOINT_RESTORE capability, splitting this functionality off of CAP_SYS_ADMIN. The needs for the kernel to correctly checkpoint and restore a process (e.g. used to move processes between containers) continues to grow, and it became clear that the security implications were lower than those of CAP_SYS_ADMIN yet distinct from other capabilities. Using this capability is now the preferred method for doing things like changing /proc/self/exe. debugfs boot-time visibility restriction
Peter Enderborg added the debugfs boot parameter to control the visibility of the kernel s debug filesystem. The contents of debugfs continue to be a common area of sensitive information being exposed to attackers. While this was effectively possible by unsetting CONFIG_DEBUG_FS, that wasn t a great approach for system builders needing a single set of kernel configs (e.g. a distro kernel), so now it can be disabled at boot time. more seccomp architecture support
Michael Karcher implemented the SuperH seccomp hooks, Guo Ren implemented the C-SKY seccomp hooks, and Max Filippov implemented the xtensa seccomp hooks. Each of these included the ever-important updates to the seccomp regression testing suite in the kernel selftests. stack protector support for RISC-V
Guo Ren implemented -fstack-protector (and -fstack-protector-strong) support for RISC-V. This is the initial global-canary support while the patches to GCC to support per-task canaries is getting finished (similar to the per-task canaries done for arm64). This will mean nearly all stack frame write overflows are no longer useful to attackers on this architecture. It s nice to see this finally land for RISC-V, which is quickly approaching architecture feature parity with the other major architectures in the kernel. new tasklet API
Romain Perier and Allen Pais introduced a new tasklet API to make their use safer. Much like the timer_list refactoring work done earlier, the tasklet API is also a potential source of simple function-pointer-and-first-argument controlled exploits via linear heap overwrites. It s a smaller attack surface since it s used much less in the kernel, but it is the same weak design, making it a sensible thing to replace. While the use of the tasklet API is considered deprecated (replaced by threaded IRQs), it s not always a simple mechanical refactoring, so the old API still needs refactoring (since that CAN be done mechanically is most cases). x86 FSGSBASE implementation
Sasha Levin, Andy Lutomirski, Chang S. Bae, Andi Kleen, Tony Luck, Thomas Gleixner, and others landed the long-awaited FSGSBASE series. This provides task switching performance improvements while keeping the kernel safe from modules accidentally (or maliciously) trying to use the features directly (which exposed an unprivileged direct kernel access hole). filter x86 MSR writes
While it s been long understood that writing to CPU Model-Specific Registers (MSRs) from userspace was a bad idea, it has been left enabled for things like MSR_IA32_ENERGY_PERF_BIAS. Boris Petkov has decided enough is enough and has now enabled logging and kernel tainting (TAINT_CPU_OUT_OF_SPEC) by default and a way to disable MSR writes at runtime. (However, since this is controlled by a normal module parameter and the root user can just turn writes back on, I continue to recommend that people build with CONFIG_X86_MSR=n.) The expectation is that userspace MSR writes will be entirely removed in future kernels. uninitialized_var() macro removed
I made treewide changes to remove the uninitialized_var() macro, which had been used to silence compiler warnings. The rationale for this macro was weak to begin with ( the compiler is reporting an uninitialized variable that is clearly initialized ) since it was mainly papering over compiler bugs. However, it creates a much more fragile situation in the kernel since now such uses can actually disable automatic stack variable initialization, as well as mask legitimate unused variable warnings. The proper solution is to just initialize variables the compiler warns about. function pointer cast removals
Oscar Carter has started removing function pointer casts from the kernel, in an effort to allow the kernel to build with -Wcast-function-type. The future use of Control Flow Integrity checking (which does validation of function prototypes matching between the caller and the target) tends not to work well with function casts, so it d be nice to get rid of these before CFI lands. flexible array conversions
As part of Gustavo A. R. Silva s on-going work to replace zero-length and one-element arrays with flexible arrays, he has documented the details of the flexible array conversions, and the various helpers to be used in kernel code. Every commit gets the kernel closer to building with -Warray-bounds, which catches a lot of potential buffer overflows at compile time. That s it for now! Please let me know if you think anything else needs some attention. Next up is Linux v5.10.

2021, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
CC BY-SA 4.0

1 March 2021

Utkarsh Gupta: FOSS Activites in February 2021

Here s my (seventeenth) monthly update about the activities I ve done in the F/L/OSS world.

Debian
This was my 26th month of active contributing to Debian. I became a DM in late March 2019 and a DD on Christmas 19! \o/ This month was a nice mix of amusement, excitement, nervousness, and craziness. More on it below.
Anyway, whilst I was super-insanely busy this month, I still did some Debian stuff here and there. Here are the following things I worked on:

Uploads and bug fixes:

Other $things:
  • Attended the Debian LTS team meeting.
  • Mentoring for newcomers.
  • Moderation of -project mailing list.
  • Sponsored ruby-rspec-stubbed-env for C dric Boutillier, heh :P

Interesting Bits!
  • Last month, I wrote:
    Besides, there s something more that is in the pipelines. Can t talk about it now, shh. But hopefully very sooooooon!
    And now I can talk about it! So here it is..
    I ve joined Canonical as a SDE to work on Ubuntu, full time!!! \o/
    Fully remote + dream job/work + most of the work is in the open-source domain + the beessstttt co-workers one could ever ask for! It s been an amazing time so far and I ll talk more about it later this month.
    But for now, here s our team monitor selfie (with Rick missing because of his secret plan ! )

    We ll soon e-meet them in a more detailed manner in the next blog post, that is, later this month!
  • In another exciting news, I got 2 more CVEs assigned!!! \o/
    No, it is not something that I found, it was discovered by Tavis Ormandy. I just assigned them a CVE ID, CVE-2021-26937 for screen and CVE-2021-27135 for xterm.
    This is my 2nd and 3rd, so I am (still) very excited about this! ^_^

Debian (E)LTS
Debian Long Term Support (LTS) is a project to extend the lifetime of all Debian stable releases to (at least) 5 years. Debian LTS is not handled by the Debian security team, but by a separate group of volunteers and companies interested in making it a success. And Debian Extended LTS (ELTS) is its sister project, extending support to the Jessie release (+2 years after LTS support). This was my sixteenth month as a Debian LTS and seventh month as a Debian ELTS paid contributor.
I was assigned 60.00 hours for LTS and 60.00 hours for ELTS and worked on the following things:
(however, I had overworked for 9 hours for both, LTS and ELTS, last month so I had to work for 51 hours for both this month!)

LTS CVE Fixes and Announcements:

ELTS CVE Fixes and Announcements:

Other (E)LTS Work:
  • Front-desk duty from 22-02 until 28-02 for both LTS and ELTS.
  • Triaged privoxy, dnsmasq, openldap, libzstd, ruby-mechanize, firefox-esr, thunderbird, screen, xterm, glibc, isync, rails, openscad, imagemagick, avahi, gdk-pixbuf, python-reportlab, python-aiohttp, spip, gdisk, and jasper.
  • Marked CVE-2021-20214/privoxy as not-affected for stretch.
  • Marked CVE-2021-27645/glibc as no-dsa for stretch.
  • Marked CVE-2021-20247/isync as no-dsa for stretch.
  • Marked CVE-2020-28599/openscad as no-dsa for stretch.
  • Markec CVE-2021-2024 1,4-6 /imagemagick as ignored for stretch.
  • Marked CVE-2021-26720/avahi as postponed for jessie.
  • Marked CVE-2021-20240/gdk-pixbuf as not-affected for jessie.
  • Marked CVE-2021-27645/glibc as no-dsa for jessie.
  • Marked CVE-2020-28463/python-reportlab as postponed for jessie.
  • Document extra CVEs as notes for imagemagick in jessie.
  • Auto EOL ed libupnp, webkit2gtk, libraw, jackson-dataformat-cbor, node-lodash, linux, asterisk, yara, python-django, botan1.10, smarty3, xen, u-boot, steghide, mumble, gsoap, ruby-twitter-stream, isync, nodejs, openscad, mupdf, mongo-java-driver, firefox-esr, thunderbird, and salt for jessie.
  • Sponsored upload for php-horde-text-filter for Sylvain and published its DLA announcement.
  • Got CVE-2021-26937 for screen. Yay, this is the 2nd one I got assigned! \o/
  • Got CVE-2021-27135 for xterm. Woah, this is the 3rd one, am I on a roll or what? \o/
  • Co-ordinated with package maintainer (and upstream) of ca-certificates for backporting patch to stretch.
  • Co-ordinated with package maintainer of ca-certificates for backporting patch to stretch.
  • Co-ordinated with package maintainer of screen for fixing vulnerabilites in stretch.
  • Attended monthly meeting for Debian LTS.
  • Answered questions (& discussions) on IRC (#debian-lts and #debian-elts).
  • Cross-checked LTS survey results, emailed Ola about the problems found.
  • General and other discussions on LTS private and public mailing list.

Until next time.
:wq for today.

11 February 2021

Julien Danjou: Debugging C code on macOS

Debugging C code on macOSI started to write C 25 years ago now, with many different tools over the year. As many open source developers, I spent most of my life working with the GNU tools out there.As I've been using an Apple computer over the last years, I had to adapt to this environment and learn the tricks of the trade. Here are some of my notes so a search engine can index them and I'll be able to find them later.

Debugger: lldbI was used to gdb for most of years doing C. I never managed to install gdb correctly on macOS as it needs certificates, authorization, you name it, to work properly.macOS provides a native debugger named lldb, which really looks like gdb to me it runs in a terminal with a prompt.I had to learn the few commands I mostly use, which are:
  • lldb -- myprogram -options to run the program with options
  • r to run the program
  • bt or bt N to get a backtrace of the latest N frames
  • f N to select frame N
  • p V to print some variable value or memory address
Those commands cover 99% of my use case with a debugger when writing C, so once I lost my old gdb habits, I was good to go.

Debugging Memory Overflows

On GNU/LinuxOne of my favorite tools when writing C has always been Electric Fence (and DUMA more recently). It's a library that overrides the standard memory manipulation function (e.g., malloc) and instantly makes the program crash when an out of memory error is produced, rather than corrupting the heap.Heap corruption issues are hard to debug without such tools as they can happen at any time and stay unnoticed for a while, crashing your program in a totally different location later.There's no need to compile your program with those libraries. By using the dynamic loader, you can preload them and overload the standard C library functions.
LD_PRELOAD=/usr/lib/libefence.so.0.0 my-program
Run my-program with Eletric Fence loaded on GNU/Linux
My gdb configuration has been sprinkle with my friends efence and duma, and I would activate them from gdb easily with this configuration in ~/.gdbinit:
define efence
        set environment EF_PROTECT_BELOW 0
        set environment EF_ALLOW_MALLOC_0 1
        set environment LD_PRELOAD /usr/lib/libefence.so.0.0
        echo Enabled Electric Fence\n
end
document efence
Enable memory allocation debugging through Electric Fence (efence(3)).
        See also nofence and underfence.
end
define underfence
        set environment EF_PROTECT_BELOW 1
        set environment EF_ALLOW_MALLOC_0 1
        set environment LD_PRELOAD /usr/lib/libefence.so.0.0
        echo Enabled Electric Fence for underflow detection\n
end
document underfence
Enable memory allocation debugging for underflows through Electric Fence
(efence(3)).
        See also nofence and efence.
end
define nofence
        unset environment LD_PRELOAD
        echo Disabled Electric Fence\n
end
document nofence
Disable memory allocation debugging through Electric Fence (efence(3)).
end
define duma
        set environment DUMA_PROTECT_BELOW 0
        set environment DYMA_ALLOW_MALLOC_0 1
        set environment LD_PRELOAD /usr/lib/libduma.so
        echo Enabled DUMA\n
end
document duma
Enable memory allocation debugging through DUMA (duma(3)).
        See also noduma and underduma.
end

On macOSI've been looking for equivalent features in macOS, and after many hours of research, I found out that this feature is shipped natively with libgmalloc. It works in the same way, and its features are documented by Apple.My ~/.lldbinit file now contains the following:
command alias gm _regexp-env DYLD_INSERT_LIBRARIES=/usr/lib/libgmalloc.dylib
This command alias allows enabling gmalloc by just typing gm at the lldb prompt and then run the program again to see if it crashes with gmalloc enabled.

Debugging CPythonIt's not a mystery that I spend a lot of time writing Python code that's the main reason I've been doing C lately.When playing with CPython, it can be useful to, e.g., dump the content of PyObject structs on the heap or get the Python backtrace.I've been using cpython-lldb for this with great success. It adds a few bells and whistles when debugging CPython or extensions inside lldb. For example, the alias py-bt is handy to get the Python traceback of your calls rather than a bunch of cryptic C frames.Now, you should be ready to debug your nasty issues and memory problems on macOS efficiently!

6 February 2021

Andrew Cater: Debian 10.8 release process - We're almost there - Signing and pushing about to happen

Steve is about to sign the release and to begin the push across to the mirrors.Although it sometimes seems like an age, this will be approximately 8 hours rather than 15 hours - a 50% improvement in release timetable means that the behind the scenes work has been well worth while.We always find tweaks, improvements, things we forgot and genuine bugs. Once again: We've found that live images can be significantly memory intensive on older hardware or machines with limited memory. The boot process for any live CD image expands a squashfs so that the image runs entirely in memory. This is particularly noticeable on the 32 bit i386 images. These really require a minimum of 2GB of memory if you are using the heavier weight desktop images like KDE or Gnome: much less and they will either be unreasonably slow or just fail to work.There is also now a note on the download pages warning on this.Thanks once again to maswan early on and Sledge, RattusRattus, Isy, Schweer, linux-fan and sqr not for helping out in testing images. I'm listening in to an impromptu chat explaining the behind the scenes steps to run to actually sign and push the final images so we're a short while away from publishing to the principal CD image machine. At the same time, the torrent seeders will be started and the scripts will push to update the mirrors. And so to the next time :)Handy tip For people running mirrors of debian-cd: If you're low on disk space, perhaps you could remove the 10.7 images by hand before running your next sync scripts to allow space for 10.8 to move in: any ftpsync or rsync process might only remove the old images after copying in the new ones. It's only about 220GB - but you don't want that to be an instantaneous 440GB.

1 February 2021

Utkarsh Gupta: FOSS Activites in January 2021

Here s my (sixteenth) monthly update about the activities I ve done in the F/L/OSS world.

Debian
This was my 25th month of contributing to Debian. I became a DM in late March 2019 and a DD on Christmas 19! \o/ This month was bat-shit crazy. Why? We ll come to it later, probably 15th of this month?
Anyway, besides being crazy, hectic, adventerous, and the first of 2021, this month I was super-insanely busy. With what? Hm, more about this later this month! ^_^ However, I still did some Debian stuff here and there. Here are the following things I worked on:

Uploads and bug fixes:

Other $things:
  • Attended the Debian Ruby team meeting.
  • Mentoring for newcomers.
  • Moderation of -project mailing list.
  • Sponsored golang-github-gorilla-css for Fedrico.

Debian (E)LTS
Debian Long Term Support (LTS) is a project to extend the lifetime of all Debian stable releases to (at least) 5 years. Debian LTS is not handled by the Debian security team, but by a separate group of volunteers and companies interested in making it a success. And Debian Extended LTS (ELTS) is its sister project, extending support to the Jessie release (+2 years after LTS support). This was my sixteenth month as a Debian LTS and seventh month as a Debian ELTS paid contributor.
I was assigned 26.00 hours for LTS and 36.75 hours for ELTS and worked on the following things:
(however, I worked extra for 9 hours for LTS and 9 hours for ELTS this month, which I intend to balance from the next month!)

LTS CVE Fixes and Announcements:

ELTS CVE Fixes and Announcements:

Other (E)LTS Work:
  • Front-desk duty from 28-12 until 03-01 and from 25-01 until 31-01 for both LTS and ELTS.
  • Triaged dropbear, gst-plugins-bad1.0, phpmyadmin, qemu, firefox-esr, thunderbird, openldap, libdatetime-timezone-perl, tzdata, jasper, ckeditor, liblivemedia, wavpack, and ruby-redcarpet.
  • Marked CVE-2019-12953/dropbear as postponed for jessie.
  • Marked CVE-2019-12953/dropbear as postponed for stretch.
  • Marked CVE-2018-19841/wavpack as not-affected for jessie.
  • Marked CVE-2019-1010315/wavpack as not-affected for jessie.
  • Marked CVE-2019-1010317/wavpack as not-affected for jessie.
  • Marked CVE-2021-21252/phpmyadmin as no-dsa for stretch.
  • Marked CVE-2021-20196/qemu as postponed for stretch.
  • Marked CVE-2021-21252/phpmyadmin as no-dsa for jessie.
  • Marked CVE-2021-20196/qemu as postponed for jessie.
  • Marked CVE-2020-11947/qemu as postponed for jessie.
  • Marked CVE-2021-3326/glibc as no-dsa for jessie.
  • Marked CVE-2021-3326/glibc as no-dsa for stretch.
  • Marked CVE-2020-35517/qemu as not-affected instead of postponed for jessie.
  • Marked CVE-2021-2627 1,2 /ckeditor as postponed for jessie.
  • Marked CVE-2020-24027/liblivemedia as no-dsa for stretch.
  • Marked CVE-2021-2627 1,2 /ckeditor as postponed for stretch.
  • Auto EOL ed csync2, firefox-esr, linux, thunderbird, collabtive, activemq, and xen for jessie.
  • Got my first ever CVE assigned - CVE-2021-3181 for mutt. Weeeehooooo! \o/
  • Attended the monthly LTS meeting. Logs here.
  • General discussion on LTS private and public mailing list.

Interesting Bits!
  • This January, on 23rd and 24th, we had Mini DebConf India 2021 online.
    I had a talk as well, titled, Why Point Releases are important and how you can help prepare them?". It was a fun and a very short talk, where I just list out the reasons and ways to help in the preparation of point releases . I did some experimentation with this talk, figuring out what works for the audience and what doesn t and where can I improve for the next time I talk about this topic! \o/
    You can listen to the talk here and let me know if you have any feedback! Anyway, the conference lasted for 2 days and I also did some volunteering (talk director, talk miester) in Hindi and English, both! It was all so fun and new. Anyway, here s the picture we took:
  • In another exciting news, I got my first CVE assigned!!! \o/
    No, it is not something that I found, it was discovered by Tavis Ormandy. I just assigned this a CVE ID, CVE-2021-3181.
    This is my first, so I am very excited about this! ^_^
  • Besides, there s something more that is in the pipelines. Can t talk about it now, shh. But hopefully very sooooooon!

Other $things! \o/ This month was tiresome, with most of the time being spent on the Debian stuff, I did very little work outside it, really. The issues and patches that I sent are:
  • Issue #700 for redcarpet, asking for a reproducer for CVE-2020-26298 and some additional patch related queries.
  • Issue #7 for in-parallel, asking them to not use relative paths for tests.
  • Issue #8 for in-parallel, reporting a test failure for the library.
  • Issue #2 for rake-ant, asking them to bump their dependencies to a newer version.
  • PR #3 for rake-ant, bumping the dependencies to a newer version, fixing the above issue, heh.
  • Issue #4 for rake-ant, requesting to drop git from their gemspec.
  • PR #5 for rake-ant, dropping git from gemspec, fixing the above issue, heh.
  • Issue #95 for WavPack, asking for a review of past security vulnerabilites wrt v4.70.0.
  • Reviewed PR #128 for ruby-openid, addressing the past regression with CVE fix merge.
  • Reviewed PR #63 for cocoapods-acknowledgements, updating redcarpet to v3.5.1, as a safety measure due to recently discovered vulnerability.
  • Issue #1331 for bottle, asking for relevant commits for CVE-2020-28473 and clarifying other things.
  • Issue #5 for em-redis, reporting test failures on IPv6-only build machines.
  • Issue #939 for eventmachine, reporting test failures for em-redis on IPv6-only build machines.

Until next time.
:wq for today.

4 January 2021

Russ Allbery: Review: The Once and Future Witches

Review: The Once and Future Witches, by Alix E. Harrow
Publisher: Redhook Books
Copyright: October 2020
ISBN: 0-316-42202-9
Format: Kindle
Pages: 515
Once upon a time there were three sisters. They were born in a forgotten kingdom that smelled of honeysuckle and mud, where the Big Sandy ran wide and the sycamores shone white as knuckle-bones on the banks. The sisters had no mother and a no-good father, but they had each other; it might have been enough. But the sisters were banished from their kingdom, broken and scattered.
The Once and Future Witches opens with Juniper, the youngest, arriving in the city of New Salem. The year is 1893, but not in our world, not quite; Juniper has witch-ways in her pocket and a few words of power. That's lucky for her because the wanted posters arrived before she did. Unbeknownst to her or to each other, her sisters, Agnes and Bella, are already in New Salem. Agnes works in a cotton mill after having her heart broken one too many times; the mill is safer because you can't love a cotton mill. Bella is a junior librarian, meek and nervous and uncertain but still fascinated by witch-tales and magic. It's Bella who casts the spell, partly by accident, partly out of wild hope, but it was Juniper arriving in the city who provided the final component that made it almost work. Not quite, not completely, but briefly the lost tower of Avalon appears in St. George's Square. And, more importantly, the three sisters are reunited. The world of the Eastwood sisters has magic, but the people in charge of that world aren't happy about it. Magic is a female thing, contrary to science and, more importantly, God. History has followed a similar course to our world in part because magic has been ruthlessly suppressed. Inquisitors are a recent memory and the cemetery has a witch-yard, where witches are buried unnamed and their ashes sown with salt. The city of New Salem is called New Salem because Old Salem, that stronghold of witchcraft, was burned to the ground and left abandoned, fit only for tourists to gawk at the supposedly haunted ruins. The women's suffrage movement is very careful to separate itself from any hint of witchcraft or scandal, making its appeals solely within the acceptable bounds of the church. Juniper is the one who starts to up-end all of that in New Salem. Juniper was never good at doing what she was told. This is an angry book that feels like something out of another era, closer in tone to a Sheri S. Tepper or Joanna Russ novel than the way feminism is handled in recent work. Some of that is the era of the setting, before women even had the right to vote. But primarily it's because Harrow, like those earlier works, is entirely uninterested in making excuses or apologies for male behavior. She takes an already-heated societal conflict and gives the underdogs magic, which turns it into a war. There is likely a better direct analogy from the suffrage movement, but the comparison that came to my mind was if Martin Luther King, Jr. proved ineffective or had not existed, and instead Malcolm X or the Black Panthers became the face of the Civil Rights movement. It's also an emotionally exhausting book. The protagonists are hurt and lost and shattered. Their moments of victory are viciously destroyed. There is torture and a lot of despair. It works thematically; all the external solutions and mythical saviors fail, but in the process the sisters build their own strength and their own community and rescue themselves. But it's hard reading at times if you're emotionally invested in the characters (and I was very invested). Harrow does try to balance the losses with triumphs and that becomes more effective and easier to read in the back half of the book, but I struggled with the grimness at the start. One particular problem for me was that the sisters start the book suspicious and distrustful of each other because of lies and misunderstandings. This is obvious to the reader, but they don't work through it until halfway through the book. I can't argue with this as a piece of characterization it made sense to me that they would have reacted to their past the way that they did. But it was still immensely frustrating to read, since in the meantime awful things were happening and I wanted them to band together to fight. They also worry over the moral implications of the fate of their father, whereas I thought the only problem was that the man couldn't die more than once. There too, it makes sense given the moral framework the sisters were coerced into, but it is not my moral framework and it was infuriating to see them stay trapped in it for so long. The other thing that I found troubling thematically is that Harrow personalizes evil. I thought the more interesting moral challenge posed in this book is a society that systematically abuses women and suppresses their power, but Harrow gradually supplants that systemic conflict with a villain who has an identity and a backstory. It provides a more straightforward and satisfying climax, and she does avoid the trap of letting triumph over one character solve all the broader social problems, but it still felt too easy. Worse, the motives of the villain turn out to be at right angles to the structure of the social oppression. It's just a tool he's using, and while that's also believable, it means the transfer of the narrative conflict from the societal to the personal feels like a shying away from a sharper political point. Harrow lets the inhabitants of New Salem off too easily by giving them the excuse of being manipulated by an evil mastermind. What I thought Harrow did handle well was race, and it feels rare to be able to say this about a book written by and about white women. There are black women in New Salem as well, and they have their own ways and their own fight. They are suspicious of the Eastwood sisters because they're worried white women will stir up trouble and then run away and leave the consequences to fall on black women... and they're right. An alliance only forms once the white women show willingness to stay for the hard parts. Black women are essential to the eventual success of the protagonists, but the opposite is not necessarily true; they have their own networks, power, and protections, and would have survived no matter what the Eastwoods did. The book is the Eastwoods' story, so it's mostly concerned with white society, but I thought Harrow avoided both making black women too magical or making white women too central. They instead operate in parallel worlds that can form the occasional alliance of mutual understanding. It helps that Cleopatra Quinn is one of the best characters of the book. This was hard, emotional reading. It's the sort of book where everything has a price, even the ending. But I'm very glad I read it. Each of the three sisters gets their own, very different character arc, and all three of those arcs are wonderful. Even Agnes, who was the hardest character for me to like at the start of the book and who I think has the trickiest story to tell, becomes so much stronger and more vivid by the end of the book. Sometimes the descriptions are trying a bit too hard and sometimes the writing is not quite up to the intended goal, but some of the descriptions are beautiful and memorable, and Harrow's way of weaving the mythic and the personal together worked for me. This is a more ambitious book than The Ten Thousand Doors of January, and while I think the ambition exceeded Harrow's grasp in a few places and she took a few thematic short-cuts, most of it works. The characters felt like living and changing people, which is not easy given how heavily the story structure leans on maiden, mother, and crone archetypes. It's an uncompromising and furious book that turns the anger of 1970s feminist SF onto themes that are very relevant in 2021. You will have to brace yourself for heartbreak and loss, but I think it's fantasy worth reading. Recommended. Rating: 8 out of 10

28 December 2020

Guido G nther: phosh overview

phosh is graphical shell for mobile, touch based devices like smart phones. It's the default graphical shell on Purism's Librem 5 (and that's where it came to life) but projects like postmarketOS, Mobian and Debian have picked it up putting it into use on other devices as well and contributing patches. This post is meant as a short overview how things are tied together so further posts can provide more details. A PHone SHell As mobile shell phosh provides the interface components commonly found on mobile devices to phosh's overview 2 phosh's lockscreen phosh's overview 1
  • launch applications
  • switch between running applications and close them
  • lock and unlock the screen
  • display status information (e.g. network connectivity, battery level)
  • provide quick access to things like torch or Bluetooth
  • show notifications
It uses GObject object system and GTK to build up the user interface components. Mobile specific patterns are brought in via libhandy. Since phosh is meant to blend into GNOME as seamlessly as possible it uses the common interfaces present there via D-Bus like org.gnome.Screensaver or org.gnome.keyring.SystemPrompter and retrieves user configuration like keybindings via GSettings from preexisting schema. The components of a running graphical session roughly look like this: phosh session The blue boxes are the very same found on GNOME desktop sessions while the white ones are currently only found on phones. feedbackd is explained quickly: It's used for providing haptic or visual user feedback and makes your phone rumble and blink when applications (or the shell) want to notify the user about certain events like incoming phone calls or new messages. What about phoc and squeekboard? phoc and squeekboard Although some stacks combine the graphical shell with the display server (the component responsible for drawing applications and handling user input) this isn't the case for phosh. phosh relies on a Wayland compositor to be present for that. Keeping shell and compositor apart has some advantages like being able to restart the shell without affecting other applications but also adds the need for some additional communication between compositor and shell. This additional communication is implemented via Wayland protocols. The Wayland compositor used with phosh is called phoc for PHone Compositor. One of these additional protocols is wlr-layer-shell. It allows the shell to reserve space on the screen that is not used by other applications and allows it to draw things like the top and bottom bar or lock screen. Other protocols used by phosh (and hence implemented by phoc) are wlr-output-management to get information on and control properties of monitors or wlr-foreign-toplevel-management to get information about other windows on the display. The later is used to allow to switch between running applications. However these (and other) Wayland protocols are not implemented in phoc from scratch. phoc leverages the wlroots library for that. The library also handles many other compositor parts like interacting with the video and input hardware. The details on how phoc actually puts things up on the screen deserves a separate post. For the moment it's sufficient to note that phosh requires a Wayland compositor like phoc. We've not talked about entering text without a physical keyboard yet - phosh itself does not handle that either. squeekboard is the on screen keyboard for text (and emoji) input. It again uses Wayland protocols to talk to the Wayland compositor and it's (like phosh) a component that wants exclusive access to some areas of the screen (where the keyboard is drawn) and hence leverages the layer-shell protocol. Very roughly speaking it turns touch input in that area into text and sends that back to the compositor that then passes it back to the application that currently gets the text input. squeekboard's main author dcz has some more details here. The session So how does the graphical session in the picture above come into existence? As this is meant to be close to a regular GNOME session it's done via gnome-session that is invoked somewhat like:
phoc -E 'gnome-session --session=phosh'
So the compositor phoc is started up, launches gnome-session which then looks at phosh.session for the session's components. These are phosh, squeekboard and gnome-settings-daemon. These then either connect to already running services via D-Bus (e.g. NetworkManager, ModemManager, ...) or spawn them via D-Bus activation when required (e.g. feedbackd). Calling conventions So when talking about phosh it's good to keep several things apart:
  • phosh - the graphical shell
  • phoc - the compositor
  • squeekboard - the on screen keyboard
  • phosh.session: The session that ties these and GNOME together
On top of that people sometimes refer to 'Phosh' as the software collection consisting of the above plus more components from GNOME (Settings, Contacs, Clocks, Weather, Evince, ...) and components that currently aren't part of GNOME but adapt to small screen sizes, use the same technologies and are needed to make a phone fun to use e.g. Geary for email, Calls for making phone calls and Chats for SMS handling. Since just overloading the term Phosh is confusing GNOME/Phosh Mobile Environment or Phosh Mobile Environment have been used to describe the above collection of software and I've contacted GNOME on how to name this properly, to not infringe on the GNOME trademark but also give proper credit and hopefully being able to move things upstream that can live upstream. That's it for a start. phosh's development documentation can be browsed here but is also available in the source code. Besides the projects mentioned above credits go to Purism for allowing me and others to work on the above and other parts related to moving Free Software on mobile Linux forward.

26 December 2020

Paul Tagliamonte: Reverse Engineering my Christmas Tree

Over the course of the last year and a half, I ve been doing some self-directed learning on how radios work. I ve gone from a very basic understanding of wireless communications (there s usually some sort of antenna, I guess?) all the way through the process of learning about and implementing a set of libraries to modulate and demodulate data using my now formidable stash of SDRs. I ve been implementing all of the RF processing code from first principals and purely based on other primitives I ve written myself to prove to myself that I understand each concept before moving on. I figured that there was a fun capstone to be done here - the blind reverse engineering and implementation of the protocol my cheep Amazon power switch uses to turn on and off my Christmas Tree. All the work described in this post was done over the course of a few hours thanks to help during the demodulation from Tom Bereknyei and hlieberman.

Going in blind When I first got my switch, I checked it for any FCC markings in order to look up the FCC filings to determine the operational frequency of the device, and maybe some other information such as declared modulation or maybe even part numbers and/or diagrams. However, beyond a few regulatory stickers, there were no FCC ids or other distinguishing IDs on the device. Worse yet, it appeared to be a whitelabeled version of another product, so searching Google for the product name was very unhelpful. Since operation of this device is unlicensed, I figured I d start looking in the ISM band. The most common band used that I ve seen is the band starting at 433.05MHz up to 434.79MHz. I fired up my trusty waterfall tuned to a center frequency of 433.92MHz (since it s right in the middle of the band, and it let me see far enough up and down the band to spot the remote) and pressed a few buttons. Imagine my surprise when I realize the operational frequency of this device is 433.920MHz, exactly dead center. Weird, but lucky! After taking a capture, I started to look at understanding what the modulation type of the signal was, and how I may go about demodulating it. Using inspectrum, I was able to clearly see the signal in the capture, and it immediately stuck out to my eye to be encoded using OOK / ASK. Next, I started to measure the smallest pulse, and see if I could infer the symbols per second, and try to decode it by hand. These types of signals are generally pretty easy to decode by eye. This wound up giving me symbol rate of 2.2 Ksym/s, which is a lot faster than I expected. While I was working by hand, Tom demodulated a few messages in Python, and noticed that if you grouped the bits into groups of 4, you either had a 1000 or a 1110 which caused me to realize this was encoded using something I saw documented elsewhere, where the 0 is a short pulse, and a 1 is a long pulse, not unlike morse code, but where each symbol takes up a fixed length of time (monospace morse code?). Working on that assumption, I changed my inspectrum symbol width, and demodulated a few more by hand. This wound up demodulating nicely (and the preamble / clock sync could be represented as repeating 0s, which is handy!) and gave us a symbol rate of 612(ish) symbols per second a lot closer to what I was expecting. If we take the code for on in the inspectrum capture above and demodulate it by hand, we get 0000000000110101100100010 (treat a short pulse as a 0, and a long pulse as a 1). If you re interested in following along at home, click on the inspectrum image, and write down the bits you see, and compare it to what I have! Right, so it looks like from what we can tell so far that the packet looks something like this:
preamble / sync
stuff
Next, I took a capture of all the button presses and demodulated them by hand, and put them into a table to try and understand the format of the messages:
Button Demod'd Bits
On 0000000000110101100100010
Off 00000000001101011001010000
Dim Up 0000000000110101100110100
Dim Down 0000000000110101100100100
Timer 1h 0000000000110101100110010
Timer 2h 0000000000110101100100110
Timer 4h 0000000000110101100100000
Dim 100% 0000000000110101000101010
Dim 75% 00000000001101010001001100
Dim 50% 00000000001101010001001000
Dim 25% 0000000000110101000100000
Great! So, this is enough to attempt to control the tree with, I think so I wrote a simple modulator. My approach was to use the fact that I can break down a single symbol into 4 sub-symbol components which is to say, go back to representing a 1 as 1110, and a 0 as 1000. This let me allocate IQ space for the symbol, break the bit into 4 symbols, and if that symbol is 1, write out values from a carrier wave (cos in the real values, and sin in the imaginary values) to the buffer. Now that I can go from bits to IQ data, I can transmit that IQ data using my PlutoSDR or HackRF and try and control my tree. I gave it a try, and the tree blinked off! Success! But wait that s not enough for me I know I can t just demodulate bits and try and replay the bits forever there s stuff like addresses and keys and stuff, and I want to get a second one of these working. Let s take a look at the bits to see if we spot anything fun & interesting. At first glance, a few things jumped out at me as being weird? First is that the preamble is 10 bits long (fine, let s move along - maybe it just needs 8 in a row and there s two to ensure clocks sync?). Next is that the messages are not all the same length. I double (and triple!) checked the messages, and it s true, the messages are not all the same length. Adding an extra bit at the end didn t break anything, but I wonder if that s just due to the implementation rather than the protocol. But, good news, it looks like we have a stable prefix to the messages from the remote must be my device s address! The stable 6 bits that jump out right away are 110101. Something seems weird, though, 6 bits is a bit awkward, even for a bit limited embedded device. Why 6? But hey, wait, we had 10 bits in the preamble, what if we have an 8 bit address meaning my device is 00110101, and the preamble is 8 0 symbols! Those are numbers that someone working on an 8 bit aligned platform would pick! To test this, I added a 0 to the preamble to see if the message starts at the first 1, or if it requires all the bits to be fully decoded, and lo and behold, the tree did not turn on or off. This would seem to me to confirm that the 0s are part of the address, and I can assume we have two 8 bit aligned bytes in the prefix of the message.
preamble / sync
address
stuff
Now, when we go through the 9-10 bits of stuff , we see all sorts of weird bits floating all over the place. The first 4 bits look like it s either 1001 or 0001, but other than that, there s a lot of chaos. This is where things get really squishy. I needed more information to try and figure this out, but no matter how many times I sent a command it was always the same bits (so, no counters), and things feel very opaque still. The only way I was going to make any progress is to get another switch and see how the messages from the remote change. Off to Amazon I went, and ordered another switch from the same page, and eagerly waited its arrival.

Switch #2 The second switch showed up, and I hurriedly unboxed the kit, put batteries into the remote, and fired up my SDR to take a capture. After I captured the first button ( Off ), my heart sunk as I saw my lights connected to Switch #1 flicker off. Apparently the new switch and the old switch have the same exact address. To be sure, I demodulated the messages as before, and came out with the exact same bit pattern. This is a setback and letdown I was hoping to independently control my switches, but it also means I got no additional information about the address or button format. The upside to all of this, though, is that because the switches are controlled by either remote, I only needed one remote, so why not pull it apart and see if I can figure out what components it s using to transmit, and find any datasheets I can. The PCB was super simple, and I wound up finding a WL116SC IC on the PCB. After some googling, I found a single lone datasheet, entirely in Chinese. Thankfully, Google Translate seems to have worked well enough on technical words, and I was able to put together at least a little bit of understanding based on the documentation that was made available. I took a few screenshots below - I put the google translated text above the hanzi. From that sheet, we can see we got the basics of the 1 and 0 symbol encoding right (I was halfway expecting the bits to be flipped), and a huge find by way of a description of the bits in the message! It s a bummer that we missed the clock sync / preamble pulse before the data message, but that s OK somehow. It also turns out that 8 or 10 bit series of of 0"s wasn t clock sync at all - it was part of the address! Since it also turns out that all devices made by this manufacturer have the hardcoded address of []byte 0x00, 0x35 , that means that the vast majority of bits sent are always going to be the same for any button press on any remote made by this vendor. Seems like a waste of bits to me, but hey, what do I know. Additionally, this also tells us the trailing zeros are not part of the data encoding scheme, which is progress!
address
keycode
Now, working on the assumptions validated by the datasheet, here s the updated list of scancodes we ve found:
Button Scancode Bits Integer
On 10010001 145 / 0x91
Off 10010100 148 / 0x94
Dim Up 10011010 154 / 0x9A
Dim Down 10010010 146 / 0x92
Timer 1h 10011001 154 / 0x99
Timer 2h 10010011 147 / 0x93
Timer 4h 10010000 144 / 0x90
Dim 100% 00010101 21 / 0x15
Dim 75% 00010011 19 / 0x13
Dim 50% 00010010 18 / 0x12
Dim 25% 00010000 16 / 0x10
Interestingly, I think the Dim keys may have a confirmation that we have a good demod the codes on the bottom are missing the most significant bit, and when I look back at the scancode table in the datasheet, they make an interesting pattern the bottom two rows, right and left side values match up! If you take a look, Dim 100% is S1 , Dim 75% is S19 , Dim 50% is S8 , and Dim 25% is S20 . Cool! Since none of the other codes line up, I am willing to bet the most significant bit is a Combo indicator, and not part of the button (leaving 7 bits for the keycode). And even more interestingly, one of our scancodes ( Off , which is 0x94) shows up just below this table, in the examples. Over all, I think this tells us we have the right bits to look at for determining the scan code! Great news there!

Back to the modulation! So, armed with this knowledge, I was able to refactor my code to match the timings and understanding outlined by the datasheet and ensure things still work. The switch itself has a high degree of tolerance, so being wildly off frequency or a wildly wrong symbol rate may actually still work. It s hard to know if this is more or less correct, but matching documentation seems like a more stable foundation if nothing else. This code has been really reliable, and tends to work just as well as the remote from what I ve been able to determine. I ve been using incredibly low power to avoid any interference, and it s been very robust - a testament to the engineering that went into the outlet hardware, even though it cost less than of a lot of other switches! I have a lot of respect for the folks who built this device - it s incredibly simple, reliable and my guess is this thing will keep working even in some fairly harsh RF environments. The only downside is the fact the manufacturer used the same address for all their devices, rather than programming a unique address for each outlet and remote when the underlying WL116SC chip supports it. I m sure this was done to avoid complexity in assembly (e.g. pairing the remote and outlet, and having to keep those two items together during assembly), but it s still a bummer. I took apart the switch to see if I could dump an EEPROM and change the address in ROM, but the entire thing was potted in waterproof epoxy, which is a very nice feature if this was ever used outdoors. Not good news for tinkering, though!

Unsolved Mysteries At this point, even though I understand the protocol enough to control the device, it still feels like I hit a dead end in my understanding. I m not able to figure out how exactly the scancodes are implemented, and break them down into more specific parts. They are stable and based on the physical wiring of the remote, so I think I m going to leave it a magic number. I have what I was looking for, and these magic constants appear to be the right one to use, even if I did understand how to create the codes itself. This does leave us with a few bits we never resolved, which I ll memorialize below just to be sure I don t forget about them. Question #1: According to the datasheet there should be a preamble. Why do I not see one leading the first message? My hunch is that the trailing 0 at the end of the payload is actually just the preamble for the next message (always rendering the first message invalid?). This would let us claim there s an engineering reason why we are ignoring the weird bit, and also explain away something from the documentation. It s just weird that it wouldn t be present on the first message. This theory is mostly confirmed by measuring the timing and comparing it to the datasheet, but it s not exactly in line with the datasheet timings either (specifically, it s off by 200 s, which is kinda a lot for a system using 400 s timings). I think I could go either way on the last 0 being the preamble for the next message. It could be that the first message is technically invalid, or it could also be that this was not implemented or actively disabled by the vendor for this specific application / device. It s really hard to know without getting the source code for the WL116SC chip in this specific remote or the source in the outlet itself. Question #2: Why are some keycodes 8 bits and others 9 bits? I still have no idea why there sometimes 8 bits (for instance, On ) and other times there are 9 bits (for instance, Off ) in the 8 bit keycode field. I spent some time playing with the trailing zeros, when I try and send an Off with the most significant 8 bits (without the least significant / last 9th bit, which is a 0 ), it does not turn the tree off. If I send an On with 9 bits (an additional 0 after the least significant bit), it does work, but both On and Off work when I send 10, 11 or 12 bits padded with trailing zeros. I suspect my outlet will ignore data after the switch is done reading bits regardless of trailing zeros. The docs tell me there should only be 8 bits, but it won t work unless I send 9 bits for some commands. There s something fishy going on here, and the datasheet isn t exactly right either way. Question #3: How in the heck do those scancodes work? This one drove me nuts. I ve spent countless hours on trying to figure this out, including emailing the company that makes the WL116SC (they re really nice!), and even though they were super kind and generous with documentation and example source, I m still having a hard time lining up their documentation and examples with what I see from my remote. I think the manufacturer of my remote and switch has modified the protocol enough to where there s actually something different going on here. Bummer. I wound up in my place of last resort asking friends over Signal to try and see if they could find a pattern, as well as making multiple pleas to the twittersphere, to no avail (but thank you to Ben Hilburn, devnulling, Andreas Bombe and Larme for your repiles, help and advice!) I still don t understand how they assemble the scan code for instance, if you merely add, you won t know if a key press of 0x05 is 0x03 + 0x02 or if it s 0x01 + 0x04. On the other hand, treating it as two 4-bit integers won t work for 0x10 to 0x15 (since they need 5 bits to represent). It s also likely the most significant bit is a combo indicator, which only leaves 7 bits for the actual keypress data. Stuffing 10 bits of data into 7 bits is likely resulting in some really intricate bit work. On a last ditch whim, I tried to XOR the math into working, but some initial brute forcing to make the math work given the provided examples did not result in anything. It could be a bitpacked field that I don t understand, but I don t think I can make progress on that without inside knowledge and much more work. Here s the table containing the numbers I was working off of:
Keys Key Codes Scancode
S3 + S9 0x01 + 0x03 0x96
S6 + S12 0x07 + 0x09 0x94
S22 + S10 0x0D + 0x0F 0x3F
If anyone has thoughts on how these codes work, I d love to hear about it! Send me an email or a tweet or something - I m a bit stumped. There s some trick here that is being used to encode the combo key in a way that is decodeable. If it s actually not decodeable (which is a real possibility!), this may act as a unique button combo hash which allows the receiver to not actually determine which keys are pressed, but have a unique button that gets sent when a combo is used. I m not sure I know enough to have a theory as to which it may be.

18 December 2020

Russell Coker: Thinkpad Storage Problem

For a while I ve had a problem with my Thinkpad X1 Carbon Gen 1 [1] where storage pauses for 30 seconds, it s become more common recently and unfortunately everything seems to depend on storage (ideally a web browser playing a video from the Internet could keep doing so with no disk access, but that s not what happens).
echo 3 > /sys/block/sda/device/timeout
I ve put the above in /etc/rc.local which should make it only take 3 seconds to recover from storage problems instead of 30 seconds, the problem hasn t recurred since that change so I don t know if it works as desired. The Thinkpad is running BTRFS and no data is reported as being lost, so it s not a problem to use it at this time. The bottom of this post has an example of the errors I m getting. A friend said that his searches for such errors shows people claiming that it s a SATA controller or cable problem, which for a SATA device bolted directly to the motherboard means it s likely to be a motherboard problem (IE more expensive to fix than the $289 I paid for the laptop almost 3 years ago). A Lenovo forum discussion says that the X1 Carbon Gen1 uses an unusual connector that s not M.2 SATA and not mSATA [2]. This means that a replacement is going to be expensive, $100US for a replacement from eBay when a M.2 SATA or NVMe device would cost about $50AU from my local store. It also means that I can t use a replacement for anything else. If my laptop took a regular NVMe I d buy one and test it out, if it didn t solve the problem a spare NVMe device is always handy to have. But I m not interested in spending $100US for a part that may turn out to be useless. I bought the laptop expecting that there was nothing I could fix inside it. While it is theoretically possible to upgrade the CPU etc in a laptop most people find that the effort and expense makes it better to just replace the entire laptop. With the ultra light category of laptops the RAM is soldered on the motherboard and there are even less options for replacing things. So while it s annoying that Lenovo didn t use a standard interface for this device (they did so for other models in the range) it s not a situation that I hadn t expected. Now the question is what to do next. The Thinkpad has a SD socket and micro SD cards are getting large capacities nowadays. If it won t boot from a SD card I could boot from USB and then run from SD. So even if the internal SATA device fails entirely it should still be useful. I wonder if I could find a Thinkpad with a similar problem going cheap as I m pretty sure that a Windows user couldn t make any storage device usable the way I can with Linux. I ve looked at prices on auction sites but haven t seen a Thinkpad X1 Carbon going for anywhere near the price I got this one. The nearest I ve seen is around $350 for a laptop with significant damage. While I am a little unhappy at Lenovo s choice of interface, I ve definitely got a lot more than $289 of value out of this laptop. So I m not really complaining.
[315041.837612] ata1.00: status:   DRDY  
[315041.837613] ata1.00: failed command: WRITE FPDMA QUEUED
[315041.837616] ata1.00: cmd 61/20:48:28:1e:3e/00:00:00:00:00/40 tag 9 ncq dma 
16384 out
                         res 40/00:01:00:00:00/00:00:00:00:00/e0 Emask 0x4 
(timeout)                
[315041.837617] ata1.00: status:   DRDY  
[315041.837618] ata1.00: failed command: READ FPDMA QUEUED
[315041.837621] ata1.00: cmd 60/08:50:e0:26:84/00:00:00:00:00/40 tag 10 ncq 
dma 4096 in
                         res 40/00:01:00:00:00/00:00:00:00:00/e0 Emask 0x4 
(timeout)                
[315041.837622] ata1.00: status:   DRDY  
[315041.837625] ata1: hard resetting link
[315042.151781] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[315042.163368] ata1.00: ACPI cmd ef/02:00:00:00:00:a0 (SET FEATURES) 
succeeded
[315042.163370] ata1.00: ACPI cmd f5/00:00:00:00:00:a0 (SECURITY FREEZE LOCK) 
filtered out
[315042.163372] ata1.00: ACPI cmd ef/10:03:00:00:00:a0 (SET FEATURES) filtered 
out
[315042.183332] ata1.00: ACPI cmd ef/02:00:00:00:00:a0 (SET FEATURES) 
succeeded
[315042.183334] ata1.00: ACPI cmd f5/00:00:00:00:00:a0 (SECURITY FREEZE LOCK) 
filtered out
[315042.183336] ata1.00: ACPI cmd ef/10:03:00:00:00:a0 (SET FEATURES) filtered 
out
[315042.193332] ata1.00: configured for UDMA/133
[315042.193789] sd 0:0:0:0: [sda] tag#10 FAILED Result: hostbyte=DID_OK 
driverbyte=DRIVER_SENSE
[315042.193791] sd 0:0:0:0: [sda] tag#10 Sense Key : Illegal Request [current] 
[315042.193793] sd 0:0:0:0: [sda] tag#10 Add. Sense: Unaligned write command
[315042.193795] sd 0:0:0:0: [sda] tag#10 CDB: Read(10) 28 00 00 84 26 e0 00 00 
08 00
[315042.193797] print_req_error: I/O error, dev sda, sector 8660704
[315042.193810] ata1: EH complete

5 December 2020

Thorsten Alteholz: My Debian Activities in November 2020

FTP master Unfortunately a day only has 24h. As the freeze is approaching, I had to concentrate a bit more on keeping my packages in shape. So this month I only accepted nine packages. The good news, I rejected no package. The overall number of packages that got accepted was 328. Debian LTS This was my seventy-seventh month that I did some work for the Debian LTS initiative, started by Raphael Hertzog at Freexian. This month my all in all workload has been 22.75h. During that time I did LTS uploads of:
  • [DLA 2446-1] moin security update for two CVEs
  • [DLA 2451-1] libvncserver security update for one CVE
  • [DLA 2459-1] golang-1.7 security update for two CVEs
  • [DLA 2460-1] golang-1.8 security update for three CVEs
  • [DLA 2468-1] tcpflow security update for one CVE
  • [DLA 2469-1] qemu security update for five CVEs
I also started to work on x11vnc and slirp. Last but not least I did some days of frontdesk duties. Debian ELTS This month was the twenty ninth ELTS month. During my allocated time I uploaded:
  • ELA-319-1 for libass
  • ELA-320-1 for tcpflow
  • ELA-321-1 for qemu
Unfortunately I also had to give back some hours. Last but not least I did some days of frontdesk duties. Other stuff This month I uploaded new upstream versions of: I fixed one or two bugs in: I improved packaging of: and there have been even some new packages: As it is again this time of the year, I would also like to draw some attention to the Debian Med Advent Calendar. Like the past years, the Debian Med team starts a bug squashing event from the December 1st to 24th. Every bug that is closed will be registered in the calendar. So instead of taking something from the calendar, this special one will be filled and at Christmas hopefully every Debian Med related bug is closed. Don t hesitate, start to squash :-). The announcement on the mailing list can be found here.

22 November 2020

Arturo Borrero Gonz lez: How to use nftables from python

Netfilter logo One of the most interesting (and possibly unknown) features of the nftables framework is the native python interface, which allows python programs to access all nft features programmatically, from the source code. There is a high-level library, libnftables, which is responsible for translating the human-readable syntax from the nft binary into low-level expressions that the nf_tables kernel subsystem can run. The nft command line utility basically wraps this library, where all actual nftables logic lives. You can only imagine how powerful this library is. Originally written in C, ctypes is used to allow native wrapping of the shared lib object using pure python. To use nftables in your python script or program, first you have to install the libnftables library and the python bindings. In Debian systems, installing the python3-nftables package should be enough to have everything ready to go. To interact with libnftables you have 2 options, either use the standard nft syntax or the JSON format. The standard format allows you to send commands exactly like you would do using the nft binary. That format is intended for humans and it doesn t make a lot of sense in a programmatic interaction. Whereas JSON is pretty convenient, specially in a python environment, where there are direct data structure equivalents. The following code snippet gives you an example of how easy this is to use:
#!/usr/bin/env python3

import nftables
import json
nft = nftables.Nftables()
nft.set_json_output(True)
rc, output, error = nft.cmd("list ruleset")
print(json.loads(output))
This is functionally equivalent to running nft -j list ruleset. Basically, all you have to do in your python code is:
  • import the nftables & json modules
  • init the libnftables instance
  • configure library behavior
  • run commands and parse the output (ideally using JSON)
The key here is to use the JSON format. It allows adding ruleset modification in batches, i.e. to create tables, chains, rules, sets, stateful counters, etc in a single atomic transaction, which is the proper way to update firewalling and NAT policies in the kernel and to avoid inconsistent intermediate states. The JSON schema is pretty well documented in the libnftables-json(5) manpage. The following example is copy/pasted from there, and illustrates the basic idea behind the JSON format. The structure accepts an arbitrary amount of commands which are interpreted in order of appearance. For instance, the following standard syntax input:
flush ruleset
add table inet mytable
add chain inet mytable mychain
add rule inet mytable mychain tcp dport 22 accept
Translates into JSON as such:
  "nftables": [
      "flush":   "ruleset": null  ,
      "add":   "table":  
        "family": "inet",
        "name": "mytable"
     ,
      "add":   "chain":  
        "family": "inet",
        "table": "mytable",
        "chain": "mychain"
     
      "add":   "rule":  
        "family": "inet",
        "table": "mytable",
        "chain": "mychain",
        "expr": [
              "match":  
                "left":   "payload":  
                    "protocol": "tcp",
                    "field": "dport"
                 ,
                "right": 22
             ,
              "accept": null  
        ]
     
] 
I encourage you to take a look at the manpage if you want to know about how powerful this interface is. I ve created a git repository to host several source code examples using different features of the library: https://github.com/aborrero/python-nftables-tutorial. I plan to introduce more code examples as I learn and create them. There are several relevant projects out there using this nftables python integration already. One of the most important pieces of software is firewalld. They started using the JSON format back in 2019. In the past, people interacting with iptables programmatically would either call the iptables binary directly or, in the case of some C programs, hack libiptc/libxtables libraries into their source code. The native python approach to use libnftables is a huge step forward, which should come handy for developers, network engineers, integrators and other folks using the nftables framework in a pythonic environment. If you are interested to know how this python binding works, I invite you to take a look at the upstream source code, nftables.py, which contains all the magic behind the scenes.

Markus Koschany: My Free Software Activities in October 2020

Welcome to gambaru.de. Here is my monthly report (+ the first week in November) that covers what I have been doing for Debian. If you re interested in Java, Games and LTS topics, this might be interesting for you. Debian Games
  • I released a new version of debian-games, a collection of metapackages for games. As expected the Python 2 removal takes its toll on games in Debian that depend on pygame or other Python 2 libraries. Currently we have lost more games in 2020 than could be newly introduced to the archive. All in all it could be better but also a lot worse.
  • New upstream releases were packaged for freeorion and xaos.
  • Most of the time was spent on upgrading the bullet physics library to version 3.06, testing all reverse-dependencies and requesting a transition for it. (#972395) Similar to bullet I also updated box2d, the 2D counterpart. The only reverse-dependency, caveexpress fails to build from source with box2d 2.4.1, so unless I can fix it, it doesn t make much sense to upload the package to unstable.
  • Some package polishing: I could fix two bugs in stormbaancoureur, patch by Helmut Grohne, and ardentryst that required a dependency on python3-future to start.
  • I sponsored mgba and pekka-kana-2 for Ryan Tandy and Carlos Donizete Froes
  • and started to work on porting childsplay to Python 3.
  • Finally I did a NMU for bygfoot to work around a GCC 10 FTBFS.
Debian Java
pdfsam
  • I uploaded pdfsam and its related sejda libraries to unstable and applied an upstream patch to fix an error with Debian s jackson-jr version. Everything should be usable and up-to-date now.
  • I updated mina2 and investigated a related build failure in apache-directory-server, packaged a new upstream release of commons-io and undertow and fixed a security vulnerability in junit4 by upgrading to version 4.13.1.
  • The upgrade of jflex to version 1.8.2 took a while. The package is available in experimental now but regression tests with ratt showed, that several reverse-dependencies FTBFS with 1.8.2. Since all of these projects work fine with 1.7.0, I intend to postpone the upload to unstable. No need to break something.
Misc
  • This month also saw new upstream versions of wabt and binaryen.
  • I intend to update ublock-origin in Buster but I haven t heard back from the release team yet. (#973695)
Debian LTS This was my 56. month as a paid contributor and I have been paid to work 20,75 hours on Debian LTS, a project started by Rapha l Hertzog. In that time I did the following:
  • DLA-2440-1. Issued a security update for poppler fixing 9 CVE.
  • DLA-2445-1. Issued a security update for libmaxminddb fixing 1 CVE.
  • DLA-2447-1. Issued a security update for pacemaker fixing 1 CVE. The update had to be reverted because of an unexpected permission problem. I am in contact with one of the users who reported the regression and my intention is to update pacemaker to the latest supported release in the 1.x branch. If further tests show no regressions anymore, a new update will follow shortly.
  • Investigated CVE-2020-24614 in fossil and marked the issue as no-dsa because the impact for Debian users was low.
  • Investigated the open security vulnerabilities in ansible (11) and prepared some preliminary patches. The work is ongoing.
  • Fixed the remaining zsh vulnerabilities in Stretch in line with Debian 8 Jessie , so that all versions in Debian are equally protected.
ELTS Extended Long Term Support (ELTS) is a project led by Freexian to further extend the lifetime of Debian releases. It is not an official Debian project but all Debian users benefit from it without cost. The current ELTS release is Debian 8 Jessie . This was my 29. month and I have been paid to work 15 hours on ELTS.
  • ELA-302-1. Issued a security update for poppler fixing 2 CVE. Investigated Debian bug #942391, identified the root cause and reverted the patch for CVE-2018-13988.
  • ELA-303-1. Issued a security update for junit4 fixing 1 CVE.
  • ELA-316-1. Issued a security update for zsh fixing 7 CVE.
Thanks for reading and see you next time.

12 October 2020

Markus Koschany: My Free Software Activities in September 2020

Welcome to gambaru.de. Here is my monthly report (+ the first week in October) that covers what I have been doing for Debian. If you re interested in Java, Games and LTS topics, this might be interesting for you. Debian Games
warzone2100
Debian Java
pdfsam
  • The focus was on two major packages this month, PDFsam, a tool to manipulate PDF files and Netbeans, one of the three well known Java IDEs. I basically updated every PDFsam related sejda dependency and packaged a new library libsejda-common-java, which is currently waiting in the NEW queue. As soon as this one has been approved, we should be able to see the latest release in Debian soon.
  • Unfortunately I came to the conclusion that maintaining Netbeans in Debian is no longer a viable solution. I have been the sole maintainer for the past five years and managed to package the basic Java IDE in Stretch. I also had a 98% ready package for Buster but there were some bugs that made it unfit for a stable release in my opinion. The truth is, it takes a lot of time to patch Netbeans, just to make the build system DFSG compliant and to build the IDE from source. We have never managed to provide more functionality than the basic Java IDE features too. Still, we had to maintain dozens of build-dependencies and there was a constant struggle to make everything work with just a single version of a library. While the Debian way works great for most common projects, it doesn t scale very well for very complex ones like Java IDEs. Neither Eclipse nor Netbeans are really fully maintainable in Debian since they consist of hundreds of different jar files, even if the toolchain was perfect, it would require too much time to maintain all those Debian packages.
  • I voiced that sentiment on our debian-java mailinglist while also discussing the situation of complex server packages like Apache Solr. Similar to Netbeans it requires hundreds of jar files to get running. I believe our users are better served in those cases by using tools like flatpak for desktop packages or jdeb for server packages. The idea is to provide a Debian toolchain which would download a source package from upstream and then use jdeb to create a Debian package. Thus we could provide packages for very complex Java software again, although only via the Debian contrib distribution. The pros are: software is available as Debian packages and integrates well with your system and considerably less time is needed to maintain such packages: Cons: not available in Debian main, no security support, not checked for DFSG compliance.
  • Should we do that for all of our packages? No. This should really be limited to packages that otherwise would not be in Debian at all and are too complex to maintain, when even a whole team of normal contributors would struggle.
  • Finally the consequences were: the Netbeans IDE has been removed from Debian main but the Netbeans platform package, libnb-platform18-java, is up-to-date again just like visualvm, which depends on it.
  • New upstream releases were packaged for jboss-xnio, activemq, httpcomponents-client, jasypt and undertow to address several security vulnerabilities.
  • I also packaged a new version of sweethome3d, an Interior 2D design application .
Misc
  • The usual suspects: I updated binaryen and ublock-origin.
  • I eventually filed a RFA for privacybadger. As I mentioned in my last post, the upstream maintainer would like to see regular updates in Debian stable but I don t want to regularly contribute time for this task. If someone is ready for the job, let me know.
  • I did a NMU for xjig to fix Debian bug. (#932742)
Debian LTS This was my 55. month as a paid contributor and I have been paid to work 31,75 hours on Debian LTS, a project started by Rapha l Hertzog. In that time I did the following:
  • Investigated and fixed a regression in squid3 when using the icap server. (#965012)
  • DLA-2394-1. Issued a security update for squid3 fixing 4 CVE.
  • DLA-2400-1. Issued a security update for activemq fixing 1 CVE.
  • DLA-2403-1. Issued a security update for rails fixing 1 CVE.
  • DLA-2404-1. Issued a security update for eclipse-wtp fixing 1 CVE.
  • DLA-2405-1. Issued a security update for httpcomponents-client fixing 1 CVE.
  • Triaged open CVE for guacamole-server and guacamole-client and prepared patches for CVE-2020-9498 and CVE-2020-9497.
  • Prepared patches for 7 CVE in libonig.
ELTS Extended Long Term Support (ELTS) is a project led by Freexian to further extend the lifetime of Debian releases. It is not an official Debian project but all Debian users benefit from it without cost. The current ELTS release is Debian 8 Jessie . This was my 28. month and I have been paid to work 15 hours on ELTS.
  • ELA-291-1. Issued a security update for libproxy fixing 1 CVE.
  • ELA-294-1. Issued a security update for squid3 fixing 4 CVE.
  • ELA-295-1. Issued a security update for rails fixing 2 CVE.
  • ELA-296-1. Issued a security update for httpcomponents-client fixing 1 CVE.
Thanks for reading and see you next time.

26 September 2020

Andrew Cater: There are things that money can't buy - and sensible Debian colleagues are worth gold and diamonds :)

Participating in the Debian media testing on debian-cd. One of my colleagues has just spent time to sort out an email issue having spent a couple of hours with me the other night. I now have good, working email for the first time in years - I can't value that highly enough.Sledge, RattusRattus, Isy and myself are all engaged in testing various CD images. At the same time, they're debugging a new application to save us from wiki problems when we do this - and we're also able to use a video link which is really handy to chat backwards and forwards and means I can sit virtually in Cambridge :)Lots of backchat and messages flying backwards and forwards - couldn't wish for a better way to spend an afternoon with friends.


20 September 2020

Andy Simpkins: Using an IP camera in conference calls

My webcam has broken, something that I have been using a lot during the last few months for some reason. A friend of mine suggested that I use the mic and camera on my mobile phone instead. There is a simple app droidcam that makes the phone behave as a simple webcam, it also has a client application to run on your PC to capture the web-stream and present it as a video device. All well and good but I would like to keep propitiatory software off my PCs (I have a hard enough time accepting it on a phone but I have to draw a line somewhere). I decided that there had to be a simple way to ingest that stream and present it as a local video device on a Linux box. It turns out that it is a lot simpler than I thought it would be. I had it working within 10 minutes! Packages needed: ffmpeg, v4l2loopback-utils
sudo apt-get install ffmpeg v4l2loopback-utils
Start the loop-back device:
sudo modprobe v4l2loopback
Ingest video and present on loop-back device:
ffmpeg -re -i <URL_VideoStream> -vcodec rawvideo -pix_fmt yuv420p -f v4l2 /dev/video0
Where

-re
Read input at native frame rate. Mainly used to simulate a grab
device, or live input stream (e.g. when reading from a file). Should
not be used with actual grab devices or live input streams (where it
can cause packet loss). By default ffmpeg attempts to read the
input(s) as fast as possible. This option will slow down the reading
of the input(s) to the native frame rate of the input(s). It is
useful for real-time output (e.g. live streaming).
-i <source>
In this case my phone running droidcam http://10.14.1.100:4747/video
-vcodec rawvideo
Select the output video codec to raw video (as expected for /dev/video#)
-pix_fmt yuv420p
Set pixel format. in this example tell ffmpeg that the video will
be yuv colour space at 420p resolution
-f v4l2 /dev/video0
Force the output to be in video for Linux 2 (v4l2) format bound to
device /dev/video0

14 September 2020

Jonathan McDowell: onak 0.6.1 released

Yesterday I did the first release of my OpenPGP compatible keyserver, onak, in 4 years. Actually, 2 releases because I discovered my detection for various versions of libnettle needed some fixing. It was largely driven by the need to get an updated package sorted for Debian due to the removal of dh-systemd, but it should have come sooner. This release has a number of clean-ups for dealing with the hostility shown to the keyserver network in recent years. In particular it implements some of dkg s Abuse-Resistant OpenPGP Keystores, and finally adds support for verifying signatures fully. That opens up the ability to run a keyserver that will only allow verifiable updates to keys. This doesn t tie in with folk who want to run PGP based systems because of the anonymity, but for those of us who think PGP s strength is in the web of trust it s pretty handy. And it s all configurable to taste; you can turn off all the verification if you want, or verify everything but not require any signatures, or even enable v3 keys if you feel like it. The main reason this release didn t come sooner is that I m painfully aware of the bits that are missing. In particular:
  • Documentation. It s all out of date, it needs a lot of work.
  • FastCGI support. Presently you need to run the separate CGI binaries.
  • SKS Gossip support. onak only supports the email syncing option. If you run a stand alone server this is fine, but Gossip is the right approach for a proper keyserver network.
Anyway. Available locally or via GitHub.
0.6.0 - 13th September 2020
  • Move to CMake over autoconf
  • Add support for issuer fingerprint subpackets
  • Add experimental support for v5 keys
  • Add read-only OpenPGP keyring backed DB backend
  • Move various bits into their own subdirectories in the source tree
  • Add support for full signature verification
  • Drop v3 keys by default when cleaning keys
  • Various code cleanups
  • Implement pieces of draft-dkg-openpgp-abuse-resistant-keystore-03
  • Add support for a fingerprint blacklist (e.g. Evil32)
  • Deprecate the .conf configuration file format
  • Drop version info from armored output
  • Add option to deny new keys and only allow updates to existing keys
  • Various pieces of work removing support for 32 bit key IDs and coping with colliding 64 bit key IDs.
  • Remove support for libnettle versions that lack the full SHA2 suite
0.6.1 - 13th September 2020
  • Fixes for compilation without nettle + with later releases of nettle

12 September 2020

Markus Koschany: My Free Software Activities in August 2020

Welcome to gambaru.de. Here is my monthly report (+ the first week in September) that covers what I have been doing for Debian. If you re interested in Java, Games and LTS topics, this might be interesting for you. Debian Games
teeworlds
  • I packaged a new upstream release of teeworlds, the well-known 2D multiplayer shooter with cute characters called tees to resolve a Python 2 bug (although teeworlds is actually a C++ game). The update also fixed a severe remote denial-of-service security vulnerability, CVE-2020-12066. I prepared a patch for Buster and will send it to the security team later today.
  • I sponsored updates of mgba, a Game Boy Advance emulator, for Ryan Tandy, and osmose-emulator for Carlos Donizete Froes.
  • I worked around a RC GCC 10 bug in megaglest by compiling with -fcommon.
  • Thanks to Gerardo Ballabio who packaged a new upstream version of galois which I uploaded for him.
  • Also thanks to Reiner Herrmann and Judit Foglszinger who fixed a regression (crash) in monsterz due to the earlier port to Python 3. Reiner also made fans of supertuxkart happy by packaging the latest upstream release version 1.2.
Debian Java Misc
  • I was contacted by the upstream maintainer of privacybadger, a privacy addon for Firefox and Chromium, who dislikes the idea of having a stable and unchanging version in Debian stable releases. Obviously I can t really do much about it although I believe the release team would be open-minded for regular point updates of browser addons though. However I don t intend to do regular updates for all of my packages in stable unless there is a really good reason to do so. At the moment I m willing to make an exception for ublock-origin and https-everywhere because I feel these addons should be core browser functionality anyway. I talked about this on our Debian Mozilla Extension Maintainers mailinglist and it seems someone is interested to take over privacybadger and prepare regular stable point updates. Let s see how it turns out.
  • Finally this month saw the release of ublock-origin 1.29.0 and the creation of two different browser-specific binary packages for Firefox and Chromium. I have talked about it before and I believe two separate packages for ublock-origin are more aligned to upstream development and make the whole addon easier to maintain which benefits users, upstream and maintainers.
  • imlib2, an image library, and binaryen also got updated this month.
Debian LTS This was my 54. month as a paid contributor and I have been paid to work 20 hours on Debian LTS, a project started by Rapha l Hertzog. In that time I did the following:
  • DLA-2303-1. Issued a security update for libssh fixing 1 CVE.
  • DLA-2327-1. Issued a security update for lucene-solr fixing 1 CVE.
  • DLA-2369-1. Issued a security update for libxml2 fixing 8 CVE.
  • Triaged CVE-2020-14340, jboss-xnio as not-affected for Stretch.
  • Triaged CVE-2020-13941, lucene-solr as no-dsa because the security impact was minor.
  • Triaged CVE-2019-17638, jetty9 as not-affected for Stretch and Buster.
  • squid3: I backported the patches for CVE-2020-15049, CVE-2020-15810, CVE-2020-15811 and CVE-2020-24606 from squid 4 to squid 3.
ELTS Extended Long Term Support (ELTS) is a project led by Freexian to further extend the lifetime of Debian releases. It is not an official Debian project but all Debian users benefit from it without cost. The current ELTS release is Debian 8 Jessie . This was my 27. month and I have been paid to work 14,25 hours on ELTS.
  • ELA-271-1. Issued a security update for squid3 fixing 19 CVE. Most of the work was already done before ELTS started, only the patch for CVE-2019-12529 had to be adjusted for the nettle version in Jessie.
  • ELA-273-1. Issued a security update for nss fixing 1 CVE.
  • ELA-276-1. Issued a security update for libjpeg-turbo fixing 2 CVE.
  • ELA-277-1. Issued a security update for graphicsmagick fixing 1 CVE.
  • ELA-279-1. Issued a security update for imagemagick fixing 3 CVE.
  • ELA-280-1. Issued a security update for libxml2 fixing 4 CVE.
Thanks for reading and see you next time.

9 September 2020

Reproducible Builds: Reproducible Builds in August 2020

Welcome to the August 2020 report from the Reproducible Builds project. In our monthly reports, we summarise the things that we have been up to over the past month. The motivation behind the Reproducible Builds effort is to ensure no flaws have been introduced from the original free software source code to the pre-compiled binaries we install on our systems. If you re interested in contributing to the project, please visit our main website.


This month, Jennifer Helsby launched a new reproduciblewheels.com website to address the lack of reproducibility of Python wheels. To quote Jennifer s accompanying explanatory blog post:
One hiccup we ve encountered in SecureDrop development is that not all Python wheels can be built reproducibly. We ship multiple (Python) projects in Debian packages, with Python dependencies included in those packages as wheels. In order for our Debian packages to be reproducible, we need that wheel build process to also be reproducible
Parallel to this, transparencylog.com was also launched, a service that verifies the contents of URLs against a publicly recorded cryptographic log. It keeps an append-only log of the cryptographic digests of all URLs it has seen. (GitHub repo) On 18th September, Bernhard M. Wiedemann will give a presentation in German, titled Wie reproducible builds Software sicherer machen ( How reproducible builds make software more secure ) at the Internet Security Digital Days 2020 conference.

Reproducible builds at DebConf20 There were a number of talks at the recent online-only DebConf20 conference on the topic of reproducible builds. Holger gave a talk titled Reproducing Bullseye in practice , focusing on independently verifying that the binaries distributed from ftp.debian.org are made from their claimed sources. It also served as a general update on the status of reproducible builds within Debian. The video (145 MB) and slides are available. There were also a number of other talks that involved Reproducible Builds too. For example, the Malayalam language mini-conference had a talk titled , ? ( I want to join Debian, what should I do? ) presented by Praveen Arimbrathodiyil, the Clojure Packaging Team BoF session led by Elana Hashman, as well as Where is Salsa CI right now? that was on the topic of Salsa, the collaborative development server that Debian uses to provide the necessary tools for package maintainers, packaging teams and so on. Jonathan Bustillos (Jathan) also gave a talk in Spanish titled Un camino verificable desde el origen hasta el binario ( A verifiable path from source to binary ). (Video, 88MB)

Development work After many years of development work, the compiler for the Rust programming language now generates reproducible binary code. This generated some general discussion on Reddit on the topic of reproducibility in general. Paul Spooren posted a request for comments to OpenWrt s openwrt-devel mailing list asking for clarification on when to raise the PKG_RELEASE identifier of a package. This is needed in order to successfully perform rebuilds in a reproducible builds context. In openSUSE, Bernhard M. Wiedemann published his monthly Reproducible Builds status update. Chris Lamb provided some comments and pointers on an upstream issue regarding the reproducibility of a Snap / SquashFS archive file. [ ]

Debian Holger Levsen identified that a large number of Debian .buildinfo build certificates have been tainted on the official Debian build servers, as these environments have files underneath the /usr/local/sbin directory [ ]. He also filed against bug for debrebuild after spotting that it can fail to download packages from snapshot.debian.org [ ]. This month, several issues were uncovered (or assisted) due to the efforts of reproducible builds. For instance, Debian bug #968710 was filed by Simon McVittie, which describes a problem with detached debug symbol files (required to generate a traceback) that is unlikely to have been discovered without reproducible builds. In addition, Jelmer Vernooij called attention that the new Debian Janitor tool is using the property of reproducibility (as well as diffoscope when applying archive-wide changes to Debian:
New merge proposals also include a link to the diffoscope diff between a vanilla build and the build with changes. Unfortunately these can be a bit noisy for packages that are not reproducible yet, due to the difference in build environment between the two builds. [ ]
56 reviews of Debian packages were added, 38 were updated and 24 were removed this month adding to our knowledge about identified issues. Specifically, Chris Lamb added and categorised the nondeterministic_version_generated_by_python_param and the lessc_nondeterministic_keys toolchain issues. [ ][ ] Holger Levsen sponsored Lukas Puehringer s upload of the python-securesystemslib pacage, which is a dependency of in-toto, a framework to secure the integrity of software supply chains. [ ] Lastly, Chris Lamb further refined his merge request against the debian-installer component to allow all arguments from sources.list files (such as [check-valid-until=no]) in order that we can test the reproducibility of the installer images on the Reproducible Builds own testing infrastructure and sent a ping to the team that maintains that code.

Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of these patches, including:

diffoscope diffoscope is our in-depth and content-aware diff utility that can not only locate and diagnose reproducibility issues, it provides human-readable diffs of all kinds. In August, Chris Lamb made the following changes to diffoscope, including preparing and uploading versions 155, 156, 157 and 158 to Debian:
  • New features:
    • Support extracting data of PGP signed data. (#214)
    • Try files named .pgp against pgpdump(1) to determine whether they are Pretty Good Privacy (PGP) files. (#211)
    • Support multiple options for all file extension matching. [ ]
  • Bug fixes:
    • Don t raise an exception when we encounter XML files with <!ENTITY> declarations inside the Document Type Definition (DTD), or when a DTD or entity references an external resource. (#212)
    • pgpdump(1) can successfully parse some binary files, so check that the parsed output contains something sensible before accepting it. [ ]
    • Temporarily drop gnumeric from the Debian build-dependencies as it has been removed from the testing distribution. (#968742)
    • Correctly use fallback_recognises to prevent matching .xsb binary XML files.
    • Correct identify signed PGP files as file(1) returns data . (#211)
  • Logging improvements:
    • Emit a message when ppudump version does not match our file header. [ ]
    • Don t use Python s repr(object) output in Calling external command messages. [ ]
    • Include the filename in the not identified by any comparator message. [ ]
  • Codebase improvements:
    • Bump Python requirement from 3.6 to 3.7. Most distributions are either shipping with Python 3.5 or 3.7, so supporting 3.6 is not only somewhat unnecessary but also cumbersome to test locally. [ ]
    • Drop some unused imports [ ], drop an unnecessary dictionary comprehensions [ ] and some unnecessary control flow [ ].
    • Correct typo of output in a comment. [ ]
  • Release process:
    • Move generation of debian/tests/control to an external script. [ ]
    • Add some URLs for the site that will appear on PyPI.org. [ ]
    • Update author and author email in setup.py for PyPI.org and similar. [ ]
  • Testsuite improvements:
    • Update PPU tests for compatibility with Free Pascal versions 3.2.0 or greater. (#968124)
    • Mark that our identification test for .ppu files requires ppudump version 3.2.0 or higher. [ ]
    • Add an assert_diff helper that loads and compares a fixture output. [ ][ ][ ][ ]
  • Misc:
In addition, Mattia Rizzolo documented in setup.py that diffoscope works with Python version 3.8 [ ] and Frazer Clews applied some Pylint suggestions [ ] and removed some deprecated methods [ ].

Website This month, Chris Lamb updated the main Reproducible Builds website and documentation to:
  • Clarify & fix a few entries on the who page [ ][ ] and ensure that images do not get to large on some viewports [ ].
  • Clarify use of a pronoun re. Conservancy. [ ]
  • Use View all our monthly reports over View all monthly reports . [ ]
  • Move a is a suffix out of the link target on the SOURCE_DATE_EPOCH age. [ ]
In addition, Javier Jard n added the freedesktop-sdk project [ ] and Kushal Das added SecureDrop project [ ] to our projects page. Lastly, Michael P hn added internationalisation and translation support with help from Hans-Christoph Steiner [ ].

Testing framework The Reproducible Builds project operate a Jenkins-based testing framework to power tests.reproducible-builds.org. This month, Holger Levsen made the following changes:
  • System health checks:
    • Improve explanation how the status and scores are calculated. [ ][ ]
    • Update and condense view of detected issues. [ ][ ]
    • Query the canonical configuration file to determine whether a job is disabled instead of duplicating/hardcoding this. [ ]
    • Detect several problems when updating the status of reporting-oriented metapackage sets. [ ]
    • Detect when diffoscope is not installable [ ] and failures in DNS resolution [ ].
  • Debian:
    • Update the URL to the Debian security team bug tracker s Git repository. [ ]
    • Reschedule the unstable and bullseye distributions often for the arm64 architecture. [ ]
    • Schedule buster less often for armhf. [ ][ ][ ]
    • Force the build of certain packages in the work-in-progress package rebuilder. [ ][ ]
    • Only update the stretch and buster base build images when necessary. [ ]
  • Other distributions:
    • For F-Droid, trigger jobs by commits, not by a timer. [ ]
    • Disable the Archlinux HTML page generation job as it has never worked. [ ]
    • Disable the alternative OpenWrt rebuilder jobs. [ ]
  • Misc;
Many other changes were made too, including:
  • Chris Lamb:
    • Use <pre> HTML tags when dumping fixed-width debugging data in the self-serve package scheduler. [ ]
  • Mattia Rizzolo:
  • Vagrant Cascadian:
    • Mark that the u-boot Universal Boot Loader should not build architecture independent packages on the arm64 architecture anymore. [ ]
Finally, build node maintenance was performed by Holger Levsen [ ], Mattia Rizzolo [ ][ ] and Vagrant Cascadian [ ][ ][ ][ ]

Mailing list On our mailing list this month, Leo Wandersleb sent a message to the list after he was wondering how to expand his WalletScrutiny.com project (which aims to improve the security of Bitcoin wallets) from Android wallets to also monitor Linux wallets as well:
If you think you know how to spread the word about reproducibility in the context of Bitcoin wallets through WalletScrutiny, your contributions are highly welcome on this PR [ ]
Julien Lepiller posted to the list linking to a blog post by Tavis Ormandy titled You don t need reproducible builds. Morten Linderud (foxboron) responded with a clear rebuttal that Tavis was only considering the narrow use-case of proprietary vendors and closed-source software. He additionally noted that the criticism that reproducible builds cannot prevent against backdoors being deliberately introduced into the upstream source ( bugdoors ) are decidedly (and deliberately) outside the scope of reproducible builds to begin with. Chris Lamb included the Reproducible Builds mailing list in a wider discussion regarding a tentative proposal to include .buildinfo files in .deb packages, adding his remarks regarding requiring a custom tool in order to determine whether generated build artifacts are identical in a reproducible context. [ ] Jonathan Bustillos (Jathan) posted a quick email to the list requesting whether there was a list of To do tasks in Reproducible Builds. Lastly, Chris Lamb responded at length to a query regarding the status of reproducible builds for Debian ISO or installation images. He noted that most of the technical work has been performed but there are at least four issues until they can be generally advertised as such . He pointed that the privacy-oriented Tails operation system, which is based directly on Debian, has had reproducible builds for a number of years now. [ ]

If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

2 September 2020

Kees Cook: security things in Linux v5.6

Previously: v5.5. Linux v5.6 was released back in March. Here s my quick summary of various features that caught my attention: WireGuard
The widely used WireGuard VPN has been out-of-tree for a very long time. After 3 1/2 years since its initial upstream RFC, Ard Biesheuvel and Jason Donenfeld finished the work getting all the crypto prerequisites sorted out for the v5.5 kernel. For this release, Jason has gotten WireGuard itself landed. It was a twisty road, and I m grateful to everyone involved for sticking it out and navigating the compromises and alternative solutions. openat2() syscall and RESOLVE_* flags
Aleksa Sarai has added a number of important path resolution scoping options to the kernel s open() handling, covering things like not walking above a specific point in a path hierarchy (RESOLVE_BENEATH), disabling the resolution of various magic links (RESOLVE_NO_MAGICLINKS) in procfs (e.g. /proc/$pid/exe) and other pseudo-filesystems, and treating a given lookup as happening relative to a different root directory (as if it were in a chroot, RESOLVE_IN_ROOT). As part of this, it became clear that there wasn t a way to correctly extend the existing openat() syscall, so he added openat2() (which is a good example of the efforts being made to codify Extensible Syscall arguments). The RESOLVE_* set of flags also cover prior behaviors like RESOLVE_NO_XDEV and RESOLVE_NO_SYMLINKS. pidfd_getfd() syscall
In the continuing growth of the much-needed pidfd APIs, Sargun Dhillon has added the pidfd_getfd() syscall which is a way to gain access to file descriptors of a process in a race-less way (or when /proc is not mounted). Before, it wasn t always possible make sure that opening file descriptors via /proc/$pid/fd/$N was actually going to be associated with the correct PID. Much more detail about this has been written up at LWN. openat() via io_uring
With my attack surface reduction hat on, I remain personally suspicious of the io_uring() family of APIs, but I can t deny their utility for certain kinds of workloads. Being able to pipeline reads and writes without the overhead of actually making syscalls is pretty great for performance. Jens Axboe has added the IORING_OP_OPENAT command so that existing io_urings can open files to be added on the fly to the mapping of available read/write targets of a given io_uring. While LSMs are still happily able to intercept these actions, I remain wary of the growing syscall multiplexer that io_uring is becoming. I am, of course, glad to see that it has a comprehensive (if out of tree ) test suite as part of liburing. removal of blocking random pool
After making algorithmic changes to obviate separate entropy pools for random numbers, Andy Lutomirski removed the blocking random pool. This simplifies the kernel pRNG code significantly without compromising the userspace interfaces designed to fetch cryptographically secure random numbers. To quote Andy, This series should not break any existing programs. /dev/urandom is unchanged. /dev/random will still block just after booting, but it will block less than it used to. See LWN for more details on the history and discussion of the series. arm64 support for on-chip RNG
Mark Brown added support for the future ARMv8.5 s RNG (SYS_RNDR_EL0), which is, from the kernel s perspective, similar to x86 s RDRAND instruction. This will provide a bootloader-independent way to add entropy to the kernel s pRNG for early boot randomness (e.g. stack canary values, memory ASLR offsets, etc). Until folks are running on ARMv8.5 systems, they can continue to depend on the bootloader for randomness (via the UEFI RNG interface) on arm64. arm64 E0PD
Mark Brown added support for the future ARMv8.5 s E0PD feature (TCR_E0PD1), which causes all memory accesses from userspace into kernel space to fault in constant time. This is an attempt to remove any possible timing side-channel signals when probing kernel memory layout from userspace, as an alternative way to protect against Meltdown-style attacks. The expectation is that E0PD would be used instead of the more expensive Kernel Page Table Isolation (KPTI) features on arm64. powerpc32 VMAP_STACK
Christophe Leroy added VMAP_STACK support to powerpc32, joining x86, arm64, and s390. This helps protect against the various classes of attacks that depend on exhausting the kernel stack in order to collide with neighboring kernel stacks. (Another common target, the sensitive thread_info, had already been moved away from the bottom of the stack by Christophe Leroy in Linux v5.1.) generic Page Table dumping
Related to RISCV s work to add page table dumping (via /sys/fs/debug/kernel_page_tables), Steven Price extracted the existing implementations from multiple architectures and created a common page table dumping framework (and then refactored all the other architectures to use it). I m delighted to have this because I still remember when not having a working page table dumper for ARM delayed me for a while when trying to implement upstream kernel memory protections there. Anything that makes it easier for architectures to get their kernel memory protection working correctly makes me happy. That s in for now; let me know if there s anything you think I missed. Next up: Linux v5.7.

2020, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
CC BY-SA 4.0

16 August 2020

Gunnar Wolf: DebConf20 talk recorded

Following Antonio Terceiro s post on tips for using ffmpeg for editing video, I will also share a bit of my experience producing my video for my session in DebConf20. I recorded my talk today. As Terceiro mentioned, even though I m used to speaking in front of my webcam (i.e. for my classes and some smaller conferences I ve worked on during the COVID lockdown), it does feel a bit weird to present a live talk to nobody :- OK, one step back. Why are we doing this? Because our hardworking friends of the DebConf20 video team recommended so. In order to minimize connecitvity issues from the variety of speakers throughout the world, we were requested to pre-record the exposition part of our talks, send them to the video team (deadline: today 2020-08-16, in case you still owe yours!), and make sure to be present at the end of the talk for the Q&A session. Of course, for a 45 minute talk, I prepared a 30 minute presentation, saving time for said Q&A session. Anyway, I used the excellent OBS studiolive video mixing/editing program (of course, Debian packages are available. This allowed me to set up several predefined views (combinations and layouts of the presentation, webcam, and maybe some other sources) and professionally and elegantly switch between them on the fly. I am still a newbie with OBS, but I surely see it becoming a part of my day to day streaming. Of course, my setup still was obvious (me looking right every now and then to see or control OBS, as I work on a dual-monitor setup ) Anyway, the experience was very good, much smoother and faster than what I usually have to do when editing video. But just as I was finishing thanking the (future) audience and closing the recording I had to tell the camera, oh, fuck! The button labeled Start Recording Had not been pressed. So, did I just lose 30 minutes of my life, plus a half-decent delivered talk? No, fortunately not. I had previously been playing with OBS, and configured some things. The button I did press was Start Streaming . So, my talk (swearing included, of course) was dutifully streamed over to my YouTube channel. It seems up to five people got a sneak preview as to what will my DebConf participation be (of course, I ve de-listed the video). I pulled it with the always-handy youtube-dl, edited out my curses using kdenlive, and pushed it to the DebConf video server. Oh, make sure you follow the advice for recording presentations. It has all the relevant advice, the settings you should use, and much more welcome information if you are new to this. So Next week, DebConf20! Be there or be square!

Next.

Previous.