Search Results: "mez"

21 December 2024

Dirk Eddelbuettel: anytime 0.3.11 on CRAN: Maintenance

A follow-up release 0.3.11 to the recent 0.3.10 release release of the anytime package arrived on CRAN two days ago. The package is fairly feature-complete, and code and functionality remain mature and stable, of course. anytime is a very focused package aiming to do just one thing really well: to convert anything in integer, numeric, character, factor, ordered, input format to either POSIXct (when called as anytime) or Date objects (when called as anydate) and to do so without requiring a format string as well as accomodating different formats in one input vector. See the anytime page, or the GitHub repo for a few examples, and the beautiful documentation site for all documentation. This release simply skips one test file. CRAN labeled an error M1mac yet it did not reproduce on any of the other M1 macOS I can access (macbuilder, GitHub Actions) as this appeared related to a local setting of timezone values I could not reproduce anywwhere. So the only way to get rid of the fail is to not to run the test. Needless to say the upload process was a little tedious as I got the passive-aggressive not responding treatment on a first upload and the required email answer it lead to. Anyway, after a few days, and even more deep breaths, it is taken care of and now the package result standing is (at least currently) pristinely clean. The short list of changes follows.

Changes in anytime version 0.3.11 (2024-12-18)

Skip a test file

Courtesy of my CRANberries, there is also a diffstat report of changes relative to the previous release. The issue tracker tracker off the GitHub repo can be use for questions and comments. More information about the package is at the package page, the GitHub repo and the documentation site.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can now sponsor me at GitHub.

1 November 2024

Colin Watson: Free software activity in October 2024

Almost all of my Debian contributions this month were sponsored by Freexian. You can also support my work directly via Liberapay. Ansible I noticed that Ansible had fallen out of Debian testing due to autopkgtest failures. This seemed like a problem worth fixing: in common with many other people, we use Ansible for configuration management at Freexian, and it probably wouldn t make our sysadmins too happy if they upgraded to trixie after its release and found that Ansible was gone. The problems here were really just slogging through test failures in both the ansible-core and ansible packages, but their test suites are large and take a while to run so this took some time. I was able to contribute a few small fixes to various upstreams in the process:

test: Make git archive prefix fit in 32-bit ssize_t (though upstream went for a different approach)
test: replace more deprecated assertEquals
Fix import error on Python 3.13
Make test_start_daemon_with_no_mock less flaky

This should now get back into testing tomorrow. OpenSSH Martin- ric Racine reported that ssh-audit didn t list the ext-info-s feature as being available in Debian s OpenSSH 9.2 packaging in bookworm, contrary to what OpenSSH upstream said on their specifications page at the time. I spent some time looking into this and realized that upstream was mistakenly saying that implementations of ext-info-c and ext-info-s were added at the same time, while in fact ext-info-s was added rather later. ssh-audit now has clearer output, and the OpenSSH maintainers have corrected their specifications page. I looked into a report of an ssh failure in certain cases when using GSS-API key exchange (which is a Debian patch). Once again, having integration tests was a huge win here: the affected scenario is quite a fiddly one, but I was able to set it up in the test, and thereby make sure it doesn t regress in future. It still took me a couple of hours to get all the details right, but in the past this sort of thing took me much longer with a much lower degree of confidence that the fix was correct. On upstream s advice, I cherry-picked some key exchange fixes needed for big-endian architectures. Python team I packaged python-evalidate, needed for a new upstream version of buildbot. The Python 3.13 transition rolls on. I fixed problems related to it in htmlmin, humanfriendly, postgresfixture (contributed upstream), pylint, python-asyncssh (contributed upstream), python-oauthlib, python3-simpletal, quodlibet, zope.exceptions, and zope.interface. A trickier Python 3.13 issue involved the cgi module. Years ago I ported zope.publisher to the multipart module because cgi.FieldStorage was broken in some situations, and as a result I got a recommendation into Python s dead batteries PEP 594. Unfortunately there turns out to be a name conflict between multipart and python-multipart on PyPI; python-multipart upstream has been working to disentangle this, though we still need to work out what to do in Debian. All the same, I needed to fix python-wadllib and multipart seemed like the best fit; I contributed a port upstream and temporarily copied multipart into Debian s python-wadllib source package to allow its tests to pass. I ll come back and fix this properly once we sort out the multipart vs. python-multipart packaging. tzdata moved some timezone definitions to tzdata-legacy, which has broken a number of packages. I added tzdata-legacy build-dependencies to alembic and python-icalendar to deal with this in those packages, though there are still some other instances of this left. I tracked down an nltk regression that caused build failures in many other packages. I fixed Rust crate versioning issues in pydantic-core, python-bcrypt, and python-maturin (mostly fixed by Peter Michael Green and Jelmer Vernoo , but it needed a little extra work). I fixed other build failures in entrypoints, mayavi2, python-pyvmomi (mostly fixed by Alexandre Detiste, but it needed a little extra work), and python-testing.postgresql (ditto). I fixed python3-simpletal to tolerate future versions of dh-python that will drop their dependency on python3-setuptools. I fixed broken symlinks in python-treq. I removed (build-)depends on python3-pkg-resources from alembic, autopep8, buildbot, celery, flufl.enum, flufl.lock, python-public, python-wadllib (contributed upstream), pyvisa, routes, vulture, and zodbpickle (contributed upstream). I upgraded astroid, asyncpg (fixing a Python 3.13 failure and a build failure), buildbot (noticing an upstream test bug in the process), dnsdiag, frozenlist, netmiko (fixing a Python 3.13 failure), psycopg3, pydantic-settings, pylint, python-asyncssh, python-bleach, python-btrees, python-cytoolz, python-django-pgtrigger, python-django-test-migrations, python-gssapi, python-icalendar, python-json-log-formatter, python-pgbouncer, python-pkginfo, python-plumbum, python-stdlib-list, python-tokenize-rt, python-treq (fixing a Python 3.13 failure), python-typeguard, python-webargs (fixing a build failure), pyupgrade, pyvisa, pyvisa-py (fixing a Python 3.13 failure), toolz, twisted, vulture, waitress (fixing CVE-2024-49768 and CVE-2024-49769), wtf-peewee, wtforms, zodbpickle, zope.exceptions, zope.interface, zope.proxy, zope.security, and zope.testrunner to new upstream versions. I tried to fix a regression in python-scruffy, but I need testing feedback. I requested removal of python-testing.mysqld.

27 May 2024

Thomas Koch: Minimal overhead VMs with Nix and MicroVM

Posted on March 17, 2024

Tags: debian, free software, nix

Joachim Breitner wrote about a Convenient sandboxed development environment and thus reminded me to blog about MicroVM. I ve toyed around with it a little but not yet seriously used it as I m currently not coding. MicroVM is a nix based project to configure and run minimal VMs. It can mount and thus reuse the hosts nix store inside the VM and thus has a very small disk footprint. I use MicroVM on a debian system using the nix package manager. The MicroVM author uses the project to host production services. Otherwise I consider it also a nice way to learn about NixOS after having started with the nix package manager and before making the big step to NixOS as my main system. The guests root filesystem is a tmpdir, so one must explicitly define folders that should be mounted from the host and thus be persistent across VM reboots. I defined the VM as a nix flake since this is how I started from the MicroVM projects example:

 
  description = "Haskell dev MicroVM";
  inputs.impermanence.url = "github:nix-community/impermanence";
  inputs.microvm.url = "github:astro/microvm.nix";
  inputs.microvm.inputs.nixpkgs.follows = "nixpkgs";
  outputs =   self, impermanence, microvm, nixpkgs  :
    let
      persistencePath = "/persistent";
      system = "x86_64-linux";
      user = "thk";
      vmname = "haskell";
      nixosConfiguration = nixpkgs.lib.nixosSystem  
          inherit system;
          modules = [
            microvm.nixosModules.microvm
            impermanence.nixosModules.impermanence
            ( pkgs, ...  :  
            environment.persistence.$ persistencePath  =  
                hideMounts = true;
                users.$ user  =  
                  directories = [
                    "git" ".stack"
                  ];
                 ;
               ;
              environment.sessionVariables =  
                TERM = "screen-256color";
               ;
              environment.systemPackages = with pkgs; [
                ghc
                git
                (haskell-language-server.override   supportedGhcVersions = [ "94" ];  )
                htop
                stack
                tmux
                tree
                vcsh
                zsh
              ];
              fileSystems.$ persistencePath .neededForBoot = nixpkgs.lib.mkForce true;
              microvm =  
                forwardPorts = [
                    from = "host"; host.port = 2222; guest.port = 22;  
                    from = "guest"; host.port = 5432; guest.port = 5432;   # postgresql
                ];
                hypervisor = "qemu";
                interfaces = [
                    type = "user"; id = "usernet"; mac = "00:00:00:00:00:02";  
                ];
                mem = 4096;
                shares = [  
                  # use "virtiofs" for MicroVMs that are started by systemd
                  proto = "9p";
                  tag = "ro-store";
                  # a host's /nix/store will be picked up so that no
                  # squashfs/erofs will be built for it.
                  source = "/nix/store";
                  mountPoint = "/nix/.ro-store";
                   
                  proto = "virtiofs";
                  tag = "persistent";
                  source = "~/.local/share/microvm/vms/$ vmname /persistent";
                  mountPoint = persistencePath;
                  socket = "/run/user/1000/microvm-$ vmname -persistent";
                 
                ];
                socket = "/run/user/1000/microvm-control.socket";
                vcpu = 3;
                volumes = [];
                writableStoreOverlay = "/nix/.rwstore";
               ;
              networking.hostName = vmname;
              nix.enable = true;
              nix.nixPath = ["nixpkgs=$ builtins.storePath <nixpkgs> "];
              nix.settings =  
                extra-experimental-features = ["nix-command" "flakes"];
                trusted-users = [user];
               ;
              security.sudo =  
                enable = true;
                wheelNeedsPassword = false;
               ;
              services.getty.autologinUser = user;
              services.openssh =  
                enable = true;
               ;
              system.stateVersion = "24.11";
              systemd.services.loadnixdb =  
                description = "import hosts nix database";
                path = [pkgs.nix];
                wantedBy = ["multi-user.target"];
                requires = ["nix-daemon.service"];
                script = "cat $ persistencePath /nix-store-db-dump nix-store --load-db";
               ;
              time.timeZone = nixpkgs.lib.mkDefault "Europe/Berlin";
              users.users.$ user  =  
                extraGroups = [ "wheel" "video" ];
                group = "user";
                isNormalUser = true;
                openssh.authorizedKeys.keys = [
                  "ssh-rsa REDACTED"
                ];
                password = "";
               ;
              users.users.root.password = "";
              users.groups.user =  ;
             )
          ];
         ;
    in  
      packages.$ system .default = nixosConfiguration.config.microvm.declaredRunner;
     ;

I start the microVM with a templated systemd user service:

[Unit]
Description=MicroVM for Haskell development
Requires=microvm-virtiofsd-persistent@.service
After=microvm-virtiofsd-persistent@.service
AssertFileNotEmpty=%h/.local/share/microvm/vms/%i/flake/flake.nix
[Service]
Type=forking
ExecStartPre=/usr/bin/sh -c "[ /nix/var/nix/db/db.sqlite -ot %h/.local/share/microvm/nix-store-db-dump ]   nix-store --dump-db >%h/.local/share/microvm/nix-store-db-dump"
ExecStartPre=ln -f -t %h/.local/share/microvm/vms/%i/persistent/ %h/.local/share/microvm/nix-store-db-dump
ExecStartPre=-%h/.local/state/nix/profile/bin/tmux new -s microvm -d
ExecStart=%h/.local/state/nix/profile/bin/tmux new-window -t microvm: -n "%i" "exec %h/.local/state/nix/profile/bin/nix run --impure %h/.local/share/microvm/vms/%i/flake"

The above service definition creates a dump of the hosts nix store db so that it can be imported in the guest. This is necessary so that the guest can actually use what is available in /nix/store. There is an effort for an overlayed nix store that would be preferable to this hack. Finally the microvm is started inside a tmux session named microvm . This way I can use the VM with SSH or through the console and also access the qemu console. And for completeness the virtiofsd service:

[Unit]
Description=serve host persistent folder for dev VM
AssertPathIsDirectory=%h/.local/share/microvm/vms/%i/persistent
[Service]
ExecStart=%h/.local/state/nix/profile/bin/virtiofsd \
 --socket-path=$ XDG_RUNTIME_DIR /microvm-%i-persistent \
 --shared-dir=%h/.local/share/microvm/vms/%i/persistent \
 --gid-map :995:%G:1: \
 --uid-map :1000:%U:1:

10 May 2024

Reproducible Builds: Reproducible Builds in April 2024

Welcome to the April 2024 report from the Reproducible Builds project! In our reports, we attempt to outline what we have been up to over the past month, as well as mentioning some of the important things happening more generally in software supply-chain security. As ever, if you are interested in contributing to the project, please visit our Contribute page on our website. Table of contents:

New `backseat-signed` tool to validate distributions source inputs kpcyrd announced a new tool called `backseat-signed`, after:
I figured out a somewhat straight-forward way to check if a given `git archive` output is cryptographically claimed to be the source input of a given binary package in either Arch Linux or Debian (or both).
Elaborating more in their announcement post, kpcyrd writes:
I believe this to be the reproducible source tarball thing some people have been asking about. As explained in the README, I believe reproducing autotools-generated tarballs isn t worth everybody s time and instead a distribution that claims to build from source should operate on VCS snapshots instead of tarballs with 25k lines of pre-generated shell-script.
Indeed, many distributions packages already build from VCS snapshots, and this trend is likely to accelerate in response to the xz incident. The announcement led to a lengthy discussion on our mailing list, as well as shorter followup thread from kpcyrd about bootstrapping Autotools projects.

NixOS is not reproducible Morten Linderud posted an post on his blog this month, provocatively titled, NixOS is not reproducible . Although quickly admitting that his title is indeed clickbait , Morten goes on to clarify the precise guarantees and promises that NixOS provides its users. Later in the most, Morten mentions that he was motivated to write the post because:
I have heavily invested my free-time on this topic since 2017, and met some of the accomplishments we have had with Doesn t NixOS solve this? for just as long and I thought it would be of peoples interest to clarify[.]

Certificate vulnerabilities in F-Droid s `fdroidserver` In early April, Fay Stegerman announced a certificate pinning bypass vulnerability and Proof of Concept (PoC) in the F-Droid `fdroidserver` tools for managing builds, indexes, updates, and deployments for F-Droid repositories to the `oss-security` mailing list.
We observed that embedding a v1 (JAR) signature file in an APK with `minSdk` >= 24 will be ignored by Android/apksigner, which only checks v2/v3 in that case. However, since `fdroidserver` checks v1 first, regardless of `minSdk`, and does not verify the signature, it will accept a fake certificate and see an incorrect certificate fingerprint. [ ] We also realised that the above mentioned discrepancy between `apksigner` and `androguard` (which `fdroidserver` uses to extract the v2/v3 certificates) can be abused here as well. [ ]
Later on in the month, Fay followed up with a second post detailing a third vulnerability and a script that could be used to scan for potentially affected `.apk` files and mentioned that, whilst upstream had acknowledged the vulnerability, they had not yet applied any ameliorating fixes.

Website updates There were a number of improvements made to our website this month, including Chris Lamb updating the archive page to recommend `-X` and unzipping with `TZ=UTC` [ ] and adding Maven, Gradle, JDK and Groovy examples to the `SOURCE_DATE_EPOCH` page [ ]. In addition Jan Zerebecki added a new `/contribute/opensuse/` page [ ] and Sertonix fixed the automatic RSS feed detection [ ][ ].

Reproducible Builds and Insights from an Independent Verifier for Arch Linux Joshua Drexel, Esther H nggi and Iy n M ndez Veiga of the School of Computer Science and Information Technology, Hochschule Luzern (HSLU) in Switzerland published a paper this month entitled Reproducible Builds and Insights from an Independent Verifier for Arch Linux. The paper establishes the context as follows:
Supply chain attacks have emerged as a prominent cybersecurity threat in recent years. Reproducible and bootstrappable builds have the potential to reduce such attacks significantly. In combination with independent, exhaustive and periodic source code audits, these measures can effectively eradicate compromises in the building process. In this paper we introduce both concepts, we analyze the achievements over the last ten years and explain the remaining challenges.
What is more, the paper aims to:
contribute to the reproducible builds effort by setting up a rebuilder and verifier instance to test the reproducibility of Arch Linux packages. Using the results from this instance, we uncover an unnoticed and security-relevant packaging issue affecting 16 packages related to Certbot [ ].
A PDF of the paper is available.

`libntlm` now releasing minimal source-only tarballs Simon Josefsson wrote on his blog this month that, going forward, the `libntlm` project will now be releasing what they call minimal source-only tarballs :
The XZUtils incident illustrate that tarballs with files that are not included in the git archive offer an opportunity to disguise malicious backdoors. [The] risk of hiding malware is not the only motivation to publish signed minimal source-only tarballs. With pre-generated content in tarballs, there is a risk that GNU/Linux distributions [ship] generated files coming from the tarball into the binary `.deb` or `.rpm` package file. Typically the person packaging the upstream project never realized that some installed artifacts was not re-built[.]
Simon s post goes into further details how this was achieved, and describes some potential caveats and counters some expected responses as well. A shorter version can be found in the announcement for the 1.8 release of `libntlm`.

Distribution work In Debian this month, Helmut Grohne filed a bug suggesting the removal of `dh-buildinfo`, a tool to generate and distribute `.buildinfo`-like files within binary packages. Note that this is distinct from the `.buildinfo` generation performed by `dpkg-genbuildinfo`. By contrast, the entirely optional `dh-buildinfo` generated a `debian/buildinfo` file that would be shipped within binary packages as `/usr/share/doc/package/buildinfo_$arch.gz`. Adrian Bunk recently asked about including source hashes in Debian s .buildinfo files, which prompted Guillem Jover to refresh some old patches to dpkg to make this possible, which revealed some quirks Vagrant Cascadian discovered when testing. In addition, 21 reviews of Debian packages were added, 22 were updated and 16 were removed this month adding to our knowledge about identified issues. A number issue types have been added, such as new `random_temporary_filenames_embedded_by_mesonpy` and `timestamps_added_by_librime` toolchain issues. In openSUSE, it was announced that their Factory distribution enabled bit-by-bit reproducible builds for almost all parts of the package. Previously, more parts needed to be ignored when comparing package files, but now only the signature needs to be deleted. In addition, Bernhard M. Wiedemann published `theunreproduciblepackage` as a proper `.rpm` package which it allows to better test tools intended to debug reproducibility. Furthermore, it was announced that Bernhard s work on a 100% reproducible openSUSE-based distribution will be funded by NLnet. He also posted another monthly report for his reproducibility work in openSUSE. In GNU Guix, Janneke Nieuwenhuizen submitted a patch set for creating a reproducible source tarball for Guix. That is to say, ensuring that `make dist` is reproducible when run from Git. [ ] Lastly, in Fedora, a new wiki page was created to propose a change to the distribution. Titled Changes/ReproduciblePackageBuilds , the page summarises itself as a proposal whereby A post-build cleanup is integrated into the RPM build process so that common causes of build irreproducibility in packages are removed, making most of Fedora packages reproducible.

Mailing list news On our mailing list this month:

Continuing a thread started in March 2024 about the Arch Linux minimal container now being 100% reproducible, John Gilmore followed up with a post about the practical and philosophical distinctions of local vs. remote storage of the various artifacts needed to build packages.

Chris Lamb asked the list which conferences readers are attending these days: After peak Covid and other industry-wide changes, conferences are no longer the must attend events they previously were especially in the area of software supply-chain security. In rough, practical terms, it seems harder to justify conference travel today than it did in mid-2019. The thread generated a number of responses which would be of interest to anyone planning travel in Q3 and Q4 of 2024.

James Addison wrote to the list about a quirk in Git related to its `core.autocrlf` functionality, thus helpfully passing on a slightly off-topic and perhaps not of direct relevance to anyone on the list today note that might still be the kind of issue that is useful to be aware of if-and-when puzzling over unexpected git content / checksum issues (situations that I do expect people on this list encounter from time-to-time) .

diffoscope diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made a number of changes such as uploading versions `263`, `264` and `265` to Debian and made the following additional changes:

Don t crash on invalid `.zip` files, even if we encounter their badness halfway through the file and not at the time of their initial opening. [ ]

Prevent `odt2txt` tests from always being skipped due to an (impossibly) new version requirement. [ ]

Avoid parens-in-parens in test skipping messages. [ ]

Ensure that tests with `>=`-style version constraints actually print the tool name. [ ]

In addition, Fay Stegerman fixed a crash when there are (invalid) duplicate entries in `.zip` which was originally reported in Debian bug #1068705). [ ] Fay also added a user-visible note to a diff when there are duplicate entries in ZIP files [ ]. Lastly, Vagrant Cascadian added an external tool pointer for the `zipdetails` tool under GNU Guix [ ] and proposed updates to diffoscope in Guix as well [ ] which were merged as [264] [265], fixed a regression in test coverage and increased verbosity of the test suite[ ].

Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

Chris Lamb:

#1068173 filed against `pg-gvm`.

#1068176 filed against `goldendict-ng`.

#1068372 filed against `grokevt`.

#1068374 filed against `ttconv`.

#1068375 filed against `ludevit`.

#1068795 filed against `pympress`.

#1069168 filed against `sagemath-database-conway-polynomials`.

#1069169 filed against `gap-polymaking`.

#1069663 filed against `dub`.

#1069709 filed against `dpb`.

#1069784 filed against `python-itemloaders`.

#1069822 filed against `python-gvm`.

Bernhard M. Wiedemann:

`metis` (fix build with nocheck)

`musique` (fix a date-related issue)

`orthanc-volview` (fix an issue with mtimes and sorting)

`go1.13`, `go1.14`, `go1.15` (fix a parallelism-related issue)

`postfish` (disable compile-time benchmarking)

`geany/glfw` (toolchain, random)

`edk2/ovmf/tianocore` (with Joey Li: fix a date-related issue)

`dlib` (report an issue with compile-time-CPU-detection)

`lua-lmod` (fix a date-related issue)

`gitui` (fix a date-related issue)

`openssl-3` (report an issue with random output)

`gcc14` (FTBFS-2038)

`nebula` (FTBFS-2027-11-11)

Jan Zerebecki:

rpm (Support reproducible automatic rebuilds, etc.)

openSUSE-release-tools (Create changelog for generated package sources for `SOURCE_DATE_EPOCH`)

pesign-obs-integration (Create changelog for generated package sources for `SOURCE_DATE_EPOCH`)

openSUSE post-build-checks (Set `SOURCE_DATE_EPOCH`)

obs-build (Fix changelog timezone handling)

obs-service-tar_scm (When generating changelog from Git, create the file if it does not exist.)

Thomas Goirand:

`oslo.messaging` (fix a hostname-related issue)

reprotest reprotest is our tool for building the same source code twice in different environments and then checking the binaries produced by each build for any differences. This month, reprotest version `0.7.27` was uploaded to Debian unstable) by Vagrant Cascadian who made the following additional changes:

Enable specific number of CPUs using `--vary=num_cpus.cpus=X`. [ ]

Consistently use 398 days for time variation, rather than choosing randomly each time. [ ]

Disable builds of `arch:any` packages. [ ]

Update the description for the `build_path.path` option in `README.rst`. [ ]

Update escape sequences for compatibility with Python 3.12. (#1068853). [ ]

Remove the generic upstream signing-key [ ] and update the packages signing key with the currently active team members [ ].

Update the packaging `Standards-Version` to 4.7.0. [ ]

In addition, Holger Levsen fixed some spelling errors detected by the `spellintian` tool [ ] and Vagrant Cascadian updated reprotest in GNU Guix to 0.7.27.

Reproducibility testing framework The Reproducible Builds project operates a comprehensive testing framework running primarily at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In April, an enormous number of changes were made by Holger Levsen:

Debian-related changes:

Adjust for changed internal IP addresses at Codethink. [ ]

Automatically cleanup failed diffoscope user services if there are too many failures. [ ][ ]

Configure two new nodes at infomanik.cloud. [ ][ ]

Schedule Debian experimental even less. [ ][ ]

Breakage detection:

Exclude currently building packages from breakage detection. [ ]

Be more noisy if diffoscope crashes. [ ]

Health check: provide clickable URLs in jenkins job log for failed pkg builds due to diffoscope crashes. [ ]

Limit graph to about the last 100 days of breakages only. [ ]

Fix all found files with bad permissions. [ ]

Prepare dealing with diffoscope timeouts. [ ]

Detect more cases of failure to debootstrap base system. [ ]

Include timestamps of failed job runs. [ ]

Documentation updates:

Document how to access arm64 nodes at Codethink. [ ]

Document how to use infomaniak.cloud. [ ]

Drop notes about long stalled LeMaker HiKey960 boards sponsored by HPE and hosted at ETH. [ ]

Mention `osuosl4` and `osuosl5` and explain their usage. [ ]

Mention that some packages are built differently. [ ][ ]

Improve language in a comment. [ ]

Add more notes how to query resource usage from `infomaniak.cloud`. [ ]

Node maintenance:

Add `ionos4` and `ionos14` to THANKS. [ ][ ][ ][ ][ ]

Deprecate Squid on `ionos1` and `ionos10`. [ ]

Drop obsolete script to powercycle `arm64` architecture nodes. [ ]

Update `system_health_check` for new proxy nodes. [ ]

Misc changes:

Make the `update_jdn.sh` script more robust. [ ][ ]

Update my SSH public key. [ ]

In addition, Mattia Rizzolo added some new host details. [ ]

If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

IRC: `#reproducible-builds` on `irc.oftc.net`.

Twitter: @ReproBuilds

Mastodon: @reproducible_builds@fosstodon.org

Mailing list: `rb-general@lists.reproducible-builds.org`

1 April 2024

Colin Watson: Free software activity in March 2024

My Debian contributions this month were all sponsored by Freexian.

Python team:
- I updated zope.testrunner to 6.4.
- I fixed a build failure in celery-haystack-ng, which included an upstream change to stop using d2to1.
- I backported an upstream change to fix a build failure in python-json-log-formatter.
- I updated python-typing-extensions to 4.10.0 to fix a build failure.
- I updated wcwidth to 0.2.13 to fix a build failure, which included rewriting the Debian patches to update-tables.
- I updated jsonpickle to 3.0.3 to fix a build failure. As part of this I also needed to fix its tests for new timezone name policies.
man-db:
- I updated man(1) with updated information about groff.
- I documented italic support in man(1).
- I started on a 2.12.1 release (currently waiting for translations).
openssh:
- I added an sshd@.service template to help newer versions of systemd make containers and VMs SSH-accessible over AF_VSOCK sockets.
- I upgraded openssh to 9.7p1. As part of this I noticed a small bug in the PuTTY version detection in its regression tests.
- I helped with the 64-bit time_t transition by applying a patch from Simon McVittie to temporarily disable ssh-askpass-gnome on the affected architectures.
- I re-enabled interoperability tests with Twisted Conch.
- Following the xz-utils backdoor, I spent some time testing OpenSSH upstream s proposed inline systemd notification patch and suggested a small adjustment.
debconf:
- I fixed dpkg-preconfigure to not produce errors when run from a non-existing directory.
parted:
- I applied a patch from Helmut Grohne to move files to /usr.
I fixed an off-by-one error in neovim.
I fixed a number of -Werror=implicit-function-declaration failures (db1-compat, kali, knews, openssh-ssh1, trn4, vigor).
I fixed a build failure in libsdl-perl, including a reference-counting fix contributed upstream.
I suggested an rrsync change to make it easier to use it with rsbackup.

13 March 2024

Freexian Collaborators: Debian Contributions: Upcoming Improvements to Salsa CI, /usr-move, packaging simplemonitor, and more! (by Utkarsh Gupta)

Contributing to Debian is part of Freexian s mission. This article covers the latest achievements of Freexian and their collaborators. All of this is made possible by organizations subscribing to our Long Term Support contracts and consulting services.

/usr-move, by Helmut Grohne Much of the work was spent on handling interaction with time time64 transition and sending patches for mitigating fallout. The set of packages relevant to `debootstrap` is mostly converted and the patches for `glibc` and `base-files` have been refined due to feedback from the upload to Ubuntu noble. Beyond this, he sent patches for all remaining packages that cannot move their files with `dh-sequence-movetousr` and packages using `dpkg-divert` in ways that `dumat` would not recognize.

Upcoming improvements to Salsa CI, by Santiago Ruano Rinc n Last month, Santiago Ruano Rinc n started the work on integrating sbuild into the Salsa CI pipeline. Initially, Santiago used sbuild with the `unshare` chroot mode. However, after discussion with josch, jochensp and helmut (thanks to them!), it turns out that the unshare mode is not the most suitable for the pipeline, since the level of isolation it provides is not needed, and some test suites would fail (eg: krb5). Additionally, one of the requirements of the build job is the use of ccache, since it is needed by some C/C++ large projects to reduce the compilation time. In the preliminary work with unshare last month, it was not possible to make ccache to work. Finally, Santiago changed the chroot mode, and now has a couple of POC (cf: 1 and 2) that rely on the `schroot` and `sudo`, respectively. And the good news is that ccache is successfully used by sbuild with schroot! The image here comes from an example of building `grep`. At the end of the build, `ccache -s` shows the statistics of the cache that it used, and so a little more than half of the calls of that job were cacheable. The most important pieces are in place to finish the integration of sbuild into the pipeline. Other than that, Santiago also reviewed the very useful merge request !346, made by IOhannes zm lnig to autodetect the release from debian/changelog. As agreed with IOhannes, Santiago is preparing a merge request to include the release autodetection use case in the very own Salsa CI s CI.

Packaging simplemonitor, by Carles Pina i Estany Carles started using simplemonitor in 2017, opened a WNPP bug in 2022 and started packaging simplemonitor dependencies in October 2023. After packaging five direct and indirect dependencies, Carles finally uploaded simplemonitor to unstable in February. During the packaging of simplemonitor, Carles reported a few issues to upstream. Some of these were to make the simplemonitor package build and run tests reproducibly. A reproducibility issue was reprotest overriding the timezone, which broke simplemonitor s tests. There have been discussions on resolving this upstream in simplemonitor and in reprotest, too. Carles also started upgrading or improving some of simplemonitor s dependencies.

Miscellaneous contributions

Stefano Rivera spent some time doing admin on debian.social infrastructure. Including dealing with a spike of abuse on the Jitsi server.

Stefano started to prepare a new release of dh-python, including cleaning out a lot of old Python 2.x related code. Thanks to Niels Thykier (outside Freexian) for spear-heading this work.

DebConf 24 planning is beginning. Stefano discussed venues and finances with the local team and remotely supported a site-visit by Nattie (outside Freexian).

Also in the DebConf 24 context, Santiago took part in discussions and preparations related to the Content Team.

A JIT bug was reported against pypy3 in Debian Bookworm. Stefano bisected the upstream history to find the patch (it was already resolved upstream) and released an update to pypy3 in bookworm.

Enrico participated in /usr-merge discussions with Helmut.

Colin Watson backported a python-channels-redis fix to bookworm, rediscovered while working on debusine.

Colin dug into a cluster of celery build failures and tracked the hardest bit down to a Python 3.12 regression, now fixed in unstable. celery should be back in testing once the 64-bit time_t migration is out of the way.

Thorsten Alteholz uploaded a new upstream version of cpdb-libs. Unfortunately upstream changed the naming of their release tags, so updating the watch file was a bit demanding. Anyway this version 2.0 is a huge step towards introduction of the new Common Print Dialog Backends.

Helmut send patches for 48 cross build failures.

Helmut changed debvm to use mkfs.ext4 instead of genext2fs.

Helmut sent a debci MR for improving collector robustness.

In preparation for DebConf 25, Santiago worked on the Brest Bid.

7 February 2024

Reproducible Builds: Reproducible Builds in January 2024

Welcome to the January 2024 report from the Reproducible Builds project. In these reports we outline the most important things that we have been up to over the past month. If you are interested in contributing to the project, please visit our Contribute page on our website.

How we executed a critical supply chain attack on PyTorch John Stawinski and Adnan Khan published a lengthy blog post detailing how they executed a supply-chain attack against PyTorch, a popular machine learning platform used by titans like Google, Meta, Boeing, and Lockheed Martin :
Our exploit path resulted in the ability to upload malicious PyTorch releases to GitHub, upload releases to [Amazon Web Services], potentially add code to the main repository branch, backdoor PyTorch dependencies the list goes on. In short, it was bad. Quite bad.
The attack pivoted on PyTorch s use of self-hosted runners as well as submitting a pull request to address a trivial typo in the project s `README` file to gain access to repository secrets and API keys that could subsequently be used for malicious purposes.

New Arch Linux forensic filesystem tool On our mailing list this month, long-time Reproducible Builds developer kpcyrd announced a new tool designed to forensically analyse Arch Linux filesystem images. Called `archlinux-userland-fs-cmp`, the tool is supposed to be used from a rescue image (any Linux) with an Arch install mounted to, [for example], `/mnt`. Crucially, however, at no point is any file from the mounted filesystem eval d or otherwise executed. Parsers are written in a memory safe language. More information about the tool can be found on their announcement message, as well as on the tool s homepage. A GIF of the tool in action is also available.

Issues with our `SOURCE_DATE_EPOCH` code? Chris Lamb started a thread on our mailing list summarising some potential problems with the source code snippet the Reproducible Builds project has been using to parse the `SOURCE_DATE_EPOCH` environment variable:
I m not 100% sure who originally wrote this code, but it was probably sometime in the ~2015 era, and it must be in a huge number of codebases by now. Anyway, Alejandro Colomar was working on the shadow security tool and pinged me regarding some potential issues with the code. You can see this conversation here.
Chris ends his message with a request that those with intimate or low-level knowledge of `time_t`, C types, overflows and the various parsing libraries in the C standard library (etc.) contribute with further info.

Distribution updates In Debian this month, Roland Clobus posted another detailed update of the status of reproducible ISO images on our mailing list. In particular, Roland helpfully summarised that all major desktops build reproducibly with bullseye, bookworm, trixie and sid provided they are built for a second time within the same DAK run (i.e. [within] 6 hours) . Additionally 7 of the 8 bookworm images from the official download link build reproducibly at any later time. In addition to this, three reviews of Debian packages were added, 17 were updated and 15 were removed this month adding to our knowledge about identified issues. Elsewhere, Bernhard posted another monthly update for his work elsewhere in openSUSE.

Community updates There were made a number of improvements to our website, including Bernhard M. Wiedemann fixing a number of typos of the term nondeterministic . [ ] and Jan Zerebecki adding a substantial and highly welcome section to our page about `SOURCE_DATE_EPOCH` to document its interaction with distribution rebuilds. [ ].
diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made a number of changes such as uploading versions `254` and `255` to Debian but focusing on triaging and/or merging code from other contributors. This included adding support for comparing eXtensible ARchive (.XAR/.PKG) files courtesy of Seth Michael Larson [ ][ ], as well considerable work from Vekhir in order to fix compatibility between various and subtle incompatible versions of the progressbar libraries in Python [ ][ ][ ][ ]. Thanks!

Reproducibility testing framework The Reproducible Builds project operates a comprehensive testing framework (available at tests.reproducible-builds.org) in order to check packages and other artifacts for reproducibility. In January, a number of changes were made by Holger Levsen:

Debian-related changes:

Reduce the number of `arm64` architecture workers from 24 to 16. [ ]

Use diffoscope from the Debian release being tested again. [ ]

Improve the handling when killing unwanted processes [ ][ ][ ] and be more verbose about it, too [ ].

Don t mark a job as failed if process marked as to-be-killed is already gone. [ ]

Display the architecture of builds that have been running for more than 48 hours. [ ]

Reboot `arm64` nodes when they hit an OOM (out of memory) state. [ ]

Package rescheduling changes:

Reduce IRC notifications to 1 when rescheduling due to package status changes. [ ]

Correctly set `SUDO_USER` when rescheduling packages. [ ]

Automatically reschedule packages regressing to FTBFS (build failure) or FTBR (build success, but unreproducible). [ ]

OpenWrt-related changes:

Install the `python3-dev` and `python3-pyelftools` packages as they are now needed for the `sunxi` target. [ ][ ]

Also install the `libpam0g-dev` which is needed by some OpenWrt hardware targets. [ ]

Misc:

As it s January, set the `real_year` variable to 2024 [ ] and bump various copyright years as well [ ].

Fix a large (!) number of spelling mistakes in various scripts. [ ][ ][ ]

Prevent Squid and Systemd processes from being killed by the kernel s OOM killer. [ ]

Install the `iptables` tool everywhere, else our custom `rc.local` script fails. [ ]

Cleanup the `/srv/workspace/pbuilder` directory on boot. [ ]

Automatically restart Squid if it fails. [ ]

Limit the execution of `chroot-installation` jobs to a maximum of 4 concurrent runs. [ ][ ]

Significant amounts of node maintenance was performed by Holger Levsen (eg. [ ][ ][ ][ ][ ][ ][ ] etc.) and Vagrant Cascadian (eg. [ ][ ][ ][ ][ ][ ][ ][ ]). Indeed, Vagrant Cascadian handled an extended power outage for the network running the Debian `armhf` architecture test infrastructure. This provided the incentive to replace the UPS batteries and consolidate infrastructure to reduce future UPS load. [ ] Elsewhere in our infrastructure, however, Holger Levsen also adjusted the email configuration for `@reproducible-builds.org` to deal with a new SMTP email attack. [ ]

Upstream patches The Reproducible Builds project tries to detects, dissects and fix as many (currently) unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

Bernhard M. Wiedemann:

`cython` (nondeterminstic path issue)

`deluge` (issue with modification time of `.egg` file)

`gap-ferret`, `gap-semigroups` & `gap-simpcomp` (nondeterministic `config.log` file)

`grpc` (filesystem ordering issue )

`hub` (random)

`kubernetes1.22` & `kubernetes1.23` (sort-related issue)

`kubernetes1.24` & `kubernetes1.25` (`go -trimpath` vs random issue)

`libjcat` (drop test files with random bytes)

`luajit` (Use new `d` option for deterministic bytecode output)

`meson` [ ][ ] (sort the results from Python filesystem call)

`python-rjsmin` (drop GCC instrumentation artifacts)

`qt6-virtualkeyboard+others` (bug parallelism/race)

`SoapySDR` (parallelism-related issue)

`systemd` (sorting problem)

`warewulf` (CPIO modification time issue, etc.)

Chris Lamb:

#1060254 filed against `mumble`.

James Addison:

`guake` ( Schroedinger file due to race condition)

`qhelpgenerator-qt5` (timezone localization; fix also merged upstream for QT6)

`sphinx` (search index `doctitle` sorting)

Separate to this, Vagrant Cascadian followed up with the relevant maintainers when reproducibility fixes were not included in newly-uploaded versions of the `mm-common` package in Debian this was quickly fixed, however. [ ]

If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

IRC: `#reproducible-builds` on `irc.oftc.net`.

Mailing list: `rb-general@lists.reproducible-builds.org`

Mastodon: @reproducible_builds

Twitter: @ReproBuilds

5 January 2024

Valhalla's Things: Random Sashiko + Crazy Quilt Pocket

A 18th century pocket in black jeans with a random pattern of pink running stitches forming squares and other shapes. The unfinished edges of the pieces of jeans can be seen, running more or less diagonally.

Lately I ve seen people on the internet talking about victorian crazy quilting. Years ago I had watched a Numberphile video about Hitomezashi Stitch Patterns based on numbers, words or randomness. Few weeks ago I had cut some fabric piece out of an old pair of jeans and I had a lot of scraps that were too small to do anything useful on their own. It easy to see where this can go, right? The wrong side of a pocket piece, showing a light coloured fabric with a grid drawn in pencil, a line of small stitches all around the edges and a mess of thread ends left hanging.

The wrong side of a pocket piece, showing a light coloured fabric with a grid drawn in pencil, a line of small stitches all around the edges and a mess of thread ends left hanging.

I cut a pocket shape out of old garment mockups (this required some piecing), drew a square grid, arranged scraps of jeans to cover the other side, kept everything together with a lot of pins, carefully avoided basting anything, and started covering everything in sashiko / hitomezashi stitches, starting each line with a stitch on the front or the back of the work based on the result of:

import random
random.choice(["front", "back"])

The wrong side of the other pocket piece, with just three lines of stitching and a piece of paper to mark the pattern. There are bits of jeans peeking out of the sides.

For the second piece I tried to use a piece of paper with the square grid instead of drawing it on the fabric: it worked, mostly, I would not do it again as removing the paper was more of a hassle than drawing the lines in the first place. I suspected it, but had to try it anyway. The front of the pocket seen from the wrong side, with a machine seam around the lit, whose end has been cut in a triangle so that it can be turned.

The front of the pocket seen from the wrong side, with a machine seam around the lit, whose end has been cut in a triangle so that it can be turned.

Then I added a lining from some plain black cotton from the stash; for the slit I put the lining on the front right sides together, sewn at 2 mm from the marked slit, cut it, turned the lining to the back side, pressed and then topstitched as close as possible to the slit from the front. The finished pocket attached to a belt made from the waistband of a pair of jeans (with button, buttonhole and belt loops still attached) whose raw edges (left when unpicking away the jeans) have been sewn shut by hand.

The finished pocket attached to a belt made from the waistband of a pair of jeans (with button, buttonhole and belt loops still attached) whose raw edges (left when unpicking away the jeans) have been sewn shut by hand.

I bound everything with bias tape, adding herringbone tape loops at the top to hang it from a belt (such as one made from the waistband of one of the donor pair of jeans) and that was it. The back of the pocket, showing another random pattern in two different shades of pink for the vertical and horizontal lines of stitching.

The back of the pocket, showing another random pattern in two different shades of pink for the vertical and horizontal lines of stitching.

I like the way the result feels; maybe it s a bit too stiff for a pocket, but I can see it work very well for a bigger bag, and maybe even a jacket or some other outer garment.

31 December 2023

Chris Lamb: Favourites of 2023

This post should have marked the beginning of my yearly roundups of the favourite books and movies I read and watched in 2023. However, due to coming down with a nasty bout of flu recently and other sundry commitments, I wasn't able to undertake writing the necessary four or five blog posts In lieu of this, however, I will simply present my (unordered and unadorned) highlights for now. Do get in touch if this (or any of my previous posts) have spurred you into picking something up yourself

Books

Films Recent releases

The Blue Caftan (Maryam Touzani, 2022)
The Eight Mountains (Felix van Groeningen & Charlotte Vandermeersch, 2022)
Evil Does Not Exist (Ryusuke Hamaguchi, 2023)
Killers of the Flower Moon (Martin Scorcese, 2023)
Monster (Hirokazu Kore-eda, 2023)
Passages (Ira Sachs, 2023)
Poor Things (Yorgos Lanthimos, 2023)
The Tuba Thieves (Alison O Daniel, 2023)
Theater Camp (Molly Gordon and Nick Lieberman, 2023)
T R (Todd Field, 2022)

Unenjoyable experiences included Alejandro G mez Monteverde's Sound of Freedom (2023), Alex Garland's Men (2022) and Steven Spielberg's The Fabelmans (2022).
Older releases (Films released before 2022, and not including rewatches from previous years.)

Brief Encounter (David Lean, 1945)
Clouds of Sils Maria (Olivier Assayas, 2014)
Daisy Miller (Peter Bogdanovich, 1974)
First Reformed (Paul Schrader, 2017)
Forbidden Games (Ren Cl ment, 1952)
La Noire de... (Ousmane Semb ne, 1966)
The Queen of Spades (Thorold Dickinson, 1949)
The River (Jean Renoir, 1951)
Topsy-Turvy (Mike Leigh, 1999)
Le Trou (Jacques Becker, 1960)

Distinctly unenjoyable watches included Ocean's Eleven (1960), El Topo (1970), L olo (1992), Hotel Mumbai (2018), Bulworth (1998) and and The Big Red One (1980).

12 December 2023

Raju Devidas: Nextcloud AIO install with docker-compose and nginx reverse proxy

Nextcloud is a popular self-hosted solution for file sync and share as well as cloud apps such as document editing, chat and talk, calendar, photo gallery etc. This guide will walk you through setting up Nextcloud AIO using Docker Compose. This blog post would not be possible without immense help from Sahil Dhiman a.k.a. sahilisterThere are various ways in which the installation could be done, in our setup here are the pre-requisites.

A server with docker-compose installed
An existing setup of nginx reverse proxy
A domain name pointing to your server.

Step 1 : The docker-compose file for nextcloud AIOThe original compose.yml file is present in nextcloud AIO&aposs git repo here . By taking a reference of that file, we have own compose.yml here.

services:
  nextcloud-aio-mastercontainer:
    image: nextcloud/all-in-one:latest
    init: true
    restart: always
    container_name: nextcloud-aio-mastercontainer # This line is not allowed to be changed as otherwise AIO will not work correctly
    volumes:
      - nextcloud_aio_mastercontainer:/mnt/docker-aio-config # This line is not allowed to be changed as otherwise the built-in backup solution will not work
      - /var/run/docker.sock:/var/run/docker.sock:ro # May be changed on macOS, Windows or docker rootless. See the applicable documentation. If adjusting, don&apost forget to also set &aposWATCHTOWER_DOCKER_SOCKET_PATH&apos!
    ports:
      - 8080:8080
    environment: # Is needed when using any of the options below
      # - AIO_DISABLE_BACKUP_SECTION=false # Setting this to true allows to hide the backup section in the AIO interface. See https://github.com/nextcloud/all-in-one#how-to-disable-the-backup-section
      - APACHE_PORT=32323 # Is needed when running behind a web server or reverse proxy (like Apache, Nginx, Cloudflare Tunnel and else). See https://github.com/nextcloud/all-in-one/blob/main/reverse-proxy.md
      - APACHE_IP_BINDING=127.0.0.1 # Should be set when running behind a web server or reverse proxy (like Apache, Nginx, Cloudflare Tunnel and else) that is running on the same host. See https://github.com/nextcloud/all-in-one/blob/main/reverse-proxy.md
      # - BORG_RETENTION_POLICY=--keep-within=7d --keep-weekly=4 --keep-monthly=6 # Allows to adjust borgs retention policy. See https://github.com/nextcloud/all-in-one#how-to-adjust-borgs-retention-policy
      # - COLLABORA_SECCOMP_DISABLED=false # Setting this to true allows to disable Collabora&aposs Seccomp feature. See https://github.com/nextcloud/all-in-one#how-to-disable-collaboras-seccomp-feature
      - NEXTCLOUD_DATADIR=/opt/docker/cloud.raju.dev/nextcloud # Allows to set the host directory for Nextcloud&aposs datadir.   Warning: do not set or adjust this value after the initial Nextcloud installation is done! See https://github.com/nextcloud/all-in-one#how-to-change-the-default-location-of-nextclouds-datadir
      # - NEXTCLOUD_MOUNT=/mnt/ # Allows the Nextcloud container to access the chosen directory on the host. See https://github.com/nextcloud/all-in-one#how-to-allow-the-nextcloud-container-to-access-directories-on-the-host
      # - NEXTCLOUD_UPLOAD_LIMIT=10G # Can be adjusted if you need more. See https://github.com/nextcloud/all-in-one#how-to-adjust-the-upload-limit-for-nextcloud
      # - NEXTCLOUD_MAX_TIME=3600 # Can be adjusted if you need more. See https://github.com/nextcloud/all-in-one#how-to-adjust-the-max-execution-time-for-nextcloud
      # - NEXTCLOUD_MEMORY_LIMIT=512M # Can be adjusted if you need more. See https://github.com/nextcloud/all-in-one#how-to-adjust-the-php-memory-limit-for-nextcloud
      # - NEXTCLOUD_TRUSTED_CACERTS_DIR=/path/to/my/cacerts # CA certificates in this directory will be trusted by the OS of the nexcloud container (Useful e.g. for LDAPS) See See https://github.com/nextcloud/all-in-one#how-to-trust-user-defined-certification-authorities-ca
      # - NEXTCLOUD_STARTUP_APPS=deck twofactor_totp tasks calendar contacts notes # Allows to modify the Nextcloud apps that are installed on starting AIO the first time. See https://github.com/nextcloud/all-in-one#how-to-change-the-nextcloud-apps-that-are-installed-on-the-first-startup
      # - NEXTCLOUD_ADDITIONAL_APKS=imagemagick # This allows to add additional packages to the Nextcloud container permanently. Default is imagemagick but can be overwritten by modifying this value. See https://github.com/nextcloud/all-in-one#how-to-add-os-packages-permanently-to-the-nextcloud-container
      # - NEXTCLOUD_ADDITIONAL_PHP_EXTENSIONS=imagick # This allows to add additional php extensions to the Nextcloud container permanently. Default is imagick but can be overwritten by modifying this value. See https://github.com/nextcloud/all-in-one#how-to-add-php-extensions-permanently-to-the-nextcloud-container
      # - NEXTCLOUD_ENABLE_DRI_DEVICE=true # This allows to enable the /dev/dri device in the Nextcloud container.   Warning: this only works if the &apos/dev/dri&apos device is present on the host! If it should not exist on your host, don&apost set this to true as otherwise the Nextcloud container will fail to start! See https://github.com/nextcloud/all-in-one#how-to-enable-hardware-transcoding-for-nextcloud
      # - NEXTCLOUD_KEEP_DISABLED_APPS=false # Setting this to true will keep Nextcloud apps that are disabled in the AIO interface and not uninstall them if they should be installed. See https://github.com/nextcloud/all-in-one#how-to-keep-disabled-apps
      # - TALK_PORT=3478 # This allows to adjust the port that the talk container is using. See https://github.com/nextcloud/all-in-one#how-to-adjust-the-talk-port
      # - WATCHTOWER_DOCKER_SOCKET_PATH=/var/run/docker.sock # Needs to be specified if the docker socket on the host is not located in the default &apos/var/run/docker.sock&apos. Otherwise mastercontainer updates will fail. For macos it needs to be &apos/var/run/docker.sock&apos
    # networks: # Is needed when you want to create the nextcloud-aio network with ipv6-support using this file, see the network config at the bottom of the file
      # - nextcloud-aio # Is needed when you want to create the nextcloud-aio network with ipv6-support using this file, see the network config at the bottom of the file
      # - SKIP_DOMAIN_VALIDATION=true
    # # Uncomment the following line when using SELinux
    # security_opt: ["label:disable"]
volumes: # If you want to store the data on a different drive, see https://github.com/nextcloud/all-in-one#how-to-store-the-filesinstallation-on-a-separate-drive
  nextcloud_aio_mastercontainer:
    name: nextcloud_aio_mastercontainer # This line is not allowed to be changed as otherwise the built-in backup solution will not work

I have not removed many of the commented options in the compose file, for a possibility of me using them in the future.If you want a smaller cleaner compose with the extra options, you can refer to

services:
  nextcloud-aio-mastercontainer:
    image: nextcloud/all-in-one:latest
    init: true
    restart: always
    container_name: nextcloud-aio-mastercontainer
    volumes:
      - nextcloud_aio_mastercontainer:/mnt/docker-aio-config
      - /var/run/docker.sock:/var/run/docker.sock:ro
    ports:
      - 8080:8080
    environment:
      - APACHE_PORT=32323
      - APACHE_IP_BINDING=127.0.0.1
      - NEXTCLOUD_DATADIR=/opt/docker/nextcloud
volumes:
  nextcloud_aio_mastercontainer:
    name: nextcloud_aio_mastercontainer

I am using a separate directory to store nextcloud data. As per nextcloud documentation you should be using a separate partition if you want to use this feature, however I did not have that option on my server, so I used a separate directory instead. Also we use a custom port on which nextcloud listens for operations, we have set it up as 32323 above, but you can use any in the permissible port range. The 8080 port is used the setup the AIO management interface. Both 8080 and the APACHE_PORT do not need to be open on the host machine, as we will be using reverse proxy setup with nginx to direct requests. once you have your preferred compose.yml file, you can start the containers using

$ docker-compose -f compose.yml up -d 
Creating network "clouddev_default" with the default driver
Creating volume "nextcloud_aio_mastercontainer" with default driver
Creating nextcloud-aio-mastercontainer ... done

once your container&aposs are running, we can do the nginx setup.

Step 2: Configuring nginx reverse proxy for our domain on host. A reference nginx configuration for nextcloud AIO is given in the nextcloud git repository here . You can modify the configuration file according to your needs and setup. Here is configuration that we are using


map $http_upgrade $connection_upgrade  
    default upgrade;
    &apos&apos close;
 
server  
    listen 80;
    #listen [::]:80;            # comment to disable IPv6
    if ($scheme = "http")  
        return 301 https://$host$request_uri;
     
    listen 443 ssl http2;      # for nginx versions below v1.25.1
    #listen [::]:443 ssl http2; # for nginx versions below v1.25.1 - comment to disable IPv6
    # listen 443 ssl;      # for nginx v1.25.1+
    # listen [::]:443 ssl; # for nginx v1.25.1+ - keep comment to disable IPv6
    # http2 on;                                 # uncomment to enable HTTP/2        - supported on nginx v1.25.1+
    # http3 on;                                 # uncomment to enable HTTP/3 / QUIC - supported on nginx v1.25.0+
    # quic_retry on;                            # uncomment to enable HTTP/3 / QUIC - supported on nginx v1.25.0+
    # add_header Alt-Svc &aposh3=":443"; ma=86400&apos; # uncomment to enable HTTP/3 / QUIC - supported on nginx v1.25.0+
    # listen 443 quic reuseport;       # uncomment to enable HTTP/3 / QUIC - supported on nginx v1.25.0+ - please remove "reuseport" if there is already another quic listener on port 443 with enabled reuseport
    # listen [::]:443 quic reuseport;  # uncomment to enable HTTP/3 / QUIC - supported on nginx v1.25.0+ - please remove "reuseport" if there is already another quic listener on port 443 with enabled reuseport - keep comment to disable IPv6
    server_name cloud.example.com;
    location /  
        proxy_pass http://127.0.0.1:32323$request_uri;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Forwarded-Scheme $scheme;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Accept-Encoding "";
        proxy_set_header Host $host;
    
        client_body_buffer_size 512k;
        proxy_read_timeout 86400s;
        client_max_body_size 0;
        # Websocket
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
     
    ssl_certificate /etc/letsencrypt/live/cloud.example.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/cloud.example.com/privkey.pem; # managed by Certbot
    ssl_session_timeout 1d;
    ssl_session_cache shared:MozSSL:10m; # about 40000 sessions
    ssl_session_tickets off;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305;
    ssl_prefer_server_ciphers on;
    # Optional settings:
    # OCSP stapling
    # ssl_stapling on;
    # ssl_stapling_verify on;
    # ssl_trusted_certificate /etc/letsencrypt/live/<your-nc-domain>/chain.pem;
    # replace with the IP address of your resolver
    # resolver 127.0.0.1; # needed for oscp stapling: e.g. use 94.140.15.15 for adguard / 1.1.1.1 for cloudflared or 8.8.8.8 for google - you can use the same nameserver as listed in your /etc/resolv.conf file

Please note that you need to have valid SSL certificates for your domain for this configuration to work. Steps on getting valid SSL certificates for your domain are beyond the scope of this article. You can give a web search on getting SSL certificates with letsencrypt and you will get several resources on that, or may write a blog post on it separately in the future.once your configuration for nginx is done, you can test the nginx configuration using

$ sudo nginx -t 
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

and then reload nginx with

$ sudo nginx -s reload

Step 3: Setup of Nextcloud AIO from the browser.To setup nextcloud AIO, we need to access it using the web browser on URL of our domain.tld:8080, however we do not want to open the 8080 port publicly to do this, so to complete the setup, here is a neat hack from sahilister

ssh -L 8080:127.0.0.1:8080 username:<server-ip>

you can bind the 8080 port of your server to the 8080 of your localhost using Unix socket forwarding over SSH.The port forwarding only last for the duration of your SSH session, if the SSH session breaks, your port forwarding will to. So, once you have the port forwarded, you can open the nextcloud AIO instance in your web browser at 127.0.0.1:8080

you will get this error because you are trying to access a page on localhost over HTTPS. You can click on advanced and then continue to proceed to the next page. Your data is encrypted over SSH for this session as we are binding the port over SSH. Depending on your choice of browser, the above page might look different.once you have proceeded, the nextcloud AIO interface will open and will look something like this.

It will show an auto generated passphrase, you need to save this passphrase and make sure to not loose it. For the purposes of security, I have masked the passwords with capsicums. once you have noted down your password, you can proceed to the Nextcloud AIO login, enter your password and then login. After login you will be greeted with a screen like this.

now you can put the domain that you want to use in the Submit domain field. Once the domain check is done, you will proceed to the next step and see another screen like this

here you can select any optional containers for the features that you might want. IMPORTANT: Please make sure to also change the time zone at the bottom of the page according to the time zone you wish to operate in.

The timezone setup is also important because the data base will get initialized according to the set time zone. This could result in wrong initialization of database and you ending up in a startup loop for nextcloud. I faced this issue and could only resolve it after getting help from sahilister . Once you are done changing the timezone, and selecting any additional features you want, you can click on Download and start the containersIt will take some time for this process to finish, take a break and look at the farthest object in your room and take a sip of water. Once you are done, and the process has finished you will see a page similar to the following one.

wait patiently for everything to turn green.

once all the containers have started properly, you can open the nextcloud login interface on your configured domain, the initial login details are auto generated as you can see from the above screenshot. Again you will see a password that you need to note down or save to enter the nextcloud interface. Capsicums will not work as passwords. I have masked the auto generated passwords using capsicums.Now you can click on Open your Nextcloud button or go to your configured domain to access the login screen.

You can use the login details from the previous step to login to the administrator account of your Nextcloud instance. There you have it, your very own cloud!

Additional Notes:

How to properly reset Nextcloud setup?While following the above steps, or while following steps from some other tutorial, you may have made a mistake, and want to start everything again from scratch. The instructions for it are present in the Nextcloud documentation here . Here is the TLDR for a docker-compose setup. These steps will delete all data, do not use these steps on an existing nextcloud setup unless you know what you are doing.

Stop your master container.

docker-compose -f compose.yml down -v

The above command will also remove the volume associated with the master container

Stop all the child containers that has been started by the master container.

docker stop nextcloud-aio-apache nextcloud-aio-notify-push nextcloud-aio-nextcloud nextcloud-aio-imaginary nextcloud-aio-fulltextsearch nextcloud-aio-redis nextcloud-aio-database nextcloud-aio-talk nextcloud-aio-collabora

Remove all the child containers that has been started by the master container

docker rm nextcloud-aio-apache nextcloud-aio-notify-push nextcloud-aio-nextcloud nextcloud-aio-imaginary nextcloud-aio-fulltextsearch nextcloud-aio-redis nextcloud-aio-database nextcloud-aio-talk nextcloud-aio-collabora

If you also wish to remove all images associated with nextcloud you can do it with

docker rmi $(docker images --filter "reference=nextcloud/*" -q)

remove all volumes associated with child containers

docker volume rm <volume-name>

remove the network associated with nextcloud

docker network rm nextcloud-aio

Additional references.
Nextcloud Github
Nextcloud reverse proxy documentation
Nextcloud Administration Guide
Nextcloud User Manual
Nextcloud Developer&aposs manual

7 December 2023

Daniel Kahn Gillmor: New OpenPGP certificate for dkg, December 2023

dkg's New OpenPGP certificate in December 2023 In December of 2023, I'm moving to a new OpenPGP certificate. You might know my old OpenPGP certificate, which had an fingerprint of C29F8A0C01F35E34D816AA5CE092EB3A5CA10DBA. My new OpenPGP certificate has a fingerprint of: D477040C70C2156A5C298549BB7E9101495E6BF7. Both certificates have the same set of User IDs:

Daniel Kahn Gillmor
<dkg@debian.org>
<dkg@fifthhorseman.net>

You can find a version of this transition statement signed by both the old and new certificates at: https://dkg.fifthhorseman.net/2023-dkg-openpgp-transition.txt The new OpenPGP certificate is:

When I have some reasonable number of certifications, i'll update the certificate associated with my e-mail addresses on https://keys.openpgp.org, in DANE, and in WKD. Until then, those lookups should continue to provide the old certificate.

9 August 2023

Antoine Beaupr : OpenPGP key transition

This is a short announcement to say that I have changed my main OpenPGP key. A signed statement is available with the cryptographic details but, in short, the reason is that I stopped using my old YubiKey NEO that I have worn on my keyring since 2015. I now have a YubiKey 5 which supports ED25519 which features much shorter keys and faster decryption. It allowed me to move all my secret subkeys on the key (including encryption keys) while retaining reasonable performance. I have written extensive documentation on how to do that OpenPGP key rotation and also YubiKey OpenPGP operations.

Warning on storing encryption keys on a YubiKey People wishing to move their private encryption keys to such a security token should be very careful as there are special precautions to take for disaster recovery. I am toying with the idea of writing an article specifically about disaster recovery for secrets and backups, dealing specifically with cases of death or disabilities.

Autocrypt changes One nice change is the impact on Autocrypt headers, which are considerably shorter. Before, the header didn't even fit on a single line in an email, it overflowed to five lines:

Autocrypt: addr=anarcat@torproject.org; prefer-encrypt=nopreference;
 keydata=xsFNBEogKJ4BEADHRk8dXcT3VmnEZQQdiAaNw8pmnoRG2QkoAvv42q9Ua+DRVe/yAEUd03EOXbMJl++YKWpVuzSFr7IlZ+/lJHOCqDeSsBD6LKBSx/7uH2EOIDizGwfZNF3u7X+gVBMy2V7rTClDJM1eT9QuLMfMakpZkIe2PpGE4g5zbGZixn9er+wEmzk2mt20RImMeLK3jyd6vPb1/Ph9+bTEuEXi6/WDxJ6+b5peWydKOdY1tSbkWZgdi+Bup72DLUGZATE3+Ju5+rFXtb/1/po5dZirhaSRZjZA6sQhyFM/ZhIj92mUM8JJrhkeAC0iJejn4SW8ps2NoPm0kAfVu6apgVACaNmFb4nBAb2k1KWru+UMQnV+VxDVdxhpV628Tn9+8oDg6c+dO3RCCmw+nUUPjeGU0k19S6fNIbNPRlElS31QGL4H0IazZqnE+kw6ojn4Q44h8u7iOfpeanVumtp0lJs6dE2nRw0EdAlt535iQbxHIOy2x5m9IdJ6q1wWFFQDskG+ybN2Qy7SZMQtjjOqM+CmdeAnQGVwxowSDPbHfFpYeCEb+Wzya337Jy9yJwkfa+V7e7Lkv9/OysEsV4hJrOh8YXu9a4qBWZvZHnIO7zRbz7cqVBKmdrL2iGqpEUv/x5onjNQwpjSVX5S+ZRBZTzah0w186IpXVxsU8dSk0yeQskblrwARAQABzSlBbnRvaW5lIEJlYXVwcsOpIDxhbmFyY2F0QHRvcnByb2plY3Qub3JnPsLBlAQTAQgAPgIbAwULCQgHAwUVCgkICwUWAgMBAAIeAQIXgBYhBI3JAc5kFGwEitUPu3khUlJ7dZIeBQJihnFIBQkacFLiAAoJEHkhUlJ7dZIeXNAP/RsX+27l9K5uGspEaMH6jabAFTQVWD8Ch1om9YvrBgfYtq2k/m4WlkMh9IpT89Ahmlf0eq+V1Vph4wwXBS5McK0dzoFuHXJa1WHThNMaexgHhqJOs
 S60bWyLH4QnGxNaOoQvuAXiCYV4amKl7hSuDVZEn/9etDgm/UhGn2KS3yg0XFsqI7V/3RopHiDT+k7+zpAKd3st2V74w6ht+EFp2Gj0sNTBoCdbmIkRhiLyH9S4B+0Z5dUCUEopGIKKOSbQwyD5jILXEi7VTZhN0CrwIcCuqNo7OXI6e8gJd8McymqK4JrVoCipJbLzyOLxZMxGz8Ki0b9O844/DTzwcYcg9I1qogCsGmZfgVze2XtGxY+9zwSpeCLeef6QOPQ0uxsEYSfVgS+onCesSRCgwAPmppPiva+UlGuIMun87gPpQpV2fqFg/V8zBxRvs6YTGcfcQjfMoBHmZTGb+jk1//QAgnXMO7fGG38YH7iQSSzkmodrH2s27ZKgUTHVxpBL85ptftuRqbR7MzIKXZsKdA88kjIKKXwMmez9L1VbJkM4k+1Kzc5KdVydwi+ujpNegF6ZU8KDNFiN9TbDOlRxK5R+AjwdS8ZOIa4nci77KbNF9OZuO3l/FZwiKp8IFJ1nK7uiKUjmCukL0od/6X2rJtAzJmO5Co93ZVrd5r48oqUvjklzzsBNBFmeC3oBCADEV28RKzbv3dEbOocOsJQWr1R0EHUcbS270CrQZfb9VCZWkFlQ/1ypqFFQSjmmUGbNX2CG5mivVsW6Vgm7gg8HEnVCqzL02BPY4OmylskYMFI5Bra2wRNNQBgjg39L9XU4866q3BQzJp3r0fLRVH8gHM54Jf0FVmTyHotR/Xiw5YavNy2qaQXesqqUv8HBIha0rFblbuYI/cFwOtJ47gu0QmgrU0ytDjlnmDNx4rfsNylwTIHS0Oc7Pezp7MzLmZxnTM9b5VMprAXnQr4rewXCOUKBSto+j4rD5/77DzXw96bbueNruaupb2Iy2OHXNGkB0vKFD3xHsXE2x75NBovtABEBAAHCwqwEGAEIACAWIQSNyQHOZBRsBIrVD7t5IVJSe3WSHgUCWZ4LegIbAgFACRB5IV
 JSe3WSHsB0IAQZAQgAHRYhBHsWQgTQlnI7AZY1qz6h3d2yYdl7BQJZngt6AAoJED6h3d2yYdl7CowH/Rp7GHEoPZTSUK8Ss7crwRmuAIDGBbSPkZbGmm4bOTaNs/gealc2tsVYpoMx7aYgqUW+t+84XciKHT+bjRv8uBnHescKZgDaomDuDKc2JVyx6samGFYuYPcGFReRcdmH0FOoPCn7bMW5mTPztV/wIA80LZD9kPKIXanfUyI3HLP0BPwZG4WTpKzJaalR1BNwu2oF6kEK0ymH3LfDiJ5Sr6emI2jrm4gH+/19ux/x+ST4tvm2PmH3BSQOPzgiqDiFd7RZoAIhmwr3FW4epsK9LtSxsi9gZ2vATBKO1oKtb6olW/keQT6uQCjqPSGojwzGRT2thEANH+5t6Vh0oDPZhrKUXRAAxHMBNHEaoo/M0sjZo+5OF3Ig1rMnI6XbKskLv6hu13cCymW0w/5E4XuYnyQ1cNC3pLvqDQbDx5mAPfBVHuqxJdRLQ3yDM/D2QIsxnkzQwi0FsJuni4vuJzWK/NHHDCvxMCh0YmSgbptUtgW8/niatd2Y6MbfRGxUHoctKtzqzivC8hKMTFrj4AbZhg/e9QVCsh5zSXtpWP0qFDJsxRMx0/432n9d4XUiy4U672r9Q09SsynB3QN6nTaCTWCIxGxjIb+8kJrRqTGwy/PElHX6kF0vQUWZNf2ITV1sd6LK/s/7sH+x4rzgUEHrsKr/qPvY3rUY/dQLd+owXesY83ANOu6oMWhSJnPMksbNa4tIKKbjmw3CFIOfoYHOWf3FtnydHNXoXfj4nBX8oSnkfhLILTJgf6JDFXfw6mTsv/jMzIfDs7PO1LK2oMK0+prSvSoM8bP9dmVEGIurzsTGjhTOBcb0zgyCmYVD3S48vZlTgHszAes1zwaCyt3/tOwrzU5JsRJVns+B/TUYaR/u3oIDMDygvE5ObWxXaFVnCC59r+zl0FazZ0ouyk2AYIR
 zHf+n1n98HCngRO4FRel2yzGDYO2rLPkXRm+NHCRvUA/i4zGkJs2AV0hsKK9/x8uMkBjHAdAheXhY+CsizGzsKjjfwvgqf84LwAzSDdZqLVE2yGTOwU0ESiArJwEQAJhtnC6pScWjzvvQ6rCTGAai6hrRiN6VLVVFLIMaMnlUp92EtgVSNpw6kANtRTpKXUB5fIPZVUrVdfEN06t96/6LE42tgifDAFyFTZY5FdHHri1GG/Cr39MpW2VqCDCtTTPVWHTUlU1ZG631BJ+9NB+ce58TmLr6wBTQrT+W367eRFBC54EsLNb7zQAspCn9pw1xf1XNHOGnrAQ4r9BXhOW5B8CzRd4nLRQwVgtw/c5M/bjemAOoq2WkwN+0mfJe4TSfHwFUozXuN274X+0Gr10fhp8xEDYuQM0qu6W3aDXMBBwIu0jTNudEELsTzhKUbqpsBc9WjwNMCZoCuSw/RTpFBV35mXbqQoQgbcU7uWZslLl9Wvv/C6rjXgd+GeX8SGBjTqq1ZkTv5UXLHTNQzPnbkNEExzqToi/QdSjFMIACnakeOSxc0ckfnsd9pfGv1PUyPyiwrHiqWFzBijzGIZEHxhNGFxAkXwTJR7Pd40a7RDxwbO6p/TSIIum41JtteehLHwTRDdQNMoyfLxuNLEtNYS0uR2jYI1EPQfCNWXCdT2ZK/l6GVP6jyB/olHBIOr+oVXqJh+48ki8cATPczhq3fUr7UivmguGwD67/4omZ4PCKtz1hNndnyYFS9QldEGo+AsB3AoUpVIA0XfQVkxD9IZr+Zu6aJ6nWq4M2bsoxABEBAAHCwXYEGAEIACACGwwWIQSNyQHOZBRsBIrVD7t5IVJSe3WSHgUCWPerZAAKCRB5IVJSe3WSHkIgEACTpxdn/FKrwH0/LDpZDTKWEWm4416l13RjhSt9CUhZ/Gm2GNfXcVTfoF/jKXXgjHcV1DHjfLUPmPVwMdqlf5ACOiFqIUM2ag/OEARh356w
 YG7YEobMjX0CThKe6AV2118XNzRBw/S2IO1LWnL5qaGYPZONUa9Pj0OaErdKIk/V1wge8Zoav2fQPautBcRLW5VA33PH1ggoqKQ4ES1hc9HC6SYKzTCGixu97mu/vjOa8DYgM+33TosLyNy+bCzw62zJkMf89X0tTSdaJSj5Op0SrRvfgjbC2YpJOnXxHr9qaXFbBZQhLjemZi6zRzUNeJ6A3Nzs+gIc4H7s/bYBtcd4ugPEhDeCGffdS3TppH9PnvRXfoa5zj5bsKFgjqjWolCyAmEvd15tXz5yNXtvrpgDhjF5ozPiNp/1EeWX4DxbH2i17drVu4fXwauFZ6lcsAcJxnvCA28RlQlmEQu/gFOx1axVXf6GIuXnQSjQN6qJbByUYrdc/cFCxPO2/lGuUxnufN9Tvb51Qh54laPgGLrlD2huQeSD9Sxa0MNUjNY0qLqaReT99Ygb2LPYGSLoFVx9iZz6sZNt07LqCx9qNgsJwsdmwYsNpMuFbc7nkWjtlEqzsXZHTvYN654p43S+hcAhmmOzQZcew6h71fAJLciiqsPBnCEdgCGFAWhZZdPkMA==

After the change, the entire key fits on a single line, neat!

Autocrypt: addr=anarcat@torproject.org; prefer-encrypt=nopreference;
 keydata=xjMEZHZPzhYJKwYBBAHaRw8BAQdAWdVzOFRW6FYVpeVaDo3sC4aJ2kUW4ukdEZ36UJLAHd7NKUFudG9pbmUgQmVhdXByw6kgPGFuYXJjYXRAdG9ycHJvamVjdC5vcmc+wpUEExYIAD4WIQS7ts1MmNdOE1inUqYCKTpvpOU0cwUCZHZgvwIbAwUJAeEzgAULCQgHAwUVCgkICwUWAgMBAAIeAQIXgAAKCRACKTpvpOU0c47SAPdEqfeHtFDx9UPhElZf7nSM69KyvPWXMocu9Kcu/sw1AQD5QkPzK5oxierims6/KUkIKDHdt8UcNp234V+UdD/ZB844BGR2UM4SCisGAQQBl1UBBQEBB0CYZha2IMY54WFXMG4S9/Smef54Pgon99LJ/hJ885p0ZAMBCAfCdwQYFggAIBYhBLu2zUyY104TWKdSpgIpOm+k5TRzBQJkdlDOAhsMAAoJEAIpOm+k5TRzBg0A+IbcsZhLx6FRIqBJCdfYMo7qovEo+vX0HZsUPRlq4HkBAIctCzmH3WyfOD/aUTeOF3tY+tIGUxxjQLGsNQZeGrQI

Note that I have implemented my own kind of ridiculous Autocrypt support for the Notmuch Emacs email client I use, see this elisp code. To import keys, I pipe the message into this script which is basically just:

sq autocrypt decode   gpg --import

... thanks to Sequoia best-of-class Autocrypt support.

Note on OpenPGP usage While some have claimed OpenPGP's death, I believe those are overstated. Maybe it's just me, but I still use OpenPGP for my password management, to authenticate users and messages, and it's the interface to my YubiKey for authenticating with SSH servers. I understand people feel that OpenPGP is possibly insecure, counter-intuitive and full of problems, but I think most of those problems should instead be attributed to its current flagship implementation, GnuPG. I have tried to work with GnuPG for years, and it keeps surprising me with evilness and oddities. I have high hopes that the Sequoia project can bring some sanity into this space, and I also hope that RFC4880bis can eventually get somewhere so we have a more solid specification with more robust crypto. It's kind of a shame that this has dragged on for so long, but Update: there's a separate draft called openpgp-crypto-refresh that might actually be adopted as the "OpenPGP RFC" soon! And it doesn't keep real work from happening in Sequoia and other implementations. Thunderbird rewrote their OpenPGP implementation with RNP (which was, granted, a bumpy road because it lost compatibility with GnuPG) and Sequoia now has a certificate store with trust management (but still no secret storage), preliminary OpenPGP card support and even a basic GnuPG compatibility layer. I'm also curious to try out the OpenPGP CA capabilities. So maybe it's just because I'm becoming an old fart that doesn't want to change tools, but so far I haven't seen a good incentive in switching away from OpenPGP, and haven't found a good set of tools that completely replace it. Maybe OpenSSH's keys and CA can eventually replace it, but I suspect they will end up rebuilding most of OpenPGP anyway, just more slowly. If they do, let's hope they avoid the mistakes our community has done in the past at least...

10 July 2023

Michael Ablassmeier: Java timezone sheanigans

While running CI tests for a application that is implemented in C and Java, some configuration scripts set the current timezone. The C implemented parts catch the change just nicely, but the java related parts still report the default image timezone. A simple example:

import java.util.*;
import java.text.*;
class simpleTest
 
        public static void main(String args[])
         
           Calendar cal = Calendar.getInstance();
           System.out.println("TIME ZONE :"+ cal.getTimeZone().getDisplayName());

Result:

vagrant@vm:~$ sudo timedatectl set-timezone America/Aruba 
vagrant@vm:~$ timedatectl 
[..]
                Time zone: America/Aruba (AST, -0400)
[..]
vagrant@vm:~$ java test.java
TIME ZONE :Central European Standard Time
vagrant@vm:~$ ls -alh /etc/localtime 
lrwxrwxrwx 1 root root 35 Jul 10 14:41 /etc/localtime -> ../usr/share/zoneinfo/America/Aruba

It appears the Java implementation uses /etc/timezone as source, not /etc/localtime.

vagrant@vm:~$ echo America/Aruba   sudo tee /etc/timezone 
America/Aruba
vagrant@vm:~$ java test.java
TIME ZONE :Atlantic Standard Time

dpkg-reconfigure tzdata updates this file as well, so using timedatectl only won t be enough (at least not on Debian based systems which run java based applications.)

14 June 2023

Freexian Collaborators: Monthly report about Debian Long Term Support, May 2023 (by Roberto C. S nchez)

Like each month, have a look at the work funded by Freexian s Debian LTS offering.

Debian LTS contributors In May, 18 contributors have been paid to work on Debian LTS, their reports are available:

Abhijith PA did 6.0h (out of 6.0h assigned and 8.0h from previous period), thus carrying over 8.0h to the next month.

Anton Gladky did 6.0h (out of 8.0h assigned and 7.0h from previous period), thus carrying over 9.0h to the next month.

Bastien Roucari s did 17.0h (out of 17.0h assigned and 3.0h from previous period), thus carrying over 3.0h to the next month.

Ben Hutchings did 17.0h (out of 16.0h assigned and 8.0h from previous period), thus carrying over 7.0h to the next month.

Chris Lamb did 18.0h (out of 18.0h assigned).

Daniel Leidert did 0.0h (out of 0h assigned and 12.0h from previous period), thus carrying over 12.0h to the next month.

Dominik George did 0.0h (out of 0h assigned and 20.34h from previous period), thus carrying over 20.34h to the next month.

Emilio Pozuelo Monfort did 32.0h (out of 18.5h assigned and 16.0h from previous period), thus carrying over 2.5h to the next month.

Guilhem Moulin did 20.0h (out of 8.5h assigned and 11.5h from previous period).

Holger Levsen did 0.0h (out of 0h assigned and 10.0h from previous period), thus carrying over 10.0h to the next month.

Lee Garrett did 0.0h (out of 0h assigned and 40.5h from previous period), thus carrying over 40.5h to the next month.

Markus Koschany did 34.5h (out of 34.5h assigned).

Roberto C. S nchez did 18.25h (out of 20.5h assigned and 11.5h from previous period), thus carrying over 13.75h to the next month.

Scarlett Moore did 20.0h (out of 20.0h assigned).

Sylvain Beucler did 34.5h (out of 29.0h assigned and 5.5h from previous period).

Thorsten Alteholz did 14.0h (out of 14.0h assigned).

Tobias Frost did 16.0h (out of 15.0h assigned and 1.0h from previous period).

Utkarsh Gupta did 5.5h (out of 5.0h assigned and 26.0h from previous period), thus carrying over 25.5h to the next month.

Evolution of the situation In May, we have released 34 DLAs. Several of the DLAs constituted notable security updates to LTS during the month of May. Of particular note were the linux (4.19) and linux-5.10 packages, both of which addressed a considerable number of CVEs. Additionally, the postgresql-11 package was updated by synchronizing it with the 11.20 release from upstream. Notable non-security updates were made to the distro-info-data database and the timezone database. The distro-info-data package was updated with the final expected release date of Debian 12, made aware of Debian 14 and Ubuntu 23.10, and was updated with the latest EOL dates for Ubuntu releases. The tzdata and libdatetime-timezone-perl packages were updated with the 2023c timezone database. The changes in these packages ensure that in addition to the latest security updates LTS users also have the latest information concerning Debian and Ubuntu support windows, as well as the latest timezone data for accurate worldwide timekeeping. LTS contributor Anton implemented an improvement to the Debian Security Tracker Unfixed vulnerabilities in unstable without a filed bug view, allowing for more effective management of CVEs which do not yet have a corresponding bug entry in the Debian BTS. LTS contributor Sylvain concluded an audit of obsolete packages still supported in LTS to ensure that new CVEs are properly associated. In this case, a package being obsolete means that it is no longer associated with a Debian release for which the Debian Security Team has direct responsibility. When this occurs, it is the responsibility of the LTS team to ensure that incoming CVEs are properly associated to packages which exist only in LTS. Finally, LTS contributors also contributed several updates to packages in unstable/testing/stable to fix CVEs. This helps package maintainers, addresses CVEs in current and future Debian releases, and ensures that the CVEs do not remain open for an extended period of time only for the LTS team to be required to deal with them much later in the future.

Thanks to our sponsors Sponsors that joined recently are in bold.

Platinum sponsors:

TOSHIBA (for 93 months)

Civil Infrastructure Platform (CIP) (for 61 months)

Gold sponsors:

Roche Diagnostics International AG (for 104 months)

Linode (for 98 months)

Babiel GmbH (for 87 months)

Plat Home (for 87 months)

University of Oxford (for 43 months)

Deveryware (for 30 months)

VyOS Inc (for 25 months)

EDF SA (for 14 months)

Silver sponsors:

Domeneshop AS (for 108 months)

Nantes M tropole (for 102 months)

Univention GmbH (for 94 months)

Universit Jean Monnet de St Etienne (for 94 months)

Ribbon Communications, Inc. (for 88 months)

Exonet B.V. (for 78 months)

Leibniz Rechenzentrum (for 72 months)

CINECA (for 61 months)

Minist re de l Europe et des Affaires trang res (for 55 months)

Cloudways Ltd (for 45 months)

Dinahosting SL (for 43 months)

Bauer Xcel Media Deutschland KG (for 37 months)

Platform.sh (for 37 months)

Moxa Inc. (for 31 months)

sipgate GmbH (for 28 months)

OVH US LLC (for 26 months)

Tilburg University (for 26 months)

GSI Helmholtzzentrum f r Schwerionenforschung GmbH (for 17 months)

Soliton Systems K.K. (for 15 months)

Bronze sponsors:

Evolix (for 109 months)

Seznam.cz, a.s. (for 109 months)

Linuxhotel GmbH (for 106 months)

Intevation GmbH (for 105 months)

Daevel SARL (for 104 months)

Bitfolk LTD (for 103 months)

Megaspace Internet Services GmbH (for 103 months)

Greenbone AG (for 102 months)

NUMLOG (for 102 months)

WinGo AG (for 102 months)

Ecole Centrale de Nantes - LHEEA (for 98 months)

Entr ouvert (for 93 months)

Adfinis AG (for 90 months)

GNI MEDIA (for 85 months)

Laboratoire LEGI - UMR 5519 / CNRS (for 85 months)

Tesorion (for 85 months)

Bearstech (for 76 months)

LiHAS (for 76 months)

Catalyst IT Ltd (for 71 months)

Supagro (for 66 months)

Demarcq SAS (for 65 months)

Universit Grenoble Alpes (for 51 months)

TouchWeb SAS (for 43 months)

SPiN AG (for 40 months)

CoreFiling (for 35 months)

Institut des sciences cognitives Marc Jeannerod (for 30 months)

Observatoire des Sciences de l Univers de Grenoble (for 27 months)

Tem Innovations GmbH (for 21 months)

WordFinder.pro (for 21 months)

CNRS DT INSU R sif (for 20 months)

Alter Way (for 13 months)

Institut Camille Jordan

3 March 2023

Sven Hoexter: exfat-fuse 1.4 in experimental

I know a few people hold on to the exFAT fuse implementation due the support for timezone offsets, so here is a small update for you. Andrew released 1.4.0, which includes the timezone offset support, which was so far only part of the git master branch. It also fixes a, from my point of view very minor, security issue CVE-2022-29973. In addition to that it's the first build with fuse3 support. If you still use this driver, pick it up in experimental (we're in the bookworm freeze right now), and give it a try. I'm personally not using it anymore beyond a very basic "does it mount" test.

7 February 2023

Stephan Lachnit: Installing Debian on F2FS rootfs with deboostrap and systemd-boot

I recently got a new NVME drive. My plan was to create a fresh Debian install on an F2FS root partition with compression for maximum performance. As it turns out, this is not entirely trivil to accomplish. For one, the Debian installer does not support F2FS (here is my attempt to add it from 2021). And even if it did, grub does not support F2FS with the extra_attr flag that is required for compression support (at least as of grub 2.06). Luckily, we can install Debian anyway with all these these shiny new features when we go the manual road with debootstrap and using systemd-boot as bootloader. We can break down the process into several steps:

Warning: Playing around with partitions can easily result in data if you mess up! Make sure to double check your commands and create a data backup if you don t feel confident about the process.

Creating the partition partble The first step is to create the GPT partition table on the new drive. There are several tools to do this, I recommend the ArchWiki page on this topic for details. For simplicity I just went with the GParted since it has an easy GUI, but feel free to use any other tool. The layout should look like this:

Type         Partition        Suggested size
 
EFI          /dev/nvme0n1p1           512MiB
Linux swap   /dev/nvme0n1p2             1GiB
Linux fs     /dev/nvme0n1p3        remainder

Notes:

The disk names are just an example and have to be adjusted for your system.
Don t set disk labels, they don t appear on the new install anyway and some UEFIs might not like it on your boot partition.
The size of the EFI partition can be smaller, in practive it s unlikely that you need more than 300 MiB. However some UEFIs might be buggy and if you ever want to install an additional kernel or something like memtest86+ you will be happy to have the extra space.
The swap partition can be omitted, it is not strictly needed. If you need more swap for some reason you can also add more using a swap file later (see ArchWiki page). If you know you want to use suspend-to-RAM, you want to increase the size to something more than the size of your memory.
If you used GParted, create the EFI partition as FAT32 and set the esp flag. For the root partition use ext4 or F2FS if available.

Creating and mounting the root partition To create the root partition, we need to install the f2fs-tools first:

sudo apt install f2fs-tools

Now we can create the file system with the correct flags:

mkfs.f2fs -O extra_attr,inode_checksum,sb_checksum,compression,encrypt /dev/nvme0n1p3

For details on the flags visit the ArchWiki page. Next, we need to mount the partition with the correct flags. First, create a working directory:

mkdir boostrap
cd boostrap
mkdir root
export DFS=$(pwd)/root

Then we can mount the partition:

sudo mount -o compress_algorithm=zstd:6,compress_chksum,atgc,gc_merge,lazytime /dev/nvme0n1p3 $DFS

Again, for details on the mount options visit the above mentioned ArchWiki page.

Bootstrapping with debootstrap First we need to install the debootstrap package:

sudo apt install debootstrap

Now we can do the bootstrapping:

debootstrap --arch=amd64 --components=main,contrib,non-free,non-free-firmware unstable $DFS http://deb.debian.org/debian

Notes:

--arch sets the CPU architecture (see Debian Wiki).
--components sets the archive components, if you don t want non-free pacakges you might want to remove some entries here.
unstable is the Debian release, you might want to change that to testing or bookworm.
$DFS points to the mounting point of the root partition.
http://deb.debian.org/debian is the Debian mirror, you might want to set that to http://ftp.de.debian.org/debian or similar if you have a fast mirror in you area.

Chrooting into the system Before we can chroot into the newly created system, we need to prepare and mount virtual kernel file systems. First create the directories:

sudo mkdir -p $DFS/dev $DFS/dev/pts $DFS/proc $DFS/sys $DFS/run $DFS/sys/firmware/efi/efivars $DFS/boot/efi

Then bind-mount the directories from your system to the mount point of the new system:

sudo mount -v -B /dev $DFS/dev
sudo mount -v -B /dev/pts $DFS/dev/pts
sudo mount -v -B /proc $DFS/proc
sudo mount -v -B /sys $DFS/sys
sudo mount -v -B /run $DFS/run
sudo mount -v -B /sys/firmware/efi/efivars $DFS/sys/firmware/efi/efivars

As a last step, we need to mount the EFI partition:

sudo mount -v -B /dev/nvme0n1p1 $DFS/boot/efi

Now we can chroot into new system:

sudo chroot $DFS /bin/bash

Configure the base system The first step in the chroot is setting the locales. We need this since we might leak the locales from our base system into the chroot and if this happens we get a lot of annoying warnings.

export LC_ALL=C.UTF-8 LANG=C.UTF-8
apt install locales console-setup

Set your locales:

dpkg-reconfigure locales

Set your keyboard layout:

dpkg-reconfigure keyboard-configuration

Set your timezone:

dpkg-reconfigure tzdata

Now you have a fully functional Debian chroot! However, it is not bootable yet, so let s fix that.

Define static file system information The first step is to make sure the system mounts all partitions on startup with the correct mount flags. This is done in /etc/fstab (see ArchWiki page). Open the file and change its content to:

# file system                               mount point   type   options                                                            dump   pass
# NVME efi partition
UUID=XXXX-XXXX                              /boot/efi     vfat   umask=0077                                                         0      0
# NVME swap
UUID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX   none          swap   sw                                                                 0      0
# NVME main partition
UUID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX   /             f2fs   compress_algorithm=zstd:6,compress_chksum,atgc,gc_merge,lazytime   0      1

You need to fill in the UUIDs for the partitions. You can use

ls -lAph /dev/disk/by-uuid/

to match the UUIDs to the more readable disk name under /dev.

Installing the kernel and bootloader First install the systemd-boot and efibootmgr packages:

apt install systemd-boot efibootmgr

Now we can install the bootloader:

bootctl install --path=/boot/efi

You can verify the procedure worked with

efibootmgr -v

The next step is to install the kernel, you can find a fitting image with:

apt search linux-image-*

In my case:

apt install linux-image-amd64

After the installation of the kernel, apt will add an entry for systemd-boot automatically. Neat! However, since we are in a chroot the current settings are not bootable. The first reason is the boot partition, which will likely be the one from your current system. To change that, navigate to /boot/efi/loader/entries, it should contain one config file. When you open this file, it should look something like this:

title      Debian GNU/Linux bookworm/sid
version    6.1.0-3-amd64
machine-id 2967cafb6420ce7a2b99030163e2ee6a
sort-key   debian
options    root=PARTUUID=f81d4fae-7dec-11d0-a765-00a0c91e6bf6 ro systemd.machine_id=2967cafb6420ce7a2b99030163e2ee6a
linux      /2967cafb6420ce7a2b99030163e2ee6a/6.1.0-3-amd64/linux
initrd     /2967cafb6420ce7a2b99030163e2ee6a/6.1.0-3-amd64/initrd.img-6.1.0-3-amd64

The PARTUUID needs to point to the partition equivalent to /dev/nvme0n1p3 on your system. You can use

ls -lAph /dev/disk/by-partuuid/

to match the PARTUUIDs to the more readable disk name under /dev. The second problem is the ro flag in options which tell the kernel to boot in read-only mode. The default is rw, so you can just remove the ro flag. Once this is fixed, the new system should be bootable. You can change the boot order with:

efibootmgr --bootorder

However, before we reboot we might add well add a user and install some basic software.

Finishing touches Add a user:

useradd -m -G sudo -s /usr/bin/bash -c 'Full Name' username

Debian provides a TUI to install Desktop Environment. To open it, run:

tasksel

Now you can finally reboot into your new system:

reboot

Resources for further reading https://ivanb.neocities.org/blogs/y2022/debootstrap
https://www.debian.org/releases/stable/amd64/apds03.en.html
https://www.addictivetips.com/ubuntu-linux-tips/set-up-systemd-boot-on-arch-linux/
https://p5r.uk/blog/2020/using-systemd-boot-on-debian-bullseye.html
https://www.linuxfromscratch.org/lfs/view/stable/chapter07/kernfs.html
Thanks for reading!

28 January 2023

Craig Small: Fixing iCalendar feeds

The local government here has all the schools use an iCalendar feed for things like when school terms start and stop and other school events occur. The department s website also has events like public holidays. The issue is that all of them don t make it an all-day event but one that happens at midnight, or one past midnight. The events synchronise fine, though Google s calendar is known for synchronising when it feels like it, not at any particular time you would like it to.

Screenshot of Android Calendar showing a tiny bar at midnight which is the event.

Even though a public holiday is all day, they are sent as appointments for midnight. That means on my phone all the events are these tiny bars that appear right up the top of the screen and are easily missed, especially when the focus of the calendar is during the day. On the phone, you can see the tiny purple bar at midnight. This is how the events appear. It s not the calendar s fault, as far as it knows the school events are happening at midnight. You can also see Lunar New Year and Australia Day appear in the all-day part of the calendar and don t scroll away. That s where these events should be.

Why are all the events appearing at midnight? The reason is the feed is incorrectly set up and has the time. The events are sent in an iCalendar format and a typical event looks like this:

BEGIN:VEVENT
DTSTART;TZID=Australia/Sydney:20230206T000000
DTEND;TZID=Australia/Sydney:20230206T000000
SUMMARY:School Term starts
END:VEVENT

The event starting and stopping date and time are the DTSTART and DTEND lines. Both of them have the date of 2023/02/06 or 6th February 2023 and a time of 00:00:00 or midnight. So the calendar is doing the right thing, we need to fix the feed! The Fix I wrote a quick and dirty PHP script to download the feed from the real site, change the DTSTART and DTEND lines to all-day events and leave the rest of it alone.

<?php
$site = $_GET['s'];
if ($site == 'site1')  
    $REMOTE_URL='https://site1.example.net/ical_feed';
  elseif ($site == 'site2')  
    $REMOTE_URL='https://site2.example.net/ical_feed';
  else  
    http_response_code(400);
    die();
 
$fp = fopen($REMOTE_URL, "r");
if (!$fp)  
    die("fopen");
 
header('Content-Type: text/calendar');
while (( $line = fgets($fp, 1024)) !== false)  
    $line = preg_replace(
        '/^(DTSTART DTEND);[^:]+:([0-9] 8 )T000[01]00/',
        '$ 1 ;VALUE=DATE:$ 2 ',
        $line);
    echo $line;
 
?>

It s pretty quick and nasty but gets the job done. So what is it doing?

Lines 2-10: Check the given variable s and match it to either site1 or site2 to obtain the URL. If you only had one site to fix you could just set the REMOTE_URL variable.
Lines 12-15: A typical fopen() and nasty error handling.
Line 16: set the content type to a calendar.
Line 17: A while loop to read the contents of the remote site line by line.
Line 18-21: This is where the magic happens, preg_replace is a Perl regular expression replacement. The PCRE is:
- Finding lines starting with DTSTART or DTEND and store it in capture 1
- Skip everything that isn t a colon. This is the timezone information. I wasn t sure if it was needed and how to combine it so I took it out. All the all-day events I saw don t have a time zone.
- Find 8 numerics (this is for YYYYMMDD) and store it in capture 2.
- Scan the Time part, a literal T then HHMMSS. Some sites use midnight some use one minute past, so it covers both.
- Replace the line with either DTSTART or DTEND (capture 1), set the value type to DATE as the default is date/time and print the date (capture 2).
Line 22: Print either the modified or original line.

You need to save the script on your web server somewhere, possibly with an alias command. The whole point of this is to change the type from a date/time to a date-only event and only print the date part of it for the start and end of it. The resulting iCalendar event looks like this:

BEGIN:VEVENT
DTSTART;VALUE=DATE:20230206
DTEND;VALUE=DATE:20230206
SUMMARY:School Term starts
END:VEVENT

The calendar then shows it properly as an all-day event. I would check the script works before doing the next step. You can use things like curl or wget to download it. If you use a normal browser, it will probably just download the translated file. If you re not seeing the right thing then it s probably the PCRE failing. You can check it online with a regex checker such as https://regex101.com. The site has saved my PCRE and match so you got something to start with. Calendar settings The last thing to do is to change the URL in your calendar settings. Each calendar system has a different way of doing it. For Google Calendar they provide instructions and you want to follow the section titled Use a link to add a public Calendar . The URL here is not the actual site s URL (which you would have put into the REMOTE_URL variable before) but the URL of your script plus the ?s=site1 part. So if you put your script aliased to /myical.php and the site ID was site1 and your website is www.example.com the URL would be https://www.example.com/myical.php?s=site1 . You should then see the events appear as all-day events on your calendar.

12 December 2022

Vasudev Kamath: Installing Debian from GRML Live CD

I had bought a Thinkpad E470 laptop back in 2018 which was lying unused for quite some time. Recently when I wanted to use it, I found that the keyboard is not working, especially some keys and after some time the laptop will hang in Lenovo boot screen. I came back to Bangalore almost after 2 years from my hometown (WFH due to Covid) and thought it was the right time to get my laptop back to normal working state. After getting the keyboard replaced I noticed that 1TB HDD is no longer fast enough for my taste!. I've to admit I never thought I would start disliking HDD so quickly thanks to modern SSD based work laptops. So as a second upgrade I got the HDD removed from my laptop and got a 240G SSD. Yeah I know its reduction from my original size but I intend to continue using my old HDD via USB SATA enclosure as an external HDD which can house the extra data which I need to save. So now that I've a SSD I need to install Debian Unstable again on it and this is where I tried something new. My colleague (name redacted on request) suggested to me use GRML live CD and install Debian via debootstrap. And after giving a thought I decided to try this out. Some reason for going ahead with this are listed below

Debian Installer does not support a proper BTRFS based root file system. It just allows btrfs as root but no subvolume support. Also I'm not sure about the luks support with btrfs as root.
I also wanted to give a try to systemd-boot as my laptop is UEFI capable and I've slowly started disliking Grub.
I really hate installing task-kde-desktop (Yeah you read it right, I've switched to be a KDE user for quite some time) which will pull tons of unwanted stuff and bloat. Well it's not just task-kde-desktop but any other task-desktop package does similar and I don't want to have too much of unused stuff and services running.

Disk Preparation As a first step I went to GRML website and downloaded current pre-release. Frankly, I'm using GRML for first time and I was not sure what to expect. When I booted it up I was bit taken a back to see its console based and I did not have a wired lan just a plain wireless dongle (Jiofi device) and was wondering what it will take to connect. But surprisingly curses based UI was pretty much straight forward to allow me to connect to Wifi AP. Another thing was the rescue CD had non-free firmware as the laptop was using ath10k device and needed non-free blobs to operate. Once I got shell prompt in rescue CD first thing I did was to reconfigure console-setup to increase font size which was very very small on default boot. Once that is done I did the following to create a 1G (FAT32) partition for EFI.

parted -a optimal -s /dev/sda mklabel gpt
parted -a optimal -s /dev/sda mkpart primary vfat 0% 1G
parted -a optimal -s /dev/sda set 1 esp on
mkfs.vfat -n boot_disk -F 32 /dev/sda1

So here is what I did: created a 1G vfat type partition and set the esp flag on it. This will be mounted to /boot/efi for systemd-boot. Next I created a single partition on the rest of the available free disk which will be used as the root file system. Next I encrypted the root parition using LUKS and then created the BTRFS file system on top of it.

cryptsetup luksFormat /dev/sda2
cryptsetup luksOpen /dev/sda2 ENC
mkfs.btrfs -L root_disk /dev/mapper/ENC

Next is to create subvolumes in BTRFS. I followed suggestion by colleague and created a top-level @ as subvolume below which created @/home @/var/log @/opt . Also enabled compression with zstd and level of 1 to avoid battery drain. Finally marked the @ as default subvolume to avoid adding it to fstab entry.

mount -o compress=zstd:1 /dev/mapper/ENC /mnt
btrfs subvol create /mnt/@
cd /mnt/@
btrfs subvol create ./home
btrfs subvol create ./opt
mkdir -p var
btrfs subvol create ./var/log
btrfs suvol set-default /mnt/@

Bootstrapping Debian Now that root disk is prepared next step was to bootstrap the root file system. I used debootstrap for this job. One thing I missed here from installer was ability to preseed. I tried looking around to figure out if we can preseed debootstrap but did not find much. If you know the procedure do point it to me.

cd /mnt/
debootstrap --include=dbus,locales,tzdata unstable @/ http://deb.debian.org/debian

Well this just gets a bare minimal installation of Debian I need to install rest of the things post this step manually by chroot into target folder @/. I like the grml-chroot command for chroot purpose, it does most of the job of mounting all required directory like /dev/ /proc /sys etc. But before entering chroot I need to mount the ESP partition we created to /boot/efi so that I can finalize the installation of kernel and systemd-boot.

umount /mnt
mount -o compress=zstd:1 /dev/mapper/ENC /mnt
mkdir -p /mnt/boot/efi
mount /dev/sda1 /mnt/boot/efi
grml-chroot /mnt /bin/bash

I remounted the root subvolume @ directly to /mnt now, remember I made @ as default subvolume before. I also mounted ESP partition with FAT32 file system to /boot/efi. Finally I used grml-chroot to get into chroot of newly bootstrapped file system. Now I will install the kernel and minimal KDE desktop installation and configure locales and time zone data for the new system. I wanted to use dracut instead of default initramfs-tools for initrd. I also need to install cryptsetup and btrfs-progs so I can decrypt and really boot into my new system.

apt-get update
apt-get install linux-image-amd64 dracut openssh-client \
                        kde-plasma-desktop plasma-workspace-wayland \
                        plasma-nm cryptsetup btrfs-progs sudo

Next is setting up crypttab and fstab entries for new system. Following entry is added to fstab

LABEL="root_disk" / btrfs defaults,compress=zstd:1 0 0

And the crypttab entry

ENCRYPTED_ROOT UUID=xxxx none discard,x-initrd.attach

I've not written actual UUID above this is just for the purpose of showing the content of /etc/crypttab. Once these entries are added we need to recreate initrd. I just reconfigured the installed kernel package for retriggerring the recreation of initrd using dracut. .. Reconfiguration was locales is done by editing /etc/locales.gen to uncomment en_US.UTF-8 and writing /etc/timezone with Asia/Kolkata. I used DEBIAN_FRONTEND=noninteractive to avoid another prompt asking for locale and timezone information.

export DEBIAN_FRONTEND=noninteractive
dpkg-reconfigure locales
dpkg-reconfigure tzdata

Added my user using adduser command and also set the root password as well. Added my user to sudo group so I can use sudo to elevate privileges.

Setting up systemd-boot So now basic usable system is ready last part is enabling the systemd-boot configuration as I'm not gonna use grub. I did following to install systemd-boot. Frankly I'm not expert of this it was colleague's suggestion. Before installing the systemd-boot I had to setup kernel command line. This can be done by writing command line to /etc/kernel/cmdline with following contents.

systemd.gpt_auto=no quiet root=LABEL=root_disk

I'm disabling systemd-gpt-generator to avoid race condition between crypttab entry and auto generated entry by systemd. I faced this mainly because of my stupidity of not adding entry root=LABEL=root_disk

apt-get install -y systemd-boot
bootctl install --make-entry-directory=yes --entry-token=machine-id
dpkg-reconfigure linux-image-6.0.0-5-amd64

Finally exit from the chroot and reboot into the freshly installed system. systemd-boot already ships a hook file zz-systemd-boot under /etc/kernel so its pretty much usable without any manual intervention. Previously after kernel installation we had to manually update kernel image in efi partitions using bootctl

Conclussion Though installing from live image is not new and debian-installer also does the same only difference is more control over installation and doing things which is installer is not letting you do (or should I say is not part of default installation?). If properly automated using scripts we can leverage this to do custom installation in large scale environments. I know there is FAI but I've not explored it and felt there is too much to setup for a simple installations with specific requirements. So finally I've a system with Debian which differs from default Debian installation :-). I should thank my colleague for rekindling nerd inside me who had stopped experimenting quite a long time back.

29 October 2022

Dirk Eddelbuettel: gettz 0.0.5 on CRAN: Maintenance

A minor routine update 0.0.5 of gettz arrived on CRAN overnight. gettz provides a possible fallback in situations where Sys.timezone() fails to determine the system timezone. That happened when e.g. the file /etc/localtime somehow is not a link into the corresponding file with zoneinfo data in, say, /usr/share/zoneinfo. Since the package was written (in the fall of 2016), R added a similar extended heuristic approach itself. This release updates a function signature to satisfy the more stringent tests by clang-15, updates the GitHub Action checkout code to suppress a nag, and changes a few remaining http documentation links to https. As with the previous releses: No functional changes, no new code, or new features. Courtesy of my CRANberries, there is a comparison to the previous release. More information is on the gettz page. For questions or comments use the issue tracker off the GitHub repo. If you like this or other open-source work I do, you can now sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

22 August 2022

Simon Josefsson: Static network config with Debian Cloud images

I self-host some services on virtual machines (VMs), and I m currently using Debian 11.x as the host machine relying on the libvirt infrastructure to manage QEMU/KVM machines. While everything has worked fine for years (including on Debian 10.x), there has always been one issue causing a one-minute delay every time I install a new VM: the default images run a DHCP client that never succeeds in my environment. I never found out a way to disable DHCP in the image, and none of the documented ways through cloud-init that I have tried worked. A couple of days ago, after reading the AlmaLinux wiki I found a solution that works with Debian. The following commands creates a Debian VM with static network configuration without the annoying one-minute DHCP delay. The three essential cloud-init keywords are the NoCloud meta-data parameters dsmode:local, static network-interfaces setting combined with the user-data bootcmd keyword. I m using a Raptor CS Talos II ppc64el machine, so replace the image link with a genericcloud amd64 image if you are using x86.

wget https://cloud.debian.org/images/cloud/bullseye/latest/debian-11-generic-ppc64el.qcow2
cp debian-11-generic-ppc64el.qcow2 foo.qcow2
cat>meta-data
dsmode: local
network-interfaces:  
 iface enp0s1 inet static
 address 192.168.98.14/24
 gateway 192.168.98.12
^D
cat>user-data
#cloud-config
fqdn: foo.mydomain
manage_etc_hosts: true
disable_root: false
ssh_pwauth: false
ssh_authorized_keys:
- ssh-ed25519 AAAA...
timezone: Europe/Stockholm
bootcmd:
- rm -f /run/network/interfaces.d/enp0s1
- ifup enp0s1
^D
virt-install --name foo --import --os-variant debian10 --disk foo.qcow2 --cloud-init meta-data=meta-data,user-data=user-data

Unfortunately virt-install from Debian 11 does not support the cloud-init network-config parameter, so if you want to use a version 2 network configuration with cloud-init (to specify IPv6 addresses, for example) you need to replace the final virt-install command with the following.

cat>network_config_static.cfg
version: 2
 ethernets:
  enp0s1:
   dhcp4: false
   addresses: [ 192.168.98.14/24, fc00::14/7 ]
   gateway4: 192.168.98.12
   gateway6: fc00::12
   nameservers:
    addresses: [ 192.168.98.12, fc00::12 ]
^D
cloud-localds -v -m local --network-config=network_config_static.cfg seed.iso user-data
virt-install --name foo --import --os-variant debian10 --disk foo.qcow2 --disk seed.iso,readonly=on --noreboot
virsh start foo
virsh detach-disk foo vdb --config
virsh console foo

There are still some warnings like the following, but it does not seem to cause any problem: [FAILED] Failed to start Initial cloud-init job (pre-networking). Finally, if you do not want the cloud-init tools installed in your VMs, I found the following set of additional user-data commands helpful. Cloud-init will not be enabled on first boot and a cron job will be added that purges some unwanted packages.

runcmd:
- touch /etc/cloud/cloud-init.disabled
- apt-get update && apt-get dist-upgrade -uy && apt-get autoremove --yes --purge && printf '#!/bin/sh\n  rm /etc/cloud/cloud-init.disabled /etc/cloud/cloud.cfg.d/01_debian_cloud.cfg && apt-get purge --yes cloud-init cloud-guest-utils cloud-initramfs-growroot genisoimage isc-dhcp-client && apt-get autoremove --yes --purge && rm -f /etc/cron.hourly/cloud-cleanup && shutdown --reboot +1;   2>&1   logger -t cloud-cleanup\n' > /etc/cron.hourly/cloud-cleanup && chmod +x /etc/cron.hourly/cloud-cleanup && reboot &

The production script I m using is a bit more complicated, but can be downloaded as vello-vm. Happy hacking!

Next.