Despite its very limited feature set, I feel editing debian/control in emacs is now a much more pleasant experience. Coming back to the features that Otto requested, the above covers a grand total of zero. Sorry, Otto. It is not you, it is me.
- Diagnostics or linting of basic issues.
- Completion suggestions for all known field names that I could think of and values for some fields.
- Folding ranges (untested). This feature enables the editor to "fold" multiple lines. It is often used with multi-line comments and that is the feature currently supported.
- On save, trim trailing whitespace at the end of lines (untested). Might not be registered correctly on the server end.
Notable omission at this time:
- An error marker for syntax errors.
- An error marker for missing a mandatory field like Package or Architecture. This also includes Standards-Version, which is admittedly mandatory by policy rather than tooling falling part.
- An error marker for adding Multi-Arch: same to an Architecture: all package.
- Error marker for providing an unknown value to a field with a set of known values. As an example, writing foo in Multi-Arch would trigger this one.
- Warning marker for using deprecated fields such as DM-Upload-Allowed, or when setting a field to its default value for fields like Essential. The latter rule only applies to selected fields and notably Multi-Arch: no does not trigger a warning.
- Info level marker if a field like Priority duplicates the value of the Source paragraph.
- No errors are raised if a field does not have a value.
- No errors are raised if a field is duplicated inside a paragraph.
- No errors are used if a field is used in the wrong paragraph.
- No spellchecking of the Description field.
- No understanding that Foo and X[CBS]-Foo are related. As an example, XC-Package-Type is completely ignored despite being the old name for Package-Type.
- Quick fixes to solve these problems... :)
Obviously, the setup should get easier over time. The first three bullet points should eventually get resolved by merges and upload meaning you end up with an apt install command instead of them. For the editor part, I would obviously love it if we can add snippets for editors to make the automatically pick up the language server when the relevant file is installed.
- Build and install the deb of the main branch of pygls from https://salsa.debian.org/debian/pygls The package is in NEW and hopefully this step will soon just be a regular apt install.
- Build and install the deb of the rts-locatable branch of my python-debian fork from https://salsa.debian.org/nthykier/python-debian There is a draft MR of it as well on the main repo.
- Build and install the deb of the lsp-support branch of debputy from https://salsa.debian.org/debian/debputy
- Configure your editor to run debputy lsp debian/control as the language server for debian/control. This is depends on your editor. I figured out how to do it for emacs (see below). I also found a guide for neovim at https://neovim.io/doc/user/lsp. Note that debputy can be run from any directory here. The debian/control is a reference to the file format and not a concrete file in this case.
(with-eval-after-load 'eglot
(add-to-list 'eglot-server-programs
'(debian-control-mode . ("debputy" "lsp" "debian/control"))))
/C=BE/O=GlobalSign nv-sa/CN=AlphaSSL CA - SHA256 - G4 /C=GB/ST=Greater Manchester/L=Salford/O=Sectigo Limited/CN=Sectigo RSA Domain Validation Secure Server CA /C=GB/ST=Greater Manchester/L=Salford/O=Sectigo Limited/CN=Sectigo RSA Organization Validation Secure Server CA /C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2 /C=US/ST=Arizona/L=Scottsdale/O=Starfield Technologies, Inc./OU=http://certs.starfieldtech.com/repository//CN=Starfield Secure Certificate Authority - G2 /C=AT/O=ZeroSSL/CN=ZeroSSL RSA Domain Secure Site CA /C=BE/O=GlobalSign nv-sa/CN=GlobalSign GCC R3 DV TLS CA 2020Rather than try to work with raw issuers (because, as Andrew Ayer says, The SSL Certificate Issuer Field is a Lie), I mapped these issuers to the organisations that manage them, and summed the counts for those grouped issuers together.
Issuer | Compromised Count |
---|---|
Sectigo | 170 |
ISRG (Let's Encrypt) | 161 |
GoDaddy | 141 |
DigiCert | 81 |
GlobalSign | 46 |
Entrust | 3 |
SSL.com | 1 |
Issuer | Issuance Volume | Compromised Count | Compromise Rate |
---|---|---|---|
Sectigo | 88,323,068 | 170 | 1 in 519,547 |
ISRG (Let's Encrypt) | 315,476,402 | 161 | 1 in 1,959,480 |
GoDaddy | 56,121,429 | 141 | 1 in 398,024 |
DigiCert | 144,713,475 | 81 | 1 in 1,786,586 |
GlobalSign | 1,438,485 | 46 | 1 in 31,271 |
Entrust | 23,166 | 3 | 1 in 7,722 |
SSL.com | 171,816 | 1 | 1 in 171,816 |
Issuer | Issuance Volume | Compromised Count | Compromise Rate |
---|---|---|---|
Entrust | 23,166 | 3 | 1 in 7,722 |
GlobalSign | 1,438,485 | 46 | 1 in 31,271 |
SSL.com | 171,816 | 1 | 1 in 171,816 |
GoDaddy | 56,121,429 | 141 | 1 in 398,024 |
Sectigo | 88,323,068 | 170 | 1 in 519,547 |
DigiCert | 144,713,475 | 81 | 1 in 1,786,586 |
ISRG (Let's Encrypt) | 315,476,402 | 161 | 1 in 1,959,480 |
SELECT SUM(sub.NUM_ISSUED[2] - sub.NUM_EXPIRED[2]) FROM ( SELECT ca.name, max(coalesce(coalesce(nullif(trim(cc.SUBORDINATE_CA_OWNER), ''), nullif(trim(cc.CA_OWNER), '')), cc.INCLUDED_CERTIFICATE_OWNER)) as OWNER, ca.NUM_ISSUED, ca.NUM_EXPIRED FROM ccadb_certificate cc, ca_certificate cac, ca WHERE cc.CERTIFICATE_ID = cac.CERTIFICATE_ID AND cac.CA_ID = ca.ID GROUP BY ca.ID ) sub WHERE sub.name ILIKE '%Amazon%' OR sub.name ILIKE '%CloudFlare%' AND sub.owner = 'DigiCert';The number I get from running that query is 104,316,112, which should be subtracted from DigiCert s total issuance figures to get a more accurate view of what DigiCert s regular customers do with their private keys. When I do this, the compromise rates table, sorted by the compromise rate, looks like this:
Issuer | Issuance Volume | Compromised Count | Compromise Rate |
---|---|---|---|
Entrust | 23,166 | 3 | 1 in 7,722 |
GlobalSign | 1,438,485 | 46 | 1 in 31,271 |
SSL.com | 171,816 | 1 | 1 in 171,816 |
GoDaddy | 56,121,429 | 141 | 1 in 398,024 |
"Regular" DigiCert | 40,397,363 | 81 | 1 in 498,732 |
Sectigo | 88,323,068 | 170 | 1 in 519,547 |
All DigiCert | 144,713,475 | 81 | 1 in 1,786,586 |
ISRG (Let's Encrypt) | 315,476,402 | 161 | 1 in 1,959,480 |
The less humans have to do with certificate issuance, the less likely they are to compromise that certificate by exposing the private key. While it may not be surprising, it is nice to have some empirical evidence to back up the common wisdom. Fully-managed TLS providers, such as CloudFlare, AWS Certificate Manager, and whatever Azure s thing is called, is the platonic ideal of this principle: never give humans any opportunity to expose a private key. I m not saying you should use one of these providers, but the security approach they have adopted appears to be the optimal one, and should be emulated universally. The ACME protocol is the next best, in that there are a variety of standardised tools widely available that allow humans to take themselves out of the loop, but it s still possible for humans to handle (and mistakenly expose) key material if they try hard enough. Legacy issuance methods, which either cannot be automated, or require custom, per-provider automation to be developed, appear to be at least four times less helpful to the goal of avoiding compromise of the private key associated with a certificate.
// Don't do this
para.innerHTML = location.href;
Unfortunately, an attacker can now manipulate the URL (and e.g. send
this link in a phishing email), and any HTML tags they add will be
interpreted by the user s browser. This could potentially be used by
the attacker to send private data to a different server.
Detecting DOM-XSS using vulnerability scanning tools is challenging -
typically this requires crawling each page of the website and
attempting to detect problems such as the one above, but there is a
significant risk of false positives, especially as the complexity of
the logic increases.
There are already ways to avoid these exploits developers
should validate untrusted input before making use of it. There are
libraries such as DOMPurify
which can help with sanitization.1
However, if you are part of a security team with responsibility for
preventing these issues, it can be complex to understand whether you
are at risk. Different developer teams may be using different
techniques and tools. It may be impossible for you to work closely
with every developer so how can you know that the frontend
team have used these libraries correctly?
.innerHTML
and other dangerous
properties. Provided these special types are created in an
appropriate way, then they can be trusted not to expose XSS
problems.
This approach will work with whichever tools the frontend developers have
chosen to use, and detection of issues can be rolled out by
infrastructure engineers without requiring frontend code changes.
Content-Security-Policy-Report-Only
headers)Publisher: | Tordotcom |
Copyright: | 2023 |
ISBN: | 1-250-83499-6 |
Format: | Kindle |
Pages: | 438 |
Welcome to the June 2023 report from the Reproducible Builds project In our reports, we outline the most important things that we have been up to over the past month. As always, if you are interested in contributing to the project, please visit our Contribute page on our website.
Corrupted build environments can deliver compromised cryptographically signed binaries. Several exploits in critical supply chains have been demonstrated in recent years, proving that this is not just theoretical. The most well secured build environments are still single points of failure when they fail. [ ] This talk will focus on the state of the art from several angles in related Free and Open Source Software projects, what works, current challenges and future plans for building trustworthy toolchains you do not need to trust.Hosted by the Software Freedom Conservancy and taking place in Portland, Oregon, FOSSY aims to be a community-focused event: Whether you are a long time contributing member of a free software project, a recent graduate of a coding bootcamp or university, or just have an interest in the possibilities that free and open source software bring, FOSSY will have something for you . More information on the event is available on the FOSSY 2023 website, including the full programme schedule.
The 2020 Solarwinds attack was a tipping point that caused a heightened awareness about the security of the software supply chain and in particular the large amount of trust placed in build systems. Reproducible Builds (R-Bs) provide a strong foundation to build defenses for arbitrary attacks against build systems by ensuring that given the same source code, build environment, and build instructions, bitwise-identical artifacts are created.However, in contrast to other papers that touch on some theoretical aspect of reproducible builds, the authors paper takes a different approach. Starting with the observation that much of the software industry believes R-Bs are too far out of reach for most projects and conjoining that with a goal of to help identify a path for R-Bs to become a commonplace property , the paper has a different methodology:
We conducted a series of 24 semi-structured expert interviews with participants from the Reproducible-Builds.org project, and iterated on our questions with the reproducible builds community. We identified a range of motivations that can encourage open source developers to strive for R-Bs, including indicators of quality, security benefits, and more efficient caching of artifacts. We identify experiences that help and hinder adoption, which heavily include communication with upstream projects. We conclude with recommendations on how to better integrate R-Bs with the efforts of the open source and free software community.A PDF of the paper is now available, as is an entry on the CISPA Helmholtz Center for Information Security website and an entry under the TeamUSEC Human-Centered Security research group.
comp.unix.programming
. Larry notes that it starts with Jayan asking about comparing binaries that might have difference in their embedded timestamps (that is, perhaps, Foreshadowing diffoscope, amiright? ) and goes on to observe that:
The antagonist is David Schwartz, who correctly says There are dozens of complex reasons why what seems to be the same sequence of operations might produce different end results, but goes on to say I totally disagree with your general viewpoint that compilers must provide for reproducability [sic]. Dwight Tovey and I (Larry Doolittle) argue for reproducible builds. I assert Any program especially a mission-critical program like a compiler that cannot reproduce a result at will is broken. Also it s commonplace to take a binary from the net, and check to see if it was trojaned by attempting to recreate it from source.
SOURCE_DATE_EPOCH
environment variable [ ], Chris Lamb made it easier to parse our summit announcement at a glance [ ], Mattia Rizzolo added the summit announcement at a glance [ ] itself [ ][ ][ ] and Rahul Bajaj added a taxonomy of variations in build environments [ ].
randomness_in_documentation_generated_by_mkdocs
toolchain issue was added by Chris Lamb [ ], and the deterministic
flag on the paths_vary_due_to_usrmerge
issue as we are not currently testing usrmerge
issues [ ] issues.
bullseye
, bookworm
, trixie
and sid
, but he also mentioned amongst many changes that not only are the non-free
images being built (and are reproducible) but that the live images are generated officially by Debian itself. [ ]
CFLAGS
environment variable. [ ]
bcachefs
(sort find / filesys)build-compare
(reports files as identical)build-time
(toolchain date)cockpit
(merged, gzip mtime)gcc13
(gcc13 toolchain LTO parallelism)ghc-rpm-macros
(toolchain parallelism)golangcli-lint
(date)gutenprint
(date+time)mage
(date (golang))mumble
(filesys)pcr
(date)python-nss
(drop sphinx .doctrees)python310
(merged, bisected+backported)warpinator
(merged, date)xroachng
(date)elinks
.multipath-tools
.mkdocstrings-python-handlers
.fribidi
.jtreg7
.python-bitstring
(forwarded upstream).gradle-kotlin-dsl
.libsdl-console
.kawari8
.freetds
.gbrowse
.bglibs
.advi
.afterstep
.simstring
.manderlbot
.erlang-proper
.comedilib
.libint
.newlib
.binutils-msp430
.c-munipack
.python-marshmallow-sqlalchemy
.mplayer
.menu
.mini-buildd
.pnetcdf
.liblopsub
.wcc
.shotcut
.icu
.libapache-poi-java
.atf
.valgrind
.amd64
, armhf
, and i386
architectures to Debian bookworm, with the exception of the Jenkins host itself which will be upgraded after the release of Debian 12.1. In addition, Mattia Rizzolo updated the email configuration for the @reproducible-builds.org
domain to correctly accept incoming mails from jenkins.debian.net
[ ] as well as to set up DomainKeys Identified Mail (DKIM) signing [ ]. And working together with Holger, Mattia also updated the Jenkins configuration to start testing Debian trixie which resulted in stopped testing Debian buster. And, finally, Jan-Benedict Glaw contributed patches for improved NetBSD testing.
#reproducible-builds
on irc.oftc.net
.
rb-general@lists.reproducible-builds.org
# header1
Some top-level stuff.
## header2
Some more details.
## header2
Some more things on a related topic.
# header2
Here I'm trying to breakup sections, so there is a "big gap" between H1 and smaller gaps between the lesser-level headings.
After going over my init file recently, making some changes, I noticed that the spacing was not at all consistent. So I figured "How hard could it be to recognize headers and insert/remove newlines before them?"
A trivial regexp search for "^#
" identifies headers, and that counting the "#
" characters lets you determine their depth. From their removing any previous newlines is the work of a moment, and inserting the appropriate number to ensure consistency is simple.
I spent 15 minutes writing the initial implementation, which was markdown-specific, then another 30 minutes adding support for org-mode files - because my work-diary is written using the org-diary package (along with other helpers, such as the org-tag-cloud.
Anyway the end result is that now when I save a markdown/org file the headers are updated automatically:
Welcome to the first report for 2023 from the Reproducible Builds project! In these reports we try and outline the most important things that we have been up to over the past month, as well as the most important things in/around the community. As a quick recap, the motivation behind the reproducible builds effort is to ensure no malicious flaws can be deliberately introduced during compilation and distribution of the software that we run on our devices. As ever, if you are interested in contributing to the project, please visit our Contribute page on our website.
the default compression for Git archives has recently changed. As result, archives downloaded from GitHub may have different checksums even though the contents are completely unchanged.This change (which was brought up on our mailing list last October) would have had quite wide-ranging implications for anyone wishing to validate and verify downloaded archives using cryptographic signatures. However, GitHub reversed this decision, updating their original announcement with a message that We are reverting this change for now. More details to follow. It appears that this was informed in part by an in-depth discussion in the GitHub Community issue tracker.
In this report, the software supply chains of the most popular Ethereum clients are cataloged and analyzed. The dependency graphs of Ethereum clients developed in Go, Rust, and Java, are studied. These client are Geth, Prysm, OpenEthereum, Lighthouse, Besu, and Teku. To do so, their dependency graphs are transformed into a unified format. Quantitative metrics are used to depict the software supply chain of the blockchain. The results show a clear difference in the size of the software supply chain required for the execution layer and consensus layer of Ethereum.
Dockerfile
and that my talk will also mention how to pin the apt/dnf/apk/pacman packages with my repro-get
tool.
SOURCE_DATE_EPOCH
environment variable this month. This means that release tarballs of the Signal desktop client do not embed nondeterministic release information. [ ][ ]
In response to [ ] criticisms, we started encouraging new apps to enable reproducible builds. It turns out that reproducible builds are not so difficult to achieve for many apps. In the past few months we ve gotten many more reproducible apps in F-Droid than before. Currently we can t highlight which apps are reproducible in the client, so maybe you haven t noticed that there are many new apps signed with upstream developers keys.(There was a discussion about this post on Hacker News.) In addition:
baseline.profm
files are nondeterministic, developed a workaround, and provided all the details required for a fix. As they note, this issue has now been fixed but the fix is not yet part of an official Android Gradle plugin release.
.dex
files. [ ]
0.2.0
and 0.2.1
releases of reproducible-apk-tools, a suite of tools to help make .apk
files reproducible. Several new subcommands and scripts were added, and a number of bugs were fixed as well [ ][ ]. They also updated the F-Droid website to improve the reproducibility-related documentation. [ ][ ]
31.0.0
and 32.0.0
(unlike earlier and later versions) have a zipalign
command that produces incorrect padding.
.zip
entries in .apk
files [ ] and then newline differences between building on Windows versus Linux that can make builds not reproducible as well. [ ] (Note that these links may require a Google account to view.)
mc
(#828683)gtk-sharp3
(#989965 & #989966)tracker.debian.org
service display results of reproducible rebuilds, not just reproducible CI results.
Elsewhere in Debian, strip-nondeterminism is our tool to remove specific non-deterministic results from a completed build. This month, version 1.13.1-1
was uploaded to Debian unstable by Holger Levsen, including a fix by FC Stegerman (obfusk) to update a regular expression for the latest version of file(1)
[ ]. (#1028892)
Lastly, 65 reviews of Debian packages were added, 21 were updated and 35 were removed this month adding to our knowledge about identified issues.
SOURCE_DATE_EPOCH
. [ ]
231
, 232
, 233
and 234
to Debian:
from __future__ import print_function
import anymore. [ ]extras_require.json
handling. [ ]Recommends
into a separate Python script. [ ]debian/tests/control
after merging support for PyPDF support. [ ]cd-iccdump
binary. [ ]test_text_proper_indentation
test to support the latest version(s) of file(1)
. [ ]extras_require.json
file to store some build/release metadata, instead of accessing the internet. [ ]file(1)
regular expression. [ ]argparse
(date-related issue)asciimatics
(build failure)asyncpg
(fails to build in 2032)cpython
(fails to build in 2038)django
(fails to build in 2038)grandorgue
(.zip
-related issue)libarchive
(fails to build in 2038)libarchive
(fails to build in 2038)librcc
(date)mbedtls
(fails to build in 2023)mozilla-nss
(fails to build in 2023)ocaml-rpm-macros
(fix fallout from an RPM-related change)perl HTTP::Cookies
(fails to build in 2038)python-aiosmtplib/python-trustme
(fails to build in 2038 due to SSL certificate)python-bmap
(fails to build in 2024)python-compileall2
(fails to build in 2038)taskwarrior
(python-tasklib
fails to build in 2038)taskwarrior
(fix fails to build in 2038)wrk
(hash ordering issue)xemacs
(fails to build in 2038 stuck)click
.towncrier
.unifrac-tools
.hamster-time-tracker
.accel-config
.python-miio
.python-graphviz
.ectrans
.fiat-ecmwf
.node-katex
.aribas
.tbox
.borgbackup2
.dnf-plugins-core
.refpolicy
.file(1)
(which is used by reproducible builds tools like diffoscope and strip-nondeterminism) that improve detection of various file formats are now included in the Debian packaging. [ ].json
or .txt
) for LTS suites and older in order to save diskspace on the Jenkins host. [ ]pbuilder
base less frequently for the stretch
, bookworm
and experimental
suites. [ ]chroot-run
script to correctly manage /dev
and /dev/pts
. [ ][ ][ ]V=s
flag to enable debugging. [ ]
#reproducible-builds
on irc.oftc.net
.
rb-general@lists.reproducible-builds.org
debdistdiff
for one purpose (comparing two different archives like Trisquel and Ubuntu) and discovered that the tool could be used for another purpose (comparing the Trisquel archive at two different points in time). At this time I realized that I needed a log of all different apt archive metadata to be able to produce an audit log of the differences in time for the archive. I create manually curated git-repositories with the Release/InRelease
and the Packages
files for each architecture/component of the well-known distributions Trisquel, Ubuntu, Debian and PureOS. Eventually I wrote scripts to automate this, which are now published in the debdistget project.
At this point, one of the early question about per-IP substitution of Release files were lingering in my mind. However with the tooling I now had available, coming up with a way to resolve this was simple! Merely have apt compute a SHA256 checksum of the just downloaded InRelease file, and see if my git repository had the same file. At this point I started reading the Apt source code, and now I had more doubts about the security of my systems than I ever had before. Oh boy how the name Apt has never before felt more Apt?! Oh well, we must leave some exercises for the students. Eventually I realized I wanted to touch as little of apt code basis as possible, and noticed the SigVerify::CopyAndVerify function called ExecGPGV which called apt-key verify which called GnuPG s gpgv
. By setting Apt::Key::gpgvcommand
I could get apt-key verify
to call another tool than gpgv
. See where I m going? I thought wrapping this up would now be trivial but for some reason the hash checksum I computed locally never matched what was on my server. I gave up and started working on other things instead.
Today I came back to this idea, and started to debug exactly how the local files looked that I got from apt and how they differed from what I had in my git repositories, that came straight from the apt archives. Eventually I traced this back to SplitClearSignedFile which takes an InRelease
file and splits it into two files, probably mimicking the (old?) way of distributing both Release
and Release.gpg
. So the clearsigned InRelease file is split into one cleartext file (similar to the Release
file) and one OpenPGP signature file (similar to the Release.gpg
file). But why didn t the cleartext variant of the InRelease file hash to the same value as the hash of the Release file? Sadly they differ by the final newline.
Having solved this technicality, wrapping the pieces up was easy, and I came up with a project apt-canary that provides a script apt-canary-gpgv that verify the local apt release files against something I call a apt canary witness file stored at a URL somewhere.
I m now running apt-canary on my Trisquel aramo laptop, a Trisquel nabia server, and Talos II ppc64el Debian machine. This means I have solved the per-IP substitution worries (or at least made them less likely to occur, having to send the same malicious release files to both GitLab and my system), and allow me to have an audit log of all release files that I actually use for installing and downloading packages.
What do you think? There are clearly a lot of work and improvements to be made. This is a proof-of-concept implementation of an idea, but instead of refining it until perfection and delaying feedback, I wanted to publish this to get others to think about the problems and various ways to resolve them.
Btw, I m going to be at FOSDEM 23 this weekend, helping to manage the Security Devroom. Catch me if you want to chat about this or other things. Happy Hacking!
War and Peace (1867) Leo Tolstoy It's strange to think that there is almost no point in reviewing this novel: who hasn't heard of War and Peace? What more could possibly be said about it now? Still, when I was growing up, War and Peace was always the stereotypical example of the 'impossible book', and even start it was, at best, a pointless task, and an act of hubris at worst. And so there surely exists a parallel universe in which I never have and will never will read the book... Nevertheless, let us try to set the scene. Book nine of the novel opens as follows:
On the twelfth of June, 1812, the forces of Western Europe crossed the Russian frontier and war began; that is, an event took place opposed to human reason and to human nature. Millions of men perpetrated against one another such innumerable crimes, frauds, treacheries, thefts, forgeries, issues of false money, burglaries, incendiarisms and murders as in whole centuries are not recorded in the annals of all the law courts of the world, but which those who committed them did not at the time regard as being crimes. What produced this extraordinary occurrence? What were its causes? [ ] The more we try to explain such events in history reasonably, the more unreasonable and incomprehensible they become to us.Set against the backdrop of the Napoleonic Wars and Napoleon's invasion of Russia, War and Peace follows the lives and fates of three aristocratic families: The Rostovs, The Bolkonskys and the Bezukhov's. These characters find themselves situated athwart (or against) history, and all this time, Napoleon is marching ever closer to Moscow. Still, Napoleon himself is essentially just a kind of wallpaper for a diverse set of personal stories touching on love, jealousy, hatred, retribution, naivety, nationalism, stupidity and much much more. As Elif Batuman wrote earlier this year, "the whole premise of the book was that you couldn t explain war without recourse to domesticity and interpersonal relations." The result is that Tolstoy has woven an incredibly intricate web that connects the war, noble families and the everyday Russian people to a degree that is surprising for a book started in 1865. Tolstoy's characters are probably timeless (especially the picaresque adventures and constantly changing thoughts Pierre Bezukhov), and the reader who has any social experience will immediately recognise characters' thoughts and actions. Some of this is at a 'micro' interpersonal level: for instance, take this example from the elegant party that opens the novel:
Each visitor performed the ceremony of greeting this old aunt whom not one of them knew, not one of them wanted to know, and not one of them cared about. The aunt spoke to each of them in the same words, about their health and her own and the health of Her Majesty, who, thank God, was better today. And each visitor, though politeness prevented his showing impatience, left the old woman with a sense of relief at having performed a vexatious duty and did not return to her the whole evening.But then, some of the focus of the observations are at the 'macro' level of the entire continent. This section about cities that feel themselves in danger might suffice as an example:
At the approach of danger, there are always two voices that speak with equal power in the human soul: one very reasonably tells a man to consider the nature of the danger and the means of escaping it; the other, still more reasonably, says that it is too depressing and painful to think of the danger, since it is not in man s power to foresee everything and avert the general course of events, and it is therefore better to disregard what is painful till it comes and to think about what is pleasant. In solitude, a man generally listens to the first voice, but in society to the second.And finally, in his lengthy epilogues, Tolstoy offers us a dissertation on the behaviour of large organisations, much of it through engagingly witty analogies. These epilogues actually turn out to be an oblique and sarcastic commentary on the idiocy of governments and the madness of war in general. Indeed, the thorough dismantling of the 'great man' theory of history is a common theme throughout the book:
During the whole of that period [of 1812], Napoleon, who seems to us to have been the leader of all these movements as the figurehead of a ship may seem to a savage to guide the vessel acted like a child who, holding a couple of strings inside a carriage, thinks he is driving it. [ ] Why do [we] all speak of a military genius ? Is a man a genius who can order bread to be brought up at the right time and say who is to go to the right and who to the left? It is only because military men are invested with pomp and power and crowds of sychophants flatter power, attributing to it qualities of genius it does not possess.Unlike some other readers, I especially enjoyed these diversions into the accounting and workings of history, as well as our narrow-minded way of trying to 'explain' things in a singular way:
When an apple has ripened and falls, why does it fall? Because of its attraction to the earth, because its stalk withers, because it is dried by the sun, because it grows heavier, because the wind shakes it, or because the boy standing below wants to eat it? Nothing is the cause. All this is only the coincidence of conditions in which all vital organic and elemental events occur. And the botanist who finds that the apple falls because the cellular tissue decays and so forth is equally right with the child who stands under the tree and says the apple fell because he wanted to eat it and prayed for it.Given all of these serious asides, I was also not expecting this book to be quite so funny. At the risk of boring the reader with citations, take this sarcastic remark about the ineptness of medicine men:
After his liberation, [Pierre] fell ill and was laid up for three months. He had what the doctors termed 'bilious fever.' But despite the fact that the doctors treated him, bled him and gave him medicines to drink he recovered.There is actually a multitude of remarks that are not entirely complimentary towards Russian medical practice, but they are usually deployed with an eye to the human element involved rather than simply to the detriment of a doctor's reputation "How would the count have borne his dearly loved daughter s illness had he not known that it was costing him a thousand rubles?" Other elements of note include some stunning set literary pieces, such as when Prince Andrei encounters a gnarly oak tree under two different circumstances in his life, and when Nat sha's 'Russian' soul is awakened by the strains of a folk song on the balalaika. Still, despite all of these micro- and macro-level happenings, for a long time I felt that something else was going on in War and Peace. It was difficult to put into words precisely what it was until I came across this passage by E. M. Forster:
After one has read War and Peace for a bit, great chords begin to sound, and we cannot say exactly what struck them. They do not arise from the story [and] they do not come from the episodes nor yet from the characters. They come from the immense area of Russia, over which episodes and characters have been scattered, from the sum-total of bridges and frozen rivers, forests, roads, gardens and fields, which accumulate grandeur and sonority after we have passed them. Many novelists have the feeling for place, [but] very few have the sense of space, and the possession of it ranks high in Tolstoy s divine equipment. Space is the lord of War and Peace, not time.'Space' indeed. Yes, potential readers should note the novel's great length, but the 365 chapters are actually remarkably short, so the sensation of reading it is not in the least overwhelming. And more importantly, once you become familiar with its large cast of characters, it is really not a difficult book to follow, especially when compared to the other Russian classics. My only regret is that it has taken me so long to read this magnificent novel and that I might find it hard to find time to re-read it within the next few years.
Coming Up for Air (1939) George Orwell It wouldn't be a roundup of mine without at least one entry from George Orwell, and, this year, that place is occupied by a book I hadn't haven't read in almost two decades Still, the George Bowling of Coming Up for Air is a middle-aged insurance salesman who lives in a distinctly average English suburban row house with his nuclear family. One day, after winning some money on a bet, he goes back to the village where he grew up in order to fish in a pool he remembers from thirty years before. Less important than the plot, however, is both the well-observed remarks and scathing criticisms that Bowling has of the town he has returned to, combined with an ominous sense of foreboding before the Second World War breaks out. At several times throughout the book, George's placid thoughts about his beloved carp pool are replaced by racing, anxious thoughts that overwhelm his inner peace:
War is coming. In 1941, they say. And there'll be plenty of broken crockery, and little houses ripped open like packing-cases, and the guts of the chartered accountant's clerk plastered over the piano that he's buying on the never-never. But what does that kind of thing matter, anyway? I'll tell you what my stay in Lower Binfield had taught me, and it was this. IT'S ALL GOING TO HAPPEN. All the things you've got at the back of your mind, the things you're terrified of, the things that you tell yourself are just a nightmare or only happen in foreign countries. The bombs, the food-queues, the rubber truncheons, the barbed wire, the coloured shirts, the slogans, the enormous faces, the machine-guns squirting out of bedroom windows. It's all going to happen. I know it - at any rate, I knew it then. There's no escape. Fight against it if you like, or look the other way and pretend not to notice, or grab your spanner and rush out to do a bit of face-smashing along with the others. But there's no way out. It's just something that's got to happen.Already we can hear psychological madness that underpinned the Second World War. Indeed, there is no great story in Coming Up For Air, no wonderfully empathetic characters and no revelations or catharsis, so it is impressive that I was held by the descriptions, observations and nostalgic remembrances about life in modern Lower Binfield, its residents, and how it has changed over the years. It turns out, of course, that George's beloved pool has been filled in with rubbish, and the village has been perverted by modernity beyond recognition. And to cap it off, the principal event of George's holiday in Lower Binfield is an accidental bombing by the British Royal Air Force. Orwell is always good at descriptions of awful food, and this book is no exception:
The frankfurter had a rubber skin, of course, and my temporary teeth weren't much of a fit. I had to do a kind of sawing movement before I could get my teeth through the skin. And then suddenly pop! The thing burst in my mouth like a rotten pear. A sort of horrible soft stuff was oozing all over my tongue. But the taste! For a moment I just couldn't believe it. Then I rolled my tongue around it again and had another try. It was fish! A sausage, a thing calling itself a frankfurter, filled with fish! I got up and walked straight out without touching my coffee. God knows what that might have tasted of.Many other tell-tale elements of Orwell's fictional writing are in attendance in this book as well, albeit worked out somewhat less successfully than elsewhere in his oeuvre. For example, the idea of a physical ailment also serving as a metaphor is present in George's false teeth, embodying his constant preoccupation with his ageing. (Readers may recall Winston Smith's varicose ulcer representing his repressed humanity in Nineteen Eighty-Four). And, of course, we have a prematurely middle-aged protagonist who almost but not quite resembles Orwell himself. Given this and a few other niggles (such as almost all the women being of the typical Orwell 'nagging wife' type), it is not exactly Orwell's magnum opus. But it remains a fascinating historical snapshot of the feeling felt by a vast number of people just prior to the Second World War breaking out, as well as a captivating insight into how the process of nostalgia functions and operates.
Howards End (1910) E. M. Forster Howards End begins with the following sentence:
One may as well begin with Helen s letters to her sister.In fact, "one may as well begin with" my own assumptions about this book instead. I was actually primed to consider Howards End a much more 'Victorian' book: I had just finished Virginia Woolf's Mrs Dalloway and had found her 1925 book at once rather 'modern' but also very much constrained by its time. I must have then unconsciously surmised that a book written 15 years before would be even more inscrutable, and, with its Victorian social mores added on as well, Howards End would probably not undress itself so readily in front of the reader. No doubt there were also the usual expectations about 'the classics' as well. So imagine my surprise when I realised just how inordinately affable and witty Howards End turned out to be. It doesn't have that Wildean shine of humour, of course, but it's a couple of fields over in the English countryside, perhaps abutting the more mordant social satires of the earlier George Orwell novels (see Coming Up for Air above). But now let us return to the story itself. Howards End explores class warfare, conflict and the English character through a tale of three quite different families at the beginning of the twentieth century: the rich Wilcoxes; the gentle & idealistic Schlegels; and the lower-middle class Basts. As the Bloomsbury Group Schlegel sisters desperately try to help the Basts and educate the rich but close-minded Wilcoxes, the three families are drawn ever closer and closer together. Although the whole story does, I suppose, revolve around the house in the title (which is based on the Forster's own childhood home), Howards End is perhaps best described as a comedy of manners or a novel that shows up the hypocrisy of people and society. In fact, it is surprising how little of the story actually takes place in the eponymous house, with the overwhelming majority of the first half of the book taking place in London. But it is perhaps more illuminating to remark that the Howards End of the book is a house that the Wilcoxes who own it at the start of the novel do not really need or want. What I particularly liked about Howards End is how the main character's ideals alter as they age, and subsequently how they find their lives changing in different ways. Some of them find themselves better off at the end, others worse. And whilst it is also surprisingly funny, it still manages to trade in heavier social topics as well. This is apparent in the fact that, although the characters themselves are primarily in charge of their own destinies, their choices are still constrained by the changing world and shifting sense of morality around them. This shouldn't be too surprising: after all, Forster's novel was published just four years before the Great War, a distinctly uncertain time. Not for nothing did Virginia Woolf herself later observe that "on or about December 1910, human character changed" and that "all human relations have shifted: those between masters and servants, husbands and wives, parents and children." This process can undoubtedly be seen rehearsed throughout Forster's Howards End, and it's a credit to the author to be able to capture it so early on, if not even before it was widespread throughout Western Europe. I was also particularly taken by Forster's fertile use of simile. An extremely apposite example can be found in the description Tibby Schlegel gives of his fellow Cambridge undergraduates. Here, Timmy doesn't want to besmirch his lofty idealisation of them with any banal specificities, and wishes that the idea of them remain as ideal Platonic forms instead. Or, as Forster puts it, to Timmy it is if they are "pictures that must not walk out of their frames." Wilde, at his most weakest, is 'just' style, but Forster often deploys his flair for a deeper effect. Indeed, when you get to the end of this section mentioning picture frames, you realise Forster has actually just smuggled into the story a failed attempt on Tibby's part to engineer an anonymous homosexual encounter with another undergraduate. It is a credit to Forster's sleight-of-hand that you don't quite notice what has just happened underneath you and that the books' reticence to honestly describe what has happened is thus structually analogus Tibby's reluctance to admit his desires to himself. Another layer to the character of Tibby (and the novel as a whole) is thereby introduced without the imposition of clumsy literary scaffolding. In a similar vein, I felt very clever noticing the arch reference to Debussy's Pr lude l'apr s-midi d'un faune until I realised I just fell into the trap Forster set for the reader in that I had become even more like Tibby in his pseudo-scholarly views on classical music. Finally, I enjoyed that each chapter commences with an ironic and self-conscious bon mot about society which is only slightly overblown for effect. Particularly amusing are the ironic asides on "women" that run through the book, ventriloquising the narrow-minded views of people like the Wilcoxes. The omniscient and amiable narrator of the book also recalls those ironically distant voiceovers from various French New Wave films at times, yet Forster's narrator seems to have bigger concerns in his mordant asides: Forster seems to encourage some sympathy for all of the characters even the more contemptible ones at their worst moments. Highly recommended, as are Forster's A Room with a View (1908) and his slightly later A Passage to India (1913).
The Good Soldier (1915) Ford Madox Ford The Good Soldier starts off fairly simply as the narrator's account of his and his wife's relationship with some old friends, including the eponymous 'Good Soldier' of the book's title. It's an experience to read the beginning of this novel, as, like any account of endless praise of someone you've never met or care about, the pages of approving remarks about them appear to be intended to wash over you. Yet as the chapters of The Good Soldier go by, the account of the other characters in the book gets darker and darker. Although the author himself is uncritical of others' actions, your own critical faculties are slowgrly brought into play, and you gradully begin to question the narrator's retelling of events. Our narrator is an unreliable narrator in the strict sense of the term, but with the caveat that he is at least is telling us everything we need to know to come to our own conclusions. As the book unfolds further, the narrator's compromised credibility seems to infuse every element of the novel even the 'Good' of the book's title starts to seem like a minor dishonesty, perhaps serving as the inspiration for the irony embedded in the title of The 'Great' Gatsby. Much more effectively, however, the narrator's fixations, distractions and manner of speaking feel very much part of his dissimulation. It sometimes feels like he is unconsciously skirting over the crucial elements in his tale, exactly like one does in real life when recounting a story containing incriminating ingredients. Indeed, just how much the narrator is conscious of his own concealment is just one part of what makes this such an interesting book: Ford Madox Ford has gifted us with enough ambiguity that it is also possible that even the narrator cannot find it within himself to understand the events of the story he is narrating. It was initially hard to believe that such a carefully crafted analysis of a small group of characters could have been written so long ago, and despite being fairly easy to read, The Good Soldier is an almost infinitely subtle book even the jokes are of the subtle kind and will likely get a re-read within the next few years.
Anna Karenina (1878) Leo Tolstoy There are many similar themes running through War and Peace (reviewed above) and Anna Karenina. Unrequited love; a young man struggling to find a purpose in life; a loving family; an overwhelming love of nature and countless fascinating observations about the minuti of Russian society. Indeed, rather than primarily being about the eponymous Anna, Anna Karenina provides a vast panorama of contemporary life in Russia and of humanity in general. Nevertheless, our Anna is a sophisticated woman who abandons her empty existence as the wife of government official Alexei Karenin, a colourless man who has little personality of his own, and she turns to a certain Count Vronsky in order to fulfil her passionate nature. Needless to say, this results in tragic consequences as their (admittedly somewhat qualified) desire to live together crashes against the rocks of reality and Russian society. Parallel to Anna's narrative, though, Konstantin Levin serves as the novel's alter-protagonist. In contrast to Anna, Levin is a socially awkward individual who straddles many schools of thought within Russia at the time: he is neither a free-thinker (nor heavy-drinker) like his brother Nikolai, and neither is he a bookish intellectual like his half-brother Serge. In short, Levin is his own man, and it is generally agreed by commentators that he is Tolstoy's surrogate within the novel. Levin tends to come to his own version of an idea, and he would rather find his own way than adopt any prefabricated view, even if confusion and muddle is the eventual result. In a roughly isomorphic fashion then, he resembles Anna in this particular sense, whose story is a counterpart to Levin's in their respective searches for happiness and self-actualisation. Whilst many of the passionate and exciting passages are told on Anna's side of the story (I'm thinking horse race in particular, as thrilling as anything in cinema ), many of the broader political thoughts about the nature of the working classes are expressed on Levin's side instead. These are stirring and engaging in their own way, though, such as when he joins his peasants to mow the field and seems to enter the nineteenth-century version of 'flow':
The longer Levin mowed, the more often he felt those moments of oblivion during which it was no longer his arms that swung the scythe, but the scythe itself that lent motion to his whole body, full of life and conscious of itself, and, as if by magic, without a thought of it, the work got rightly and neatly done on its own. These were the most blissful moments.Overall, Tolstoy poses no didactic moral message towards any of the characters in Anna Karenina, and merely invites us to watch rather than judge. (Still, there is a hilarious section that is scathing of contemporary classical music, presaging many of the ideas found in Tolstoy's 1897 What is Art?). In addition, just like the earlier War and Peace, the novel is run through with a number of uncannily accurate observations about daily life:
Anna smiled, as one smiles at the weaknesses of people one loves, and, putting her arm under his, accompanied him to the door of the study.... as well as the usual sprinkling of Tolstoy's sardonic humour ("No one is pleased with his fortune, but everyone is pleased with his wit."). Fyodor Dostoyevsky, the other titan of Russian literature, once described Anna Karenina as a "flawless work of art," and if you re only going to read one Tolstoy novel in your life, it should probably be this one.
fspl prepare
creates a Filespooler job packet and emits it to stdout. It includes a small header (<100 bytes in most cases) that includes the sequence number, creation timestamp, and some other useful metadata.fspl prepare
fspl prepare
fspl prepare
, which can optionally be passed to the processing command at the receiver.-q
):
sender$ fspl queue-init -q ~/sync/b64queue
sender$ echo Hi fspl prepare -s ~/b64seq -i - fspl queue-write -q ~/sync/b64queue
fspl prepare
.fspl prepare
takes two parameters:
-s seqfile
gives the path to a sequence file used on the sender side. This file has a simple number in it that increments a unique counter for every generated job file. It is matched with the nextseq
file within the queue to make sure that the receiver processes jobs in the correct order. It MUST be separate from the file that is in the queue and should NOT be placed within the queue. There is no need to sync this file, and it would be ideal to not sync it.-i
option tells fspl prepare
to read a file for the packet payload. -i -
tells it to read stdin for this purpose. So, the payload will consist of three bytes: Hi\n (that is, including the terminating newline that echo
wrote)fspl prepare
writes the packet to its stdout. We pipe that into fspl queue-write
:
fspl queue-write
reads stdin and writes it to a file in the queue directory in a safe manner. The file will ultimately match the fspl-*.fspl
pattern and have a random string in the middle.receiver$ fspl queue-ls -q ~/sync/b64queue
ID creation timestamp filename
1 2022-05-16T20:29:32-05:00 fspl-7b85df4e-4df9-448d-9437-5a24b92904a4.fspl
receiver$ $ fspl queue-info -q ~/sync/b64queue -j 1
FSPL_SEQ=1
FSPL_CTIME_SECS=1652940172
FSPL_CTIME_NANOS=94106744
FSPL_CTIME_RFC3339_UTC=2022-05-17T01:29:32Z
FSPL_CTIME_RFC3339_LOCAL=2022-05-16T20:29:32-05:00
FSPL_JOB_FILENAME=fspl-7b85df4e-4df9-448d-9437-5a24b92904a4.fspl
FSPL_JOB_QUEUEDIR=/home/jgoerzen/sync/b64queue
FSPL_JOB_FULLPATH=/home/jgoerzen/sync/b64queue/jobs/fspl-7b85df4e-4df9-448d-9437-5a24b92904a4.fspl
receiver$ fspl queue-process -q ~/sync/b64queue --allow-job-params base64
SGkK
--allow-job-params
says that the sender is trusted to supply additional parameters for the command we will be running.base64
is the name of the command that we will run for every job. It will:
queue-info
fspl queue-process
doesn t do anything special with the output; see Handling Filespooler Command Output for details on other options. So, the base64-encoded version of our string is SGkK . We successfully sent a packet using Syncthing as a transport mechanism!
At this point, if you do a fspl queue-ls
again, you ll see the queue is empty. By default, fspl queue-process
deletes jobs that have been successfully processed.
$ arm-none-eabi-gcc -mcpu=cortex-m7 -mfloat-abi=hard
It should be fairly easy to extend this to other targets, but for now,
that's what I've got working. There's a cross-cortex-m7.txt file
containing all of the cross compilation details which is used when
running meson setup
.
All of the Picolibc-specific files live in a new picolibc directory so
they are isolated from the main glibc code.
Pre-loading a pile of hacks
Adapt Picolibc to support the Glibc test code required a bunch of
random hacks, from providing _unlocked
versions of the stdio macros
to stubbing out various unsupportable syscalls (like sleep
and
chdir
). Instead of modifying the Glibc code, I created a file called
hacks.h
which is full of this stuff and used the gcc -include
parameter to read that into the compiler before starting compilation
on all of the files.
Supporting command line parameters
The glibc tests all support a range of command line parameters, some
of which turned out to be quite useful for this work. Picolibc had
limited semihosting support for accessing the command line, but that
required modifying applications to go fetch the command line using a
special semihosting function.
To make this easier, I added a new crt0 variant for picolibc called
(oddly) semihost. This extends the existing hosted variant by
adding a call to the semihosting library to fetch the current command
line and splitting that into words at each space. It doesn't handle
any quoting, but it's sufficient for the needs here.
Avoiding glibc headers
The glibc tests use some glibc-specific extensions to the standard
POSIX C library, so I needed to include those in the test
builds. Headers for those extensions are mixed in with the rest of the
POSIX standard headers, which conflict with the Picolibc versions. To
work around this, I stuck stub #include files in the picolibc directory
which directly include the appropriate headers for the glibc API
extensions. This includes things like argp.h
and
array_length.h
. For other headers which weren't actually needed for
picolibc, I created empty files.
Adding more POSIX to Picolibc
At this point, code was compiling but failing to find various standard
POSIX functions which aren't available in Picolibc. That included some
syscalls which could be emulated using semihosting, like
gettimeofday
and getpagesize
. It also included some generally
useful additions, like replacing ecvtbuf
and fcvtbuf
with ecvt_r
and fcvt_r
. The _r
variants provide a target buffer size instead
of assuming that it was large enough as the Picolibc buf
variants
did.
Which tests are working?
So far, I've got some of the tests in malloc, math, misc and
stdio-common running.
There are a lot of tests in the malloc directory which cover glibc API
extensions or require POSIX syscalls not supported by semihosting. I
think I've added all of the tests which should be supported.
For the math tests, I'm testing the standard POSIX math APIs in both
float and double forms, except for the Bessel and Gamma
functions. Picolibc's versions of those are so bad that they violate
some pretty basic assumptions about error bounds built into the glibc
test code. Until Picolibc gets better versions of these functions,
we'll have to skip testing them this way.
In the misc directory, I've only added tests for ecvt, fcvt, gcvt,
dirname and hsearch. I don't think there are any other tests
there which should work.
Finally, for stdio-common, almost all of the tests expect a fully
functioning file system, which semihosting really can't support. As a
result, we're only able to run the printf, scanf and some of the
sprintf tests.
All in all, we're running 78 of the glibc test programs, which is a
small fraction of the total tests, but I think it's the bulk of the
tests which cover APIs that don't depend too much on the underlying
POSIX operating system.
Bugs found and fixed in Picolibc
This exercise has resulted in 17 fixes in Picolibc, which can be
classified as:
return x + x;
instead of
return x;
.make -j
. Thanks to Richard
Kettlewell for the path.
ovdb
overview storage method: the maxlisten
parameter now permits configuring their listen backlog, whose
previously hard-coded values were 128
for nnrpd and
25
for the others, which was not high enough for some uses.
The default value is now 128
for all of them, and configurable
in inn.conf. Thanks to Kevin Bowling for the patch.
Next.