Prior art We considered both code-signing (Debian s current implementation) and lp-signing (Ubuntu s current implementation) as prior art. Neither was quite suitable for various reasons.

• code-signing relies on polling a configured URL for each archive to fetch a GPG-signed list of signing requests, which would have been awkward for us to set up, and it assumes that unsigned packages are sufficiently trusted for it to be able to run dpkg -x and dpkg-source -b on them outside any containment. dpkg -x has had the occasional security vulnerability, so this seemed unwise for a service that might need to deal with signing packages for multiple customers.
• lp-signing is a microservice accepting authenticated requests, and is careful to avoid needing to manipulate packages itself. However, this relies on a different and incompatible mechanism for indicating that packages should be signed, which wasn t something we wanted to introduce in ELTS.

Workers Debusine already had an established system of external workers that run tasks under various kinds of containment. This seems like a good fit: after all, what s a request to sign a package but a particular kind of task? But there are some problems here: workers can run essentially arbitrary code (such as build scripts in source packages), and even though that s under containment, we don t want to give such machines access to highly-sensitive data such as private keys. Fortunately, we d already introduced the idea of different kinds of workers a few months beforehand, in order to be able to run privileged server tasks that have direct access to the Debusine database. We built on that and added signing workers , which are much like external workers except that they only run signing tasks, no other types of tasks run on them, and they have access to a private database with information about the keys managed by their Debusine instance. (Django s support for multiple databases made this quite easy to arrange: we were able to keep everything in the same codebase.)

Key management It s obviously bad practice to store private key material in the clear, but at the same time the signing workers are essentially oracles that will return signatures on request while ensuring that the rest of Debusine has no access to private key material, so they need to be able to get hold of it themselves. Hardware security modules (HSMs) are designed for this kind of thing, but they can be inconvenient to manage when large numbers of keys are involved. Some keys are more valuable than others. If the signing key used for an experimental archive leaks, the harm is unlikely to be particularly serious; but if the ELTS signing key leaks, many customers will be affected. To match this, we implemented two key protection arrangements for the time being: one suitable for low-value keys encrypts the key in software with a configured key and stores the public key and ciphertext in the database, while one suitable for high-value keys stores keys as PKCS #11 URIs that can be set up manually by an instance administrator. We packaged some YubiHSM tools to make this easier for our sysadmins. The signing worker calls back to the Debusine server to check whether a given work request is authorized to use a given signing key. All operations related to private keys also produce an audit log entry in the private signing database, so we can track down any misuse.

Tasks Getting Debusine to do anything new usually requires figuring out how to model the operation as a task. In this case, that was complicated by wanting to run as little code as possible on the signing workers: in particular, we didn t want to do all the complicated package manipulations there. The approach we landed on was a chain of three tasks:

• ExtractForSigning runs on a normal external worker. It takes the result of a package build and picks out the individual files from it that need to be signed, storing them as separate artifacts.
• Sign runs on a signing worker, and (of course) makes the actual signatures, storing them as artifacts.
• AssembleSignedSource runs on a normal external worker. It takes the signed artifacts and produces a source package containing them, based on the template found in the unsigned binary package.

Workflows Of course, we don t want people to have to create all those tasks directly and figure out how to connect everything together for themselves, and that s what workflows are good at. The make_signed_source workflow does all the heavy lifting of creating the right tasks with the right input data and making them depend on each other in the right ways, including fanning out multiple copies of all this if there are multiple architectures or multiple template packages involved. Since you probably don t want to stop at just having the signed source packages, it also kicks off builds to produce signed binary packages. Even this is too low-level for most people to use directly, so we wrapped it all up in our debian_pipeline workflow, which just needs to be given a few options to enable signing support (and those options can be locked down by workspace owners).

What s next? In most cases this work has been enough to allow ELTS to carry on issuing kernel security updates without too much disruption, which was the main goal; but there are other uses for a signing system. We included OpenPGP support from early on, which allows Debusine to sign its own builds, and we ll soon be extending that to sign APT repositories hosted by Debusine. The current key protection arrangements could use some work. Supporting automatically-generated software-encrypted keys and manually-generated keys in an HSM is fine as far as it goes, but it would be good to be able to have the best of both worlds by being able to automatically generate keys protected by an HSM. This needs some care, as HSMs often have quite small limits on the number of objects they can store at any one time, and the usual workaround is to export keys from the HSM under wrap (encrypted by a key known only to the HSM) so that they can be imported only when needed. We have a general idea of how to do this, but doing it efficiently will need care. We d be very interested in hearing from organizations that need this sort of thing, especially for Debian derivatives. Debusine provides lots of other features that can help you. Please get in touch with us at sales@freexian.com if any of this sounds useful to you.

Writing my proposal Before you can start writing a proposal, you need to select an organization you want to work with. Since many organizations participate in GSoC, I ve used the following criteria to narrow things down for me:

• Programming language familiarity: For me only Python (preferably) as well as shell and Go projects would have made sense. While learning another programming language is cool, I wouldn t be as effective and helpful to the project as someone who is proficient in the language already.
• Standing of the organization: Some of the organizations participating in GSoC are well-known for the outstanding quality of the software they produce. Debian is one of them, but so is e.g. the Django Foundation or PostgreSQL. And my thinking was that the higher the quality of the organization, the more there is to learn for me as a GSoC student.
• Mentor interactions: Apart from the advantage you get from mentor feedback when writing your proposal (more on that further below), it is also helpful to gauge how responsive/helpful your potential mentor is during the application phase. This is important since you will be working together for a period of at least 2 months; if the mentor-student communication doesn t work, the GSoC project is going to be difficult.
• Free and Open-Source Software (FOSS) communication platforms: I generally believe that FOSS projects should be built on FOSS infrastructure. I personally won t run proprietary software when I want to contribute to FOSS in my spare time.
• Be a user of the project: As Eric S. Raymond has pointed out in his seminal The Cathedral and the Bazaar 25 years ago

  Every good work of software starts by scratching a developer s personal itch.

Once I had some organizations in mind whose projects I d be interested in working on, I started writing proposals for them. Turns out, I started writing my proposals way too late: In the end I only managed to hand in a single one which is risky. Competition for the GSoC projects is fierce and the more quality (!) proposals you send out, the better your chances are at getting one. However, don t write proposals for the sake of it: Reviewers get way too many AI slop proposals already and you will not do yourself a favor with a low-quality proposal. Take the time to read the instructions/ideas/problem descriptions the project mentors have provided and follow their guidelines. Don t hesitate to reach out to project mentors: In my case, I ve asked Samuel Henrique a few clarification questions whereby the following (email) discussion has helped me greatly in improving my proposal. Once I ve finalized my proposal draft, I ve sent it to Samuel for a review, which again led to some improvements to the final proposal which I ve uploaded to the GSoC program webpage.

Community bonding period Once you get the information that you ve been accepted into the GSoC program (don t take it personally if you don t make it; this was my second attempt after not making the cut in 2024), get in touch with your prospective mentor ASAP. Agree upon a communication channel and some response times. Put yourself in the loop for project news and discussions whatever that means in the context of your organization: In Debian s case this boiled down to subscribing to a bunch of mailing lists and IRC channels. Also make sure to setup a functioning development environment if you haven t done so for writing the proposal already.

Payoneer setup The by far most annoying part of GSoC for me. But since you don t have a choice if you want to get the stipend, you will need to signup for an account at Payoneer. In this iteration of GSoC all participants got a personalized link to open a Payoneer account. When I tried to open an account by following this link, I got an email after the registration and email verification that my account is being blocked because Payoneer deems the email adress I gave a temporary one. Well, the email in question is most certainly anything but temporary, so I tried to get in touch with the Payoneer support - and ended up in an LLM-infused kafkaesque support hell. Emails are answered by an LLM which for me meant utterly off-topic replies and no help whatsoever. The Payoneer website offers a real-time chat, but it is yet another instance of a bullshit-spewing LLM bot. When I at last tried to call them (the support lines are not listed on the Payoneer website but were provided by the GSoC program), I kid you not, I was being told that their platform is currently suffering from technical problems and was hung up on. Only thanks to the swift and helpful support of the GSoC administrators (who get priority support from Payoneer) I was able to setup a Payoneer account in the end. Apart from showing no respect to customers, Payoneer is also ripping them off big time with fees (unless you get paid in USD). They charge you 2% for currency conversions to EUR on top of the FX spread they take. What worked for me to avoid all of those fees, was to open a USD account at Wise and have Payoneer transfer my GSoC stipend in USD to that account. Then I exchanged the USD to my local currency at Wise for significantly less than Payoneer would have charged me. Also make sure to close your Payoneer account after the end of GSoC to avoid their annual fee.

Project work With all this prelude out of the way, I can finally get to the actual work I ve been doing over the course of my GSoC project.

Background The upstream rsync project generally sees little development. Nonetheless, they released version 3.4.0 including some CVE fixes earlier this year. Unfortunately, their changes broke the -H flag. Now, Debian package maintainers need to apply those security fixes to the package versions in the Debian repositories; and those are typically a bit older. Which usually means that the patches cannot be applied as is but will need some amendments by the Debian maintainers. For these cases it is helpful to have autopkgtests defined, which check the package s functionality in an automated way upon every build. The question then is, why should the tests not be written upstream such that regressions are caught in the development rather than the distribution process? There s a lot to say on this question and it probably depends a lot on the package at hand, but for rsync the main benefits are twofold:

1. The upstream project mocks the ssh connection over which rsync is most typically used. Mocking is better than nothing but not the real thing. In addition to being a more realisitic test scenario for the typical rsync use case, involving an ssh server in the test would automatically extend the overall resilience of Debian packages as now new versions of the openssh-server package in Debian benefit from the test cases in the rsync reverse dependency.
2. The upstream rsync test framework is somewhat idiosyncratic and difficult to port to reimplementations of rsync. Given that the original rsync upstream sees little development, an extensive test suit further downstream can serve as a threshold for drop-in replacements for rsync.

Goal(s) At the start of the project, the Debian rsync package was just running (a part of) the upstream tests as autopkgtests. The relevant snippet from the build log for the rsync_3.4.1+ds1-3 package reads:

114s ------------------------------------------------------------
114s ----- overall results:
114s 36 passed
114s 7 skipped

Samuel and I agreed that it would be a good first milestone to make the skipped tests run. Afterwards, I should write some rsync test cases for local calls, i.e. without an ssh connection, effectively using rsync as a more powerful cp. And once that was done, I should extend the tests such that they run over an active ssh connection. With these milestones, I went to work.

Upstream tests Running the seven skipped upstream tests turned out to be fairly straightforward:

• Two upstream tests concern access control lists and extended filesystem attributes. For these tests to run they rely on functionality provided by the acl and xattr Debian packages. Adding those to the Build-Depends list in the debian/control file of the rsync Debian package repo made them run.
• Four upstream tests required root privileges to run. The autopkgtest tool knows the needs-root restriction for that reason. However, Samuel and I agreed that the tests should not exclusively run with root privileges. So, instead of just adding the restiction to the existing autopkgtest test, we created a new one which has the needs-root restriction and runs the upstream-tests-as-root script - which is nothing else than a symlink to the existing upstream-tests script.

The commits to implement these changes can be found in this merge request. The careful reader will have noticed that I only made 2 + 4 = 6 upstream test cases run out of 7: The leftover upstream test is checking the functionality of the --ctimes rsync option. In the context of Debian, the problem is that the Linux kernel doesn t have a syscall to set the creation time of a file. As long as that is the case, this test will always be skipped for the Debian package.

Local tests When it came to writing Debian specific test cases I started of a completely clean slate. Which is a blessing and a curse at the same time: You have full flexibility but also full responsibility. There were a few things to consider at this point in time:

• Which language to write the tests in? The programming language I am most proficient in is Python. But testing a CLI tool in Python would have been weird: it would have meant that I d have to make repeated subprocess calls to run rsync and then read from the filesystem to get the file statistics I want to check. Samuel suggested I stick with shell scripts and make use of diffoscope - one of the main tools used and maintained by the Reproducible Builds project - to check whether the file contents and file metadata are as expected after rsync calls. Since I did not have good reasons to use bash, I ve decided to write the scripts to be POSIX compliant.
• How to avoid boilerplate? If one makes use of a testing framework, which one? Writing the tests would involve quite a bit of boilerplate, mostly related to giving informative output on and during the test run, preparing the file structure we want to run rsync on, and cleaning the files up after the test has run. It would be very repetitive and in violation of DRY to have the code for this appear in every test. Good testing frameworks should provide convenience functions for these tasks. shunit2 comes with those functions, is packaged for Debian, and given that it is already being used in the curl project, I decided to go with it.
• Do we use the same directory structure and files for every test or should every test have an individual setup? The tradeoff in this question being test isolation vs. idiosyncratic code. If every test has its own setup, it takes a) more work to write the test and b) more work to understand the differences between tests. However, one can be sure that changes to the setup in one test will have no side effects on other tests. In my opinion, this guarantee was worth the additional effort in writing/reading the tests.

Having made these decisions, I simply started writing tests and ran into issues very quickly.

rsync and subsecond mtime diffs When testing the rsync --times option, I observed a weird phenomenon: If the source and destination file have modification times which differ only in the nanoseconds, an rsync --times call will not synchronize the modification times. More details about this behavior and examples can be found in the upstream issue I raised. In the Debian tests we had to occasionally work around this by setting the timestamps explicitly with touch -d.

diffoscope regression In one test case, I was expecting a difference in the modification times but diffoscope would not report a diff. After a good amount of time spent on debugging the problem (my default, and usually correct, assumption is that something about my code is seriously broken if I run into issues like that), I was able to show that diffoscope only displayed this behavior in the version in the unstable suite, not on Debian stable (which I am running on my development machine). Since everything pointed to a regression in the diffoscope project and with diffoscope being written in Python, a language I am familiar with, I wanted to spend some time investigating (and hopefully fixing) the problem. Running git bisect on the diffoscope repo helped me in identifying the commit which introduced the regression: The commit contained an optimization via an early return for bit-by-bit identical files. Unfortunately, the early return also caused an explicitly requested metadata comparison (which could be different between the files) to be skipped. With a nicely diagnosed issue like that, I was able to go to a local hackerspace event, where people work on FOSS together for an evening every month. In a group, we were able to first, write a test which showcases the broken behavior in the latest diffoscope version, and second, make a fix to the code such that the same test passes going forward. All details can be found in this merge request.

shunit2 failures At some point I had a few autopkgtests setup and passing, but adding a new one would throw me totally inexplicable errors. After trying to isolate the problem as much as possible, it turns out that shunit2 doesn t play well together we the -e shell option. The project mentions this in the release notes for the 2.1.8 version¹, but in my opinion a constraint this severe should be featured much more prominently, e.g. in the README.

Tests over an ssh connection The centrepiece of this project; everything else has in a way only been preparation for this. Obviously, the goal was to reuse the previously written local tests in some way. Not only because lazy me would have less work to do this way, but also because of a reduced long-term maintenance burden of one rather than two test sets. As it turns out, it is actually possible to accomplish that: The remote-tests script doesn t do much apart from starting an ssh server on localhost and running the local-tests script with the REMOTE environment variable set. The REMOTE environment variable changes the behavior of the local-tests script in such a way that it prepends "$REMOTE": to the destination of the rsync invocations. And given that we set REMOTE=rsync@localhost in the remote-tests script, local-tests copies the files to the exact same locations as before, just over ssh. The implementational details for this can be found in this merge request.

proposed-updates Most of my development work on the Debian rsync package took place during the Debian freeze as the release of Debian Trixie is just around the corner. This means that uploading by Debian Developers (DD) and Debian Maintainers (DM) to the unstable suite is discouraged as it makes migrating the packages to testing more difficult for the Debian release team. If DDs/DMs want to have the package version in unstable migrated to testing during the freeze they have to file an unblock request. Samuel has done this twice (1, 2) for my work for Trixie but has asked me to file the proposed-updates request for current stable (i.e. Debian Bookworm) myself after I ve backported my tests to bookworm.

Unfinished business To run the upstream tests which check access control list and extended file system attributes functionality, I ve added the acl and xattr packages to Build-Depends in debian/control. This, however, will only make the packages available at build time: If Debian users install the rsync package, the acl and xattr packages will not be installed alongside it. For that, the dependencies would have to be added to Depends or Suggests in debian/control. Depends is probably to strong of a relation since rsync clearly works well in practice without, but adding them to Suggests might be worthwhile. A decision on this would involve checking, what happens if rsync is called with the relevant options on a host machine which has those packages installed, but where the destination machine lacks them. Apart from the issue described above, the 15 tests I managed to write are are a drop in the water in light of the infinitude of rsync options and their combinations. Most glaringly, not all options of the --archive option are covered separately (which would help indicating what code path of rsync broke in a regression). To increase the likelihood of catching regressions with the autopkgtests, the test coverage should be extended in the future.

Conclusion Generally, I am happy with my contributions to Debian over the course of my small GSoC project: I ve created an extensible, easy to understand, and working autopkgtest setup for the Debian rsync package. There are two things which bother me, however:

1. In hindsight, I probably shouldn t have gone with shunit2 as a testing framework. The fact that it behaves erratically with the -e flag is a serious drawback for a shell testing framework: You really don t want a shell command to fail silently and the test to continue running.
2. As alluded to in the previous section, I m not particularly proud of the number of tests I managed to write.

On the other hand, finding and fixing the regression in diffoscope - while derailing me from the GSoC project itself - might have a redeeming quality.

DebConf25 By sheer luck I happened to work on a GSoC project at Debian over a time period during which the annual Debian conference would take place close enough to my place of residence. Samuel pointed the opportunity to attend DebConf out to me during the community bonding period and since I could make time for the event in my schedule, I signed up. DebConf was a great experience which - aside from gaining more knowledge about Debian development - allowed me to meet the actual people usually hidden behind email adresses and IRC nicks. I can wholeheartedly recommend attending a DebConf to every interested Debian user! For those who have missed this year s iteration of the conference, I can recommend the following recorded talks:

• reproduce.debian.net: Reproducing what is distributed from ftp.debian.org
• The security of Debian: An introduction to advanced users
• Bits from the DPL
• DebConf27 in your city?

While not featuring as a keynote speaker (understandably so as the newcomer to Debian community that I am), I could still contribute a bit to the conference program.

GSoC project presentation The Debian Outreach team has scheduled a session in which all GSoC and Outreachy students over the past year had the chance to present their work in a lightning talk. The session has been recorded and is available online, just like my slides and the source for them.

Debian install workshop Additionally, with so many Debian experts gathering in one place while KDE s End of 10 campaign is ongoing, I felt it natural to organize a Debian install workhop. In hindsight I can say that I underestimated how much work it would be, especially for me who does not speak a word of French. But although the turnout of people who wanted us to install Linux on their machines was disappointingly low, it was still worth it: Not only because the material in the repo can be helpful to others planning install workshops but also because it was nice to meet a) the person behind the Debian installer images and b) the local Brest/Finist re Linux user group as well as the motivated and helpful people at Infini.

Credits I want to thank the Open Source team at Google for organizing GSoC: The highly structured program with a one-to-one mentorship is a great avenue to start contributing to well established and at times intimidating FOSS projects. And as much as I disagree with Google s surveillance capitalist business model, I have to give it to them that the company at least takes its responsibility for FOSS (somewhat) seriously - unlike many other businesses which rely on FOSS and choose to freeride of it. Big thanks to the Debian community! I ve experienced nothing but friendliness in my interactions with the community. And lastly, the biggest thanks to my GSoC mentor Samuel Henrique. He has dealt patiently and competently with all my stupid newbie questions. His support enabled me to make - albeit small - contributions to Debian. It has been a pleasure to work with him during GSoC and I m looking forward to working together with him in the future.

---

1. Obviously, I ve only read them after experiencing the problem.

/usr/lib/glibc-hwcaps/x86-64-v4/libfoo0.so
/usr/lib/glibc-hwcaps/x86-64-v3/libfoo0.so
/usr/lib/glibc-hwcaps/x86-64-v2/libfoo0.so
/usr/lib/libfoo0.so

/usr/lib/x86_64-linux-gnu/ggml/glibc-hwcaps/x86-64-v4/libggml.so
/usr/lib/x86_64-linux-gnu/ggml/glibc-hwcaps/x86-64-v3/libggml.so
/usr/lib/x86_64-linux-gnu/ggml/glibc-hwcaps/x86-64-v2/libggml.so
/usr/lib/x86_64-linux-gnu/ggml/libggml.so

1. Reproducible Builds at FOSSY 2025
2. Distribution work
3. diffoscope
4. OSS Rebuild updates
5. Website updates
6. Upstream patches
7. Reproducibility testing framework

Reproducible Builds at FOSSY 2025 On Saturday 2nd August, Vagrant Cascadian and Chris Lamb will be presenting at this year s FOSSY 2025. Their talk, titled Never Mind the Checkboxes, Here s Reproducible Builds!, is being introduced as follows:

There are numerous policy compliance and regulatory processes being developed that target software development but do they solve actual problems? Does it improve the quality of software? Do Software Bill of Materials (SBOMs) actually give you the information necessary to verify how a given software artifact was built? What is the goal of all these compliance checklists anyways or more importantly, what should the goals be? If a software object is signed, who should be trusted to sign it, and can they be trusted forever?

The talk will introduce the audience to Reproducible Builds as a set of best practices which allow users and developers to verify that software artifacts were built from the source code, but also allows auditing for license compliance, providing security benefits, and removes the need to trust arbitrary software vendors. Hosted by the Software Freedom Conservancy and taking place in Portland, Oregon, USA, FOSSY aims to be a community-focused event: Whether you are a long time contributing member of a free software project, a recent graduate of a coding bootcamp or university, or just have an interest in the possibilities that free and open source software bring, FOSSY will have something for you . More information on the event is available on the FOSSY 2025 website, including the full programme schedule. Vagrant and Chris will also be staffing a table this year, where they will be available to answer any questions about Reproducible Builds and discuss collaborations with other projects.

---

Distribution work In Debian this month:

• Holger Levsen has discovered that it is now possible to bootstrap a minimal Debian trixie using 100% reproducible packages. This result can itself be reproduced, using the debian-repro-status tool and mmdebstrap s support for hooks:

    $ mmdebstrap --variant=apt --include=debian-repro-status \
         --chrooted-customize-hook=debian-repro-status \
         trixie /dev/null 2>&1   grep "Your system has"
     INFO  debian-repro-status > Your system has 100.00% been reproduced.
  

• On our mailing list this month, Helmut Grohne wrote an extensive message raising an issue related to Uploads with conflicting buildinfo filenames:

  Having several .buildinfo files for the same architecture is something that we plausibly want to have eventually. Imagine running two sets of buildds and assembling a single upload containing buildinfo files from both buildds in the same upload. In a similar vein, as a developer I may want to supply several .buildinfo files with my source upload (e.g. for multiple architectures). Doing any of this is incompatible with current incoming processing and with reprepro.

• 5 reviews of Debian packages were added, 4 were updated and 8 were removed this month adding to our ever-growing knowledge about identified issues.

In GNU Guix, Timothee Mathieu reported that a long-standing issue with reproducibility of shell containers across different host operating systems has been solved. In their message, Timothee mentions:

I discovered that pytorch (and maybe other dependencies) has a reproducibility problem of order 1e-5 when on AVX512 compared to AVX2. I first tried to solve the problem by disabling AVX512 at the level of pytorch, but it did not work. The dev of pytorch said that it may be because some components dispatch computation to MKL-DNN, I tried to disable AVX512 on MKL, and still the results were not reproducible, I also tried to deactivate in openmpi without success. I finally concluded that there was a problem with AVX512 somewhere in the dependencies graph but I gave up identifying where, as this seems very complicated.

The IzzyOnDroid Android APK repository made more progress in June. Not only have they just passed 48% reproducibility coverage, Ben started making their reproducible builds more visible, by offering rbtlog shields, a kind of badge that has been quickly picked up by many developers who are proud to present their applications reproducibility status.
Lastly, in openSUSE news, Bernhard M. Wiedemann posted another monthly update for their work there.

diffoscope diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made the following changes, including preparing and uploading versions 298, 299 and 300 to Debian:

• Add python3-defusedxml to the Build-Depends in order to include it in the Docker image. [ ]
• Handle the RPM format s HEADERSIGNATURES and HEADERIMMUTABLE as a special-case to avoid unnecessarily large diffs. Thanks to Daniel Duan for the report and suggestion. [ ][ ]
• Update copyright years. [ ]

In addition, @puer-robustus fixed a regression introduced in an earlier commit which resulted in some differences being lost. [ ][ ] Lastly, Vagrant Cascadian updated diffoscope in GNU Guix to version 299 [ ][ ] and 300 [ ][ ].

OSS Rebuild updates OSS Rebuild has added a new network analyzer that provides transparent HTTP(S) interception during builds, capturing all network traffic to monitor external dependencies and identify suspicious behavior, even in unmodified maintainer-controlled build processes. The text-based user interface now features automated failure clustering that can group similar rebuild failures and provides natural language failure summaries, making it easier to identify and understand patterns across large numbers of build failures. OSS Rebuild has also improved the local development experience with a unified interface for build execution strategies, allowing for more extensible environment setup for build execution. The team also designed a new website and logo.

Website updates Once again, there were a number of improvements made to our website this month including:

• Arnaud Brousseau added Stage , a new Linux distribution, to our Tools page.
• Chris Lamb improved the docker instructions on the diffoscope website. [ ]

---

Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

• Chris Riches:
  • rpm
• Bernhard M. Wiedemann:
  • arandr, curl, dpdk, dpdk, eww, gnucash, gramps, latex2html, llvm20, mp, nvidia-open-driver-G06-signed, obs, ovmf, pcre2, perl-XML-LibXML, perl, python-reportlab, python313, qt6-datavis3d, qt6-declarative, qt6-sensors, qt6-virtualkeyboard, rage-encryption, scummvm, timescaledb & zoxide.
  • Plus several issues with how rpmbuild expands the %jobs variable in the .src.rpm header, including: chromium, cmake, edk2, firefox-esr, gnome-keyring-sharp, gtk2-engine-aurora, gtk2-engine-cleanice, gtk2-engines, libqt5-qtlocation, libreoffice, luabind, lxmenu-data, mozc, MozillaThunderbird, perl-DateTime-Calendar-Mayan, perl-Getopt-ArgvFile, perl-MooseX-Meta-TypeConstraint-ForceCoercion, perl-XML-Entities, python-convertdate, suitesparse & webkit2gtk3
• Robin Candau:
  • gramps (use SOURCE_DATE_EPOCH when compressing man pages)
• Chris Lamb:
  • #1108273 filed against tree-puzzle.
  • #1108281 filed against cctools.
  • #1108532 filed against python-django-import-export.

---

Reproducibility testing framework The Reproducible Builds project operates a comprehensive testing framework running primarily at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In June, however, a number of changes were made by Holger Levsen, including:

• reproduce.debian.net-related:
  • Installed and deployed rebuilderd version 0.24 from Debian unstable in order to make use of the new compression feature added by Jarl Gullberg for the database. This resulted in massive decrease of the SQLite databases:
    • 79G 2.8G (all)
    • 84G 3.2G (amd64)
    • 75G 2.9G (arm64)
    • 45G 2.1G (armel)
    • 48G 2.2G (armhf)
    • 73G 2.8G (i386)
    • 72G 2.7G (ppc64el)
    • 45G 2.1G (riscv64)
    for a combined saving from 521G 20.8G. This naturally reduces the requirements to run an independent rebuilderd instance and will permit us to add more Debian suites as well.
  • During migration to the latest version of rebuilderd, make sure several services are not started. [ ]
  • Actually run rebuilderd from /usr/bin. [ ]
  • Raise temperatures for NVME devices on some riscv64 nodes that should be ignored. [ ][ ]
  • Use a 64KB kernel page size on the ppc64el architecture (see #1106757). [ ]
  • Improve ordering of some failed to reproduce statistics. [ ]
  • Detect a number of potential causes of build failures within the statistics. [ ][ ]
  • Add support for manually scheduling for the any architecture. [ ]
• Misc:
  • Update the Codethink nodes as there are now many kernels installed. [ ][ ]
  • Install linux-sysctl-defaults on Debian trixie systems as we need ping functionality. [ ]
  • Limit the fs.nr_open kernel turnable. [ ]
  • Stop submitting results to deprecated buildinfo.debian.net service. [ ][ ]

In addition, Jochen Sprickerhof greatly improved the statistics and the logging functionality, including adopting to the new database format of rebuilderd version 0.24.0 [ ] and temporarily increasing maximum log size in order to debug a nettlesome build [ ]. Jochen also dropped the CPUSchedulingPolicy=idle systemd flag on the workers. [ ]

Finally, if you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

• IRC: #reproducible-builds on irc.oftc.net.
• Mastodon: @reproducible_builds@fosstodon.org
• Mailing list: rb-general@lists.reproducible-builds.org

Debian LTS contributors In June, 20 contributors have been paid to work on Debian LTS, their reports are available:

• Abhijith PA did 14.0h (out of 14.0h assigned).
• Adrian Bunk did 23.5h (out of 23.5h assigned).
• Andreas Henriksson did 3.0h (out of 3.0h assigned and 17.0h from previous period), thus carrying over 17.0h to the next month.
• Andrej Shadura did 2.0h (out of 3.0h assigned and 7.0h from previous period), thus carrying over 8.0h to the next month.
• Bastien Roucari s did 20.0h (out of 20.0h assigned).
• Ben Hutchings did 8.0h (out of 7.5h assigned and 16.0h from previous period), thus carrying over 15.5h to the next month.
• Carlos Henrique Lima Melara did 12.0h (out of 12.0h assigned).
• Chris Lamb did 18.0h (out of 18.0h assigned).
• Daniel Leidert did 22.0h (out of 22.5h assigned and 1.0h from previous period), thus carrying over 1.5h to the next month.
• Emilio Pozuelo Monfort did 23.5h (out of 16.75h assigned and 6.75h from previous period).
• Guilhem Moulin did 14.0h (out of 11.5h assigned and 3.5h from previous period), thus carrying over 1.0h to the next month.
• Jochen Sprickerhof did 21.0h (out of 0.5h assigned and 22.75h from previous period), thus carrying over 2.25h to the next month.
• Lucas Kanashiro did 20.0h (out of 20.0h assigned).
• Markus Koschany did 23.25h (out of 17.0h assigned and 6.25h from previous period).
• Roberto C. S nchez did 21.25h (out of 20.75h assigned and 3.25h from previous period), thus carrying over 2.75h to the next month.
• Santiago Ruano Rinc n did 12.75h (out of 15.0h assigned), thus carrying over 2.25h to the next month.
• Sean Whitton did 1.0h (out of 4.25h assigned and 1.75h from previous period), thus carrying over 5.0h to the next month.
• Sylvain Beucler did 23.5h (out of 23.5h assigned).
• Thorsten Alteholz did 15.0h (out of 15.0h assigned).
• Tobias Frost did 2.5h (out of 12.0h assigned), thus carrying over 9.5h to the next month.

Evolution of the situation In June, we released 35 DLAs.

• Notable security updates:
  • mariadb-10.5, prepared by Otto Kek l inen, fixes vulnerabilities which could result in denial of service, information disclosure, or unauthorized data modification
  • python-django, prepared by Chris Lamb, fixes vulnerabilities which would result in log injection or denial of service
  • webkit2gtk, prepared by Emilio Pozuelo Monfort, fixes many vulnerabilities which could results in a wide range of issues
  • xorg-server, prepared by Emilio Pozuelo Monfort, fixes multiple vulnerabilities which may result in privilege escalation
  • sudo, prepared by Thorsten Alteholz, fixes a vulnerability which could result in privilege escalation
• Notable non-security updates:
  • debian-security-support, prepared by Santiago Ruano Rinc n, updates status of packages which receive limited security support or which have reached the end of security support
  • dns-root-data, prepared by Sylvain Beucler, updates the DNSSEC trust anchors

This month s contributions from outside the regular team include the mariadb-10.5 update mentioned above, prepared by Otto Kek l inen (the package maintainer); an update to libfile-find-rule-perl, prepared by Salvatore Bonaccorso (a member of the Debian Security Team); an update to activemq, prepared by Emmanuel Arias (a maintainer of the package). Additionally, LTS Team members contributed stable updates of the following packages:

• curl, prepared by Carlos Henrique Lima Melara
• python-tornado, prepared by Daniel Leidert
• python-flask-cors, prepared by Daniel Leidert
• common-vfs, prepared by Daniel Leidert
• cjson, prepared by Adrian Bunk
• icu, prepared by Adrian Bunk
• node-tar-fs, prepared by Adrian Bunk
• rar, prepared by Adrian Bunk

Something of particular noteworthiness is that LTS contributor Carlos Henrique Lima Melara discovered a regression in the upstream fix for CVE-2023-2753 in curl. The corrective action which he took included providing a patch to upstream, uploading a stable update of curl, and further updating the version of curl in LTS. DebConf, the annual Debian Conference, is coming up in July and, as is customary each year, the week preceding the conference will feature an event called DebCamp. The DebCamp week provides an opportunity for teams and other interested groups/individuals to meet together in person in the same venue as the conference itself, with the purpose of doing focused work, often called sprints . LTS coordinator Roberto C. S nchez has announced that the LTS Team is planning to hold a sprint primarily focused on the Debian security tracker and the associated tooling used by the LTS Team and the Debian Security Team.

Thanks to our sponsors Sponsors that joined recently are in bold.

• Platinum sponsors:
  • Toshiba Corporation (for 117 months)
  • Civil Infrastructure Platform (CIP) (for 85 months)
  • VyOS Inc (for 49 months)
• Gold sponsors:
  • Roche Diagnostics International AG (for 127 months)
  • Akamai - Linode (for 121 months)
  • Babiel GmbH (for 111 months)
  • Plat Home (for 110 months)
  • University of Oxford (for 67 months)
  • Deveryware (for 54 months)
  • EDF SA (for 39 months)
  • Dataport A R (for 14 months)
  • CERN (for 12 months)
• Silver sponsors:
  • Domeneshop AS (for 132 months)
  • Nantes M tropole (for 126 months)
  • Univention GmbH (for 118 months)
  • Universit Jean Monnet de St Etienne (for 118 months)
  • Ribbon Communications, Inc. (for 112 months)
  • Exonet B.V. (for 102 months)
  • Leibniz Rechenzentrum (for 96 months)
  • Minist re de l Europe et des Affaires trang res (for 80 months)
  • Cloudways by DigitalOcean (for 69 months)
  • Dinahosting SL (for 67 months)
  • Platform.sh SAS (for 61 months)
  • Moxa Inc. (for 55 months)
  • sipgate GmbH (for 53 months)
  • OVH US LLC (for 51 months)
  • Tilburg University (for 51 months)
  • GSI Helmholtzzentrum f r Schwerionenforschung GmbH (for 42 months)
  • THINline s.r.o. (for 15 months)
  • Copenhagen Airports A/S (for 9 months)
• Bronze sponsors:
  • Evolix (for 132 months)
  • Seznam.cz, a.s. (for 132 months)
  • Intevation GmbH (for 129 months)
  • Linuxhotel GmbH (for 129 months)
  • Daevel SARL (for 128 months)
  • Megaspace Internet Services GmbH (for 127 months)
  • Greenbone AG (for 126 months)
  • NUMLOG (for 126 months)
  • WinGo AG (for 125 months)
  • Entr ouvert (for 116 months)
  • Adfinis AG (for 114 months)
  • Tesorion (for 109 months)
  • Laboratoire LEGI - UMR 5519 / CNRS (for 108 months)
  • Bearstech (for 100 months)
  • LiHAS (for 100 months)
  • Catalyst IT Ltd (for 95 months)
  • Demarcq SAS (for 89 months)
  • Universit Grenoble Alpes (for 75 months)
  • TouchWeb SAS (for 67 months)
  • SPiN AG (for 64 months)
  • CoreFiling (for 60 months)
  • Institut des sciences cognitives Marc Jeannerod (for 55 months)
  • Observatoire des Sciences de l Univers de Grenoble (for 51 months)
  • Tem Innovations GmbH (for 46 months)
  • WordFinder.pro (for 45 months)
  • CNRS DT INSU R sif (for 44 months)
  • Soliton Systems K.K. (for 39 months)
  • Alter Way (for 37 months)
  • Institut Camille Jordan (for 27 months)
  • SOBIS Software GmbH (for 12 months)
  • Tuxera Inc. (for 3 months)

Changes in Rcpp release version 1.1.0 (2025-07-01)

• Changes in Rcpp API:
  • C++11 is now the required minimal C++ standard
  • The std::string_view type is now covered by wrap() (Lev Kandel in #1356 as discussed in #1357)
  • A last remaining DATAPTR use has been converted to DATAPTR_RO (Dirk in #1359)
  • Under R 4.5.0 or later, R_ClosureEnv is used instead of CLOENV (Dirk in #1361 fixing #1360)
  • Use of lsInternal switched to lsInternal3 (Dirk in #1362)
  • Removed compiler detection macro in a header cleanup setting C++11 as the minunum (Dirk in #1364 closing #1363)
  • Variadic templates are now used onconditionally given C++11 (Dirk in #1367 closing #1366)
  • Remove RCPP_USING_CXX11 as a #define as C++11 is now a given (Dirk in #1369)
  • Additional cleanup for __cplusplus checks (I aki in #1371 fixing #1370)
  • Unordered set construction no longer needs a macro for the pre-C++11 case (I aki in #1372)
  • Lambdas are supported in a Rcpp Sugar functions (I aki in #1373)
  • The Date(time)Vector classes now have default ctor (Dirk in #1385 closing #1384)
  • Fixed an issue where Rcpp::Language would duplicate its arguments (Kevin in #1388, fixing #1386)
• Changes in Rcpp Attributes:
  • The C++26 standard now has plugin support (Dirk in #1381 closing #1380)
• Changes in Rcpp Documentation:
  • Several typos were correct in the NEWS file (Ben Bolker in #1354)
  • The Rcpp Libraries vignette mentions PACKAGE_types.h to declare types used in RcppExports.cpp (Dirk in #1355)
  • The vignettes bibliography file was updated to current package versions, and now uses doi references (Dirk in #1389)
• Changes in Rcpp Deployment:
  • Rcpp.package.skeleton() creates URL and BugReports if given a GitHub username (Dirk in #1358)
  • R 4.4.* has been added to the CI matrix (Dirk in #1376)
  • Tests involving NA propagation are skipped under linux-arm64 as they are under macos-arm (Dirk in #1379 closing #1378)

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can sponsor me at GitHub.

This post is a review for Computing Reviews for Understanding Misunderstandings - Evaluating LLMs on Networking Questions , a article published in Association for Computing Machinery (ACM), SIGCOMM Computer Communication Review

• Accuracy: the correctness of the answers provided by LLMs;
• Detectability: how easily errors in the LLM output can be identified;
• Cause: for each incorrect answer, the underlying causes behind the error;
• Explainability: the quality of the explanations with which the LLMs support their answers;
• Effects: the impact of wrong answers on users; and
• Stability: whether a minor change, such as a change in the order of the prompts, yields vastly different answers for a single query.

• Self-correction: giving the original question and received answer back to the LLM, as well as the expected correct answer, as part of the prompt;
• One-shot prompting: adding to the prompt when answering user questions, follow this example followed by a similar correct answer;
• Majority voting: using the answer that most models agree upon; and
• Fine-tuning: further training on a specific dataset to adapt the LLM to a particular task or domain.

Debian Contributions: 2025-05 Contributing to Debian is part of Freexian s mission. This article covers the latest achievements of Freexian and their collaborators. All of this is made possible by organizations subscribing to our Long Term Support contracts and consulting services.

Updated Austin, by Colin Watson and Helmut Grohne Austin is a frame stack sampling profiler for Python. It allows profiling Python applications without instrumenting them while losing some accuracy in the process, and is the only one of its kind presently packaged for Debian. Unfortunately, it hadn t been uploaded in a while and hence the last Python version it worked with was 3.8. We updated it to a current version and also dealt with a number of architecture-specific problems (such as unintended sign promotion, 64bit time_t fallout and strictness due to -Wformat-security ) in cooperation with upstream. With luck, it will migrate in time for trixie.

Preparing for DebConf 25, by Stefano Rivera and Santiago Ruano Rinc n DebConf 25 is quickly approaching, and the organization work doesn t stop. In May, Stefano continued supporting the different teams. Just to give a couple of examples, Stefano made changes in DebConf 25 website to make BoF and sprints submissions public, so interested people can already know if a BoF or sprint for a given subject is planned, allowing coordination with the proposer; or to enhance how statistics are made public to help the work of the local team. Santiago has participated in different tasks, including the logistics of the conference, like preparing more information about the public transportation that will be available. Santiago has also taken part in activities related to fundraising and reviewing more event proposals.

Miscellaneous contributions

• Lucas fixed security issues in Valkey in unstable.
• Lucas tried to help with the update of Redis to version 8 in unstable. The package hadn t been updated for a while due to licensing issues, but now upstream maintainers fixed them.
• Lucas uploaded around 20 ruby-* packages to unstable that weren t updated for some years to make them build reproducible. Thanks to reproducible builds folks to point out those issues. Also some unblock requests (and follow-ups) were needed to make them reach trixie in time for the release.
• Lucas is organizing a Debian Outreach session for DebConf 25, reaching out to all interns of Google Summer of Code and Outreachy programs from the last year. The session will be presented by in-person interns and also video recordings from the interns interested in participating but did not manage to attend the conference.
• Lucas continuously works on DebConf Content team tasks. Replying to speakers, sponsors, and communicating internally with the team.
• Carles improved po-debconf-manager: fixed bugs reported by Catalan translator, added possibility to import packages out of salsa, added using non-default project branches on salsa, polish to get ready for DebCamp.
• Carles tested new apt in trixie and reported bugs to apt , installation-report , libqt6widget6 .
• Carles used po-debconf-manager and imported remaining 80 packages, reviewed 20 translations, submitted (MR or bugs) 54 translations.
• Carles prepared some topics for translation BoF in DebConf (gathered feedback, first pass on topics).
• Helmut gave an introductory talk about the mechanics of Linux namespaces at MiniDebConf Hamburg.
• Helmut sent 25 patches for cross compilation failures.
• Helmut reviewed, refined and applied a patch from Jochen Sprickerhof to make the Multi-Arch hinter emit more hints for pure Python modules.
• Helmut sat down with Christoph Berg (not affiliated with Freexian) and extended unschroot to support directory-based chroots with overlayfs. This is a feature that was lost in transitioning from sbuild s schroot backend to its unshare backend. unschroot implements the schroot API just enough to be usable with sbuild and otherwise works a lot like the unshare backend. As a result, apt.postgresql.org now performs its builds contained in a user namespace.
• Helmut looked into a fair number of rebootstrap failures most of which related to musl or gcc-15 and imported patches or workarounds to make those builds proceed.
• Helmut updated dumat to use sqop fixing earlier PGP verification problems thanks to Justus Winter and Neal Walfield explaining a lot of sequoia at MiniDebConf Hamburg.
• Helmut got the previous zutils update for /usr-move wrong again and had to send another update.
• Helmut looked into why debvm s autopkgtests were flaky and with lots of help from Paul Gevers and Michael Tokarev tracked it down to a race condition in qemu. He updated debvm to trigger the problem less often and also fixed a wrong dependency using Luca Boccassi s patch.
• Santiago continued the switch to sbuild for Salsa CI (that was stopped for some months), and has been mainly testing linux, since it s a complex project that heavily customizes the pipeline. Santiago is preparing the changes for linux to submit a MR soon.
• In openssh, Colin tracked down some intermittent sshd crashes to a root cause, and issued bookworm and bullseye updates for CVE-2025-32728.
• Colin spent some time fixing up fail2ban, mainly reverting a patch that caused its tests to fail and would have banned legitimate users in some common cases.
• Colin backported upstream fixes for CVE-2025-48383 (django-select2) and CVE-2025-47287 (python-tornado) to unstable.
• Stefano supported video streaming and recording for 2 miniDebConfs in May: Macei and Hamburg. These had overlapping streams for one day, which is a first for us.
• Stefano packaged the new version of python-virtualenv that includes our patches for not including the wheel for wheel.
• Stefano got all involved parties to agree (in principle) to meet at DebConf for a mediated discussion on a dispute that was brought to the technical committee.
• Anupa coordinated the swag purchase for DebConf 25 with Juliana and Nattie.
• Anupa joined the publicity team meeting for discussing the upcoming events and BoF at DebConf 25.
• Anupa worked with the publicity team to publish Bits post to welcome GSoc 2025 Interns.

• automat
• celery
• flufl.i18n
• flufl.lock
• frozenlist
• python-charset-normalizer
• python-evalidate (including pointing out an upstream release handling issue)
• python-pythonjsonlogger
• python-setproctitle
• python-telethon
• python-typing-inspection
• python-webargs
• pyzmq
• trove-classifiers (including a small upstream cleanup)
• uncertainties
• zope.testrunner

• python-django to 3:4.2.21-1 (issuing BSA-124)
• python-django-pgtrigger to 4.14.0-1

• celery (contributed upstream)
• python-setproctitle
• uncertainties (contributed upstream, after some discussion)

• django-select2: CVE-2025-48383
• python-tornado: CVE-2025-47287

• fail2ban (also reviewing and merging fix sshd 10.0 log identifier and remove runtime calls to distutils)
• karabo-bridge (contributed upstream)
• kegtron-ble
• python-click-option-group (NMU)
• python-holidays
• python-mastodon
• python-mechanize (contributed upstream)
• thermobeacon-ble

• karabo-bridge
• python-lazy-model

TL;DR: ... or just skip the cruft and install asncounter:

pip install asncounter

Also available in Debian 14 or later, or possibly in Debian 13 backports (soon to be released) if people are interested:

apt install asncounter

Then count whoever is hitting your network with:

awk ' print $2 ' /var/log/apache2/*access*.log   asncounter

or:

tail -F /var/log/apache2/*access*.log   awk ' print $2 '   asncounter

or:

tcpdump -q -n   asncounter --input-format=tcpdump --repl

or:

tcpdump -q -i eth0 -n -Q in "tcp and tcp[tcpflags] & tcp-syn != 0 and (port 80 or port 443)"   asncounter --input-format=tcpdump --repl

Read on for why this matters, and why I wrote yet another weird tool (almost) from scratch.

Background and manual work This is a tool I've been dreaming of for a long, long time. Back in 2006, at Koumbit a colleague had setup TAS ("Traffic Accounting System", " " in Russian, apparently), a collection of Perl script that would do per-IP accounting. It was pretty cool: it would count bytes per IP addresses and, from that, you could do analysis. But the project died, and it was kind of bespoke. Fast forward twenty years, and I find myself fighting off bots at the Tor Project (the irony...), with our GitLab suffering pretty bad slowdowns (see issue tpo/tpa/team#41677 for the latest public issue, the juicier one is confidential, unfortunately). (We did have some issues caused by overloads in CI, as we host, after all, a fork of Firefox, which is a massive repository, but the applications team did sustained, awesome work to fix issues on that side, again and again (see tpo/applications/tor-browser#43121 for the latest, and tpo/applications/tor-browser#43121 for some pretty impressive correlation work, I work with really skilled people). But those issues, I believe were fixed.) So I had the feeling it was our turn to get hammered by the AI bots. But how do we tell? I could tell something was hammering at the costly /commit/ and (especially costly) /blame/ endpoint. So at first, I pulled out the trusted awk, sort uniq -c sort -n tail pipeline I am sure others have worked out before:

awk ' print $1 ' /var/log/nginx/*.log   sort   uniq -c   sort -n   tail -10

For people new to this, that pulls the first field out of web server log files, sort the list, counts the number of unique entries, and sorts that so that the most common entries (or IPs) show up first, then show the top 10. That, other words, answers the question of "which IP address visits this web server the most?" Based on this, I found a couple of IP addresses that looked like Alibaba. I had already addressed an abuse complaint to them (tpo/tpa/team#42152) but never got a response, so I just blocked their entire network blocks, rather violently:

for cidr in 47.240.0.0/14 47.246.0.0/16 47.244.0.0/15 47.235.0.0/16 47.236.0.0/14; do 
  iptables-legacy -I INPUT -s $cidr -j REJECT
done

That made Ali Baba and his forty thieves (specifically their AL-3 network go away, but our load was still high, and I was still seeing various IPs crawling the costly endpoints. And this time, it was hard to tell who they were: you'll notice all the Alibaba IPs are inside the same 47.0.0.0/8 prefix. Although it's not a /8 itself, it's all inside the same prefix, so it's visually easy to pick it apart, especially for a brain like mine who's stared too long at logs flowing by too fast for their own mental health. What I had then was different, and I was tired of doing the stupid thing I had been doing for decades at this point. I had recently stumbled upon pyasn recently (in January, according to my notes) and somehow found it again, and thought "I bet I could write a quick script that loops over IPs and counts IPs per ASN". (Obviously, there are lots of other tools out there for that kind of monitoring. Argos, for example, presumably does this, but it's a kind of a huge stack. You can also get into netflows, but there's serious privacy implications with those. There are also lots of per-IP counters like promacct, but that doesn't scale. Or maybe someone already had solved this problem and I just wasted a week of my life, who knows. Someone will let me know, I hope, either way.)

ASNs and networks A quick aside, for people not familiar with how the internet works. People that know about ASNs, BGP announcements and so on can skip. The internet is the network of networks. It's made of multiple networks that talk to each other. The way this works is there is a Border Gateway Protocol (BGP), a relatively simple TCP-based protocol, that the edge routers of those networks used to announce each other what network they manage. Each of those network is called an Autonomous System (AS) and has an AS number (ASN) to uniquely identify it. Just like IP addresses, ASNs are allocated by IANA and local registries, they're pretty cheap and useful if you like running your own routers, get one. When you have an ASN, you'll use it to, say, announce to your BGP neighbors "I have 198.51.100.0/24" over here and the others might say "okay, and I have 216.90.108.31/19 over here, and I know of this other ASN over there that has 192.0.2.1/24 too! And gradually, those announcements flood the entire network, and you end up with each BGP having a routing table of the global internet, with a map of which network block, or "prefix" is announced by which ASN. It's how the internet works, and it's a useful thing to know, because it's what, ultimately, makes an organisation responsible for an IP address. There are "looking glass" tools like the one provided by routeviews.org which allow you to effectively run "trace routes" (but not the same as traceroute, which actively sends probes from your location), type an IP address in that form to fiddle with it. You will end up with an "AS path", the way to get from the looking glass to the announced network. But I digress, and that's kind of out of scope. Point is, internet is made of networks, networks are autonomous systems (AS) and they have numbers (ASNs), and they announced IP prefixes (or "network blocks") that ultimately tells you who is responsible for traffic on the internet.

Introducing asncounter So my goal was to get from "lots of IP addresses" to "list of ASNs", possibly also the list of prefixes (because why not). Turns out pyasn makes that really easy. I managed to build a prototype in probably less than an hour, just look at the first version, it's 44 lines (sloccount) of Python, and it works, provided you have already downloaded the required datafiles from routeviews.org. (Obviously, the latest version is longer at close to 1000 lines, but it downloads the data files automatically, and has many more features). The way the first prototype (and later versions too, mostly) worked is that you feed it a list of IP addresses on standard input, it looks up the ASN and prefix associated with the IP, and increments a counter for those, then print the result. That showed me something like this:

root@gitlab-02:~/anarcat-scripts# tcpdump -q -i eth0 -n -Q in "(udp or tcp)"   ./asncounter.py --tcpdump                                                                                                                                                                          
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode                                                                
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes                                                             
INFO: collecting IPs from stdin, using datfile ipasn_20250523.1600.dat.gz                                                                
INFO: loading datfile /root/.cache/pyasn/ipasn_20250523.1600.dat.gz...                                                                   
INFO: loading /root/.cache/pyasn/asnames.json                       
ASN     count   AS               
136907  7811    HWCLOUDS-AS-AP HUAWEI CLOUDS, HK                                                                                         
[----]  359     [REDACTED]
[----]  313     [REDACTED]
8075    254     MICROSOFT-CORP-MSN-AS-BLOCK, US
[---]   164     [REDACTED]
[----]  136     [REDACTED]
24940   114     HETZNER-AS, DE  
[----]  98      [REDACTED]
14618   82      AMAZON-AES, US                                                                                                           
[----]  79      [REDACTED]
prefix  count                                         
166.108.192.0/20        1294                                                                                                             
188.239.32.0/20 1056                                          
166.108.224.0/20        970                    
111.119.192.0/20        951              
124.243.128.0/18        667                                         
94.74.80.0/20   651                                                 
111.119.224.0/20        622                                         
111.119.240.0/20        566           
111.119.208.0/20        538                                         
[REDACTED]  313           

Even without ratios and a total count (which will come later), it was quite clear that Huawei was doing something big on the server. At that point, it was responsible for a quarter to half of the traffic on our GitLab server or about 5-10 queries per second. But just looking at the logs, or per IP hit counts, it was really hard to tell. That traffic is really well distributed. If you look more closely at the output above, you'll notice I redacted a couple of entries except major providers, for privacy reasons. But you'll also notice almost nothing is redacted in the prefix list, why? Because all of those networks are Huawei! Their announcements are kind of bonkers: they have hundreds of such prefixes. Now, clever people in the know will say "of course they do, it's an hyperscaler; just ASN14618 (AMAZON-AES) there is way more announcements, they have 1416 prefixes!" Yes, of course, but they are not generating half of my traffic (at least, not yet). But even then: this also applies to Amazon! This way of counting traffic is way more useful for large scale operations like this, because you group by organisation instead of by server or individual endpoint. And, ultimately, this is why asncounter matters: it allows you to group your traffic by organisation, the place you can actually negotiate with. Now, of course, that assumes those are entities you can talk with. I have written to both Alibaba and Huawei, and have yet to receive a response. I assume I never will. In their defence, I wrote in English, perhaps I should have made the effort of translating my message in Chinese, but then again English is the Lingua Franca of the Internet, and I doubt that's actually the issue.

The Huawei and Facebook blocks Another aside, because this is my blog and I am not looking for a Pullitzer here. So I blocked Huawei from our GitLab server (and before you tear your shirt open: only our GitLab server, everything else is still accessible to them, including our email server to respond to my complaint). I did so 24h after emailing them, and after examining their user agent (UA) headers. Boy that was fun. In a sample of 268 requests I analyzed, they churned out 246 different UAs. At first glance, they looked legit, like:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36

Safari on a Mac, so far so good. But when you start digging, you notice some strange things, like here's Safari running on Linux:

Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.3 (KHTML, like Gecko) Chrome/6.0.457.0 Safari/534.3

Was Safari ported to Linux? I guess that's.. possible? But here is Safari running on a 15 year old Ubuntu release (10.10):

Mozilla/5.0 (X11; Linux i686) AppleWebKit/534.24 (KHTML, like Gecko) Ubuntu/10.10 Chromium/12.0.702.0 Chrome/12.0.702.0 Safari/534.24

Speaking of old, here's Safari again, but this time running on Windows NT 5.1, AKA Windows XP, released 2001, EOL since 2019:

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-CA) AppleWebKit/534.13 (KHTML like Gecko) Chrome/9.0.597.98 Safari/534.13

Really? Here's Firefox 3.6, released 14 years ago, there were quite a lot of those:

Mozilla/5.0 (Windows; U; Windows NT 6.1; lt; rv:1.9.2) Gecko/20100115 Firefox/3.6

I remember running those old Firefox releases, those were the days. But to me, those look like entirely fake UAs, deliberately rotated to make it look like legitimate traffic. In comparison, Facebook seemed a bit more legit, in the sense that they don't fake it. most hits are from:

meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)

which, according their documentation:

crawls the web for use cases such as training AI models or improving products by indexing content directly

From what I could tell, it was even respecting our rather liberal robots.txt rules, in that it wasn't crawling the sprawling /blame/ or /commit/ endpoints, explicitly forbidden by robots.txt. So I've blocked the Facebook bot in robots.txt and, amazingly, it just went away. Good job Facebook, as much as I think you've given the empire to neo-nazis, cause depression and genocide, you know how to run a crawler, thanks. Huawei was blocked at the web server level, with a friendly 429 status code telling people to contact us (over email) if they need help. And they don't care: they're still hammering the server, from what I can tell, but then again, I didn't block the entire ASN just yet, just the blocks I found crawling the server over a couple hours.

A full asncounter run So what does a day in asncounter look like? Well, you start with a problem, say you're getting too much traffic and want to see where it's from. First you need to sample it. Typically, you'd do that with tcpdump or tailing a log file:

tail -F /var/log/apache2/*access*.log   awk ' print $2 '   asncounter

If you have lots of traffic or care about your users' privacy, you're not going to log IP addresses, so tcpdump is likely a good option instead:

tcpdump -q -n   asncounter --input-format=tcpdump --repl

If you really get a lot of traffic, you might want to get a subset of that to avoid overwhelming asncounter, it's not fast enough to do multiple gigabit/second, I bet, so here's only incoming SYN IPv4 packets:

tcpdump -q -n -Q in "tcp and tcp[tcpflags] & tcp-syn != 0 and (port 80 or port 443)"   asncounter --input-format=tcpdump --repl

In any case, at this point you're staring at a process, just sitting there. If you passed the --repl or --manhole arguments, you're lucky: you have a Python shell inside the program. Otherwise, send SIGHUP to the thing to have it dump the nice tables out:

pkill -HUP asncounter

Here's an example run:

> awk ' print $2 ' /var/log/apache2/*access*.log   asncounter
INFO: using datfile ipasn_20250527.1600.dat.gz
INFO: collecting addresses from <stdin>
INFO: loading datfile /home/anarcat/.cache/pyasn/ipasn_20250527.1600.dat.gz...
INFO: finished reading data
INFO: loading /home/anarcat/.cache/pyasn/asnames.json
count   percent ASN AS
12779   69.33   66496   SAMPLE, CA
3361    18.23   None    None
366 1.99    66497   EXAMPLE, FR
337 1.83    16276   OVH, FR
321 1.74    8075    MICROSOFT-CORP-MSN-AS-BLOCK, US
309 1.68    14061   DIGITALOCEAN-ASN, US
128 0.69    16509   AMAZON-02, US
77  0.42    48090   DMZHOST, GB
56  0.3 136907  HWCLOUDS-AS-AP HUAWEI CLOUDS, HK
53  0.29    17621   CNCGROUP-SH China Unicom Shanghai network, CN
total: 18433
count   percent prefix  ASN AS
12779   69.33   192.0.2.0/24    66496   SAMPLE, CA
3361    18.23   None        
298 1.62    178.128.208.0/20    14061   DIGITALOCEAN-ASN, US
289 1.57    51.222.0.0/16   16276   OVH, FR
272 1.48    2001:DB8::/48   66497   EXAMPLE, FR
235 1.27    172.160.0.0/11  8075    MICROSOFT-CORP-MSN-AS-BLOCK, US
94  0.51    2001:DB8:1::/48 66497   EXAMPLE, FR
72  0.39    47.128.0.0/14   16509   AMAZON-02, US
69  0.37    93.123.109.0/24 48090   DMZHOST, GB
53  0.29    27.115.124.0/24 17621   CNCGROUP-SH China Unicom Shanghai network, CN

Those numbers are actually from my home network, not GitLab. Over there, the battle still rages on, but at least the vampire bots are banging their heads against the solid Nginx wall instead of eating the fragile heart of GitLab. We had a significant improvement in latency thanks to the Facebook and Huawei blocks... Here are the "workhorse request duration stats" for various time ranges, 20h after the block:

range	mean	max	stdev
20h	449ms	958ms	39ms
7d	1.78s	5m	14.9s
30d	2.08s	3.86m	8.86s
6m	901ms	27.3s	2.43s

We went from two seconds mean to 500ms! And look at that standard deviation! 39ms! It was ten seconds before! I doubt we'll keep it that way very long but for now, it feels like I won a battle, and I didn't even have to setup anubis or go-away, although I suspect that will unfortunately come. Note that asncounter also supports exporting Prometheus metrics, but you should be careful with this, as it can lead to cardinal explosion, especially if you track by prefix (which can be disabled with --no-prefixes . Folks interested in more details should read the fine manual for more examples, usage, and discussion. It shows, among other things, how to effectively block lots of networks from Nginx, aggregate multiple prefixes, block entire ASNs, and more! So there you have it: I now have the tool I wish I had 20 years ago. Hopefully it will stay useful for another 20 years, although I'm not sure we'll have still have internet in 20 years. I welcome constructive feedback, "oh no you rewrote X", Grafana dashboards, bug reports, pull requests, and "hell yeah" comments. Hacker News, let it rip, I know you can give me another juicy quote for my blog. This work was done as part of my paid work for the Tor Project, currently in a fundraising drive, give us money if you like what you read.

• Please stop writing /var/log/btmp
• Restore sshd_config(5) documentation about rdomain

• ssh-agent: Improve systemd user service socket activation (Daniel Kahn Gillmor)
• Switch from adduser to sysusers.d (Luca Boccassi)
• Add sshd-keygen service (Luca Boccassi)

• celery
• django-modeltranslation (maintained by Freexian)
• django-phonenumber-field
• djangorestframework
• kombu
• orderly-set
• pox
• pydantic-extra-types
• python-cmarkgfm (fixing CVE-2022-39209, CVE-2023-22483, CVE-2023-22484, CVE-2023-22485, CVE-2023-22486, CVE-2023-24824, CVE-2023-26485, and CVE-2023-37463)
• python-django-crispy-forms
• python-django-extensions (fixing incompatibilities with Python 3.12: #1040091, #1040119)
• python-django-pgtrigger
• python-django-test-migrations
• python-holidays
• python-legacy-cgi
• python-pydash
• python-redis (4.3.4 to 5.2.1; needed some autopkgtest adjustments)
• python-tblib
• python-typing-extensions
• trove-classifiers
• xonsh

• python-django to 3:4.2.20-1 (issuing BSA-123)
• python-django-pgtrigger to 4.13.3

• billiard (contributed upstream)
• bleak-retry-connector
• cairosvg
• errbot (contributed upstream)
• haversine
• json-tricks
• jsonpickle (contributed upstream)
• lazygal
• mypy
• pydantic
• pydantic-core
• pydantic-settings
• pympress (contributed upstream)
• pysequoia
• pysodium (contributed upstream)
• python-decorator
• python-djvulibre (contributed upstream)
• python-momepy (contributed upstream)
• python-msgspec
• python-tz
• storm
• supysonic
• uvloop (contributed upstream)

• python-pydash: please make the build reproducible
• thunarx-python: fails to allow plugins to run / thunarx-python: fails to discover SONAME of libpython, tries to load /usr/lib/MULTIARCH/lib.so.1.0
• ttconv: package installs superfluous files under /usr/lib/python3/dist-packages

• python-vispy (also fixing unhandled failures to build documentation can occur while I was there)
• skimage

• [1] https://www.youtube.com/watch?v=rSpzAVpnXo4
• [2] https://tinyurl.com/25rpmgxx
• [3] https://www.youtube.com/watch?v=CYhncB9Ftt8
• [4] https://diziet.dreamwidth.org/19480.html
• [5] https://tinyurl.com/287ahynt
• [6] https://cheriot.org/
• [7] https://tinyurl.com/28x64xyc
• [8] https://tinyurl.com/237qycbw
• [9] https://drgabormate.com/trump-clinton-trauma/
• [10] https://tinyurl.com/22wmyogu

• [1] https://etbe.coker.com.au/2024/01/22/storage-trends-2024/

• debian-kernel mailing list:
  • posted Agenda items for kernel-team meeting on 2025-01-15 (and once more in that thread)
• debian-lts-announce mailing list:
  • posted [SECURITY] [DLA 4008-1] linux-6.1 security update
• Debian initramfs-tools package:
  • Merge requests:
    • opened and merged !153: Some small fixes
    • merged !145: Drop obsolete shellcheck overrides
    • merged !143: Declare breaking miniramfs 1.0.2 and older
    • merged !141: hook-functions: Add msm to fb modules list
    • merged !140: hook-functions: Handle msm hidden depency on qrtr
    • merged !127: Filter net kernel modules by symbol regexp from dracut
    • closed !119: Fix resume device type check
    • closed !89: init: Remove tmpfs from rootfstype option
• Debian jinja-vanish package:
  • Bugs:
    • updated #1092861: jinja-vanish: please run tests at build time
• Debian kernel-team package:
  • Merge requests:
    • merged !6: Support new linux branch layout
• Debian linux package:
  • Merge requests:
    • merged !1296: d/p/debian/cdc_ncm-cdc_mbim-use-ncm-by-default.patch: drop
  • Updated the bullseye-security branch, but didn t upload
• Debian wireless-regdb package:
  • Bugs:
    • closed #1092796: Autopkgtests fail with Python 3.13: OSError: [Errno 9] Bad file descriptor
  • Merge requests:
    • merged !6: d/tests/check-signatures: Fix subprocess cleanup
  • Uploads:
    • uploaded version 2024.10.07-2 to unstable

• My years in Iztacala: I worked as a sysadmin between 1999 and 2003. When I arrived, we already had two servers, campus and tlali, and one computer pending installation, ollin. The credit for their names is not mine.
  • campus: A mighty SPARCstation 5! Because it was the main (and for some time, the only!) server in our campus.
  • tlali: A regular PC used as a Linux server. Tlali means something like lands in n huatl, the prehispanic language spoken in central Mexico. My workplace was Iztacala, which translates as the place where there are white houses ; tlali and cali are related words.
  • ollin: was a big IBM RS/6000 system running AIX. It came to us, probably already obsolete, as a (useless) donation from Fundaci n UNAM; I don t recall the exact model, but it looked very much like this one. Ran on AIX. We had no software for it, and frankly never really got it to be productive. Funnily, its name Ollin means movement in N huatl. I added some servers to the lineup during the two years I was in Iztacala:
  • tlamantli: An Alpha 21164 server that doubled as my desktop. Given the tradition in Iztacala of naming things in N huatl, but trying to be somewhat funny, tlamantli just means a thing; I understand the word is usually bound to a quantifier.
  • tepancuate: A regular PC system we set up with OpenBSD as a firewall. It means wall in N huatl.
• Following the first CONSOL (National Free Software Conference), I was invited to work as a programmer at UPN, Universidad Pedag gica Nacional in 2003 2004. There I was not directly in charge of any of the servers (I mostly used ajusco, managed by V ctor, named after the mountain on whose slopes our campus was). But my only computer there was:
  • shmate: , meaning old rag in yiddish. The word shmate is used like thingy, although it would usually mean old and slightly worn-out thingy. It was a quite nice machine, though. I had a Pentium 4 with 512MB RAM, not bad for 2003!
• I started my present work at Instituto de Investigaciones Econ micas, UNAM 20 years ago(!), in 2005. Here I am a systems administrator, so naturally I am in charge of the servers. And over the years, we have had a fair share of machines:
  • mosca: is my desktop. It has changed hardware several times (of course) over the years, but it s still the same Debian Sid install I did in January 2005 (I must have reinstalled once, when I got it replaced by an AMD64). Its name is the Spanish name for the common fly. I have often used it to describe my work, since I got in the early 1990s an automated bilingual translator called TRANSLATE; it came on seven 5.25 floppies. As a teenager, I somehow got my hands on a copy, and installed it in my 80386SX. Fed it its own README to see how it fared. And the first sentence made me burst in laughter: TRANSLATE performs on the fly translation TRADUCE realiza traducci n sobre la mosca . Starting then, I always think of on the fly as sobre la mosca . As Groucho said, I guess Time flies like an arrow, but fruit flies like a banana.
  • lafa When I got there, we didn t have any servers; for some time, I took one of the computer lab s systems to serve our web page and receive mail. But when we got some budget approved, we bought a fsckin-big server. Big as in four-rack-units. Double CPUs (not multicore, but two independent early Xeon CPUs, if I m not mistaken. Still, it was still a 32 bits system). (lafa) is a big, more flexible kind of Arab bread than pita; I loved it when I lived in Israel. And there is an album (and song) by Teapacks, an Israeli group I am very fond of, hajaim shelja belafa (your life in a lafa), saying, hey, brother! Your life is in a lafa. You throw everything in a big pita. You didn t have time to chew, you already swallowed it .
  • joma: Our firewall. means wall in Hebrew.
  • baktun: lafa was great, but over the years, it got old. After many years, I finally got the Institute to buy a second server. We got it in December 2012. There was a lot of noise around then because the world was supposed to end on 2012.12.21, as the Mayan calendar reached a full long cycle. This long cycle is called /baktun/. So, it was fitting as the name of the new server.
  • teom: As lafa was almost immediately decomissioned and turned into a virtual machine in the much bigger baktun,, I wanted to split services, make off-hardware backups, and such. Almost two years later, my request was approved and we bought a second server. But instead of buying it from a regular provider, we got it off a stash of machines bought by our university s central IT entity. To my surprise, it had the exact same hardware configuration as baktun, bought two years earlier. Even the serial number was absurdly close. So, I had it as baktun s long-lost twin. Hence, (transliterated as teom), the Hebrew word for twin. About a year after teom arrived to my life, my twin children were also born, but their naming followed a completely different logic process than my computers
• At home or on the road: I am sure I am missing several systems over the years.
  • pato: The earliest system I had that I remember giving a name to. I built a 80386SX in 1991, buying each component separately. The box had a 1-inch square for integrators to put their branding And after some time, I carefully printed and applied a label that said Catarm quina PATO (the first word, very small). Pato (duck) is how we d call a no-brand system. Catarm quina because it was the system where I ran my BBS, CatarSYS (1992-1994).
  • malenkaya: In 2008 I got a 9 Acer Aspire One netbook (Atom N270 i386, 1GB RAM). I really loved that machine! Although it was quite limited, it was my main computer while on the road for almost five years. malenkaya means small (for female) in Russian.
  • matlalli: After malenkaya started being too limited for my regular use, I bought its successor Acer Aspire One model. This one was way larger (10.1 inches screen) and I wasn t too happy about it at the beginning, but I ended up loving it. So much, in fact, that we bought at least four very similar such computers for us and our family members. This computer was dirt cheap, and endured five further years of lugging everywhere. matlalli is due to its turquoise color: it is the N huatl word for blue or green.
  • cajita: In 2014 I got a beautiful Cubox i4 Pro computer. It took me some time to get it to boot and be generally useful, but it ended up being my home server for many years, until I had a power supply malfunction which bricked it. cajita means little box in Spanish.
  • pitentzin: Another 10.1 Acer Aspire One (the last in the lineup; the CPU is a Celeron 877, so it does run AMD64, and it supports up to 16GB RAM, I think I have it with 12). We originally bought it for my family in Argentina, but they didn t really use it much, and after a couple of years we got it back. We decided it would be the computer for the kids, at least for the time being. And although it is a 2013 laptop, it s still our everyday media station driver. Oh, and the name pitentzin? N huatl for /children/.
  • tliltik: In 2018, I bought a second-hand Thinkpad X230. It was my daily driver for about three years. I reflashed its firmware with CoreBoot, and repeated the experience for seven people IIRC in DebConf18. With it, I learned to love the Thinkpad keyboard. Naturally for a thinkpad, tliltik means black in N huatl.
  • uesebe: When COVID struck, we were all sent home, and my university lent me a nice recently bought Intel i7 HP laptop. At first, I didn t want to mess up its Windows install (so I set up a USB-drive-based installation, hence the name uesebe); when it was clear the lockdown was going to be long (and that tliltik had too many aches to be used for my daily work), I transferred the install to its HDD and used it throughout the pandemic, until mid 2022.
  • bolex: I bought this computer for my father in 2020. After he passed away in May 2022, I took his computer, and named it bolex because that s the brand of the 8mm cinema camera he loved and had since 1955, and with which he created most of his films. It is really an entry-level machine, though (a single-core, dual-threaded Celeron), and it was too limited when I started distance-teaching again, so I had to store it as an emergency system.
  • yogurtu: During the pandemics, I spent quite a bit of time fiddling with the Raspberry Pi family. But all in all, while they are nice machines for many uses, they are too limited to be daily drivers. Or even enough for taking i.e. to Debconf and have them be my conference computer. I bought an almost-new-but-used ( 2 year old) Yoga C630 ARM laptop. I often brag about my happy experience with it, and how it brings a reasonably powerful ARM Linux system to my everyday life. In our last DebConf, I didn t even pick up my USB-C power connector every day; the battery just lasts over ten hours of active work. But I m not here doing ads, right? yogurtu naturally is derived from the Yoga brand it has, but is taken from Yogurtu Ngh , a fictional character by the Argentinian comical-musical group Les Luthiers, that has marked my life.
  • misnenet: Towards mid 2023, when it was clear that bolex would not be a good daily driver, and considering we would be spending six months in Argentina, I bought a new desktop system. It seems I have something for small computers: I decided for a refurbished HP EliteDesk 800 G5 Mini i7 system. I picked it because, at close to 18 18 3.5cm it perfectly fits in my DebConf18 bag. A laptop, it is clearly not, but it can easily travel with me when needed. Oh, and the name? Because for this model, HP uses different enclosures based on the kind of processor: The i3 model has a flat, black aluminum top But mine has lots of tiny holes, covering two areas of roughly 15 7cm, with a tiny hole every ~2mm, and with a solid strip between them. Of course, (misnenet, in Hebrew) means strainer.