Search Results: "ag"

7 March 2023

Dirk Eddelbuettel: RcppFastAD 0.0.1 and 0.0.2: New Package on CRAN!

James Yang and I are thrilled to announce the new CRAN package RcppFastAD which arrived at CRAN last Monday as version 0.0.1, and is as of today at version 0.0.2 with a first set of small updates. It is based on the FastAD header-only C++ library by James which provides a C++ implementation of both forward and reverse mode of automatic differentiation in an easy-to-use header library (which we wrapped here) that is both lightweight and performant. With a little of bit of Rcpp glue, it is also easy to use from R in simple C++ applications. Included in the package are three example: a simple quadratic expression evaluating x' S x for given x and S return the expression value with a gradient, a linear regression example generalising this and using the gradient to derive to arrive at the least-squares minimizing solution, as well as the well-known Black-Scholes options pricer and its important partial derivatives delta, rho, theta and vega derived via automatic differentiation. The NEWS file for these two initial releases follows.

Changes in version 0.0.2 (2023-03-05)
  • One C++ operation is protected from operating on a nullptr
  • Additional tests have been added, tests now cover all three demo / example functions
  • Return values and code for the examples linear_regression and quadratic_expression have been adjusted

Changes in version 0.0.1 (2023-02-24)
  • Initial release version and CRAN upload

Courtesy of my CRANberries, there is also a diffstat report for the most recent release. More information is available at the repository or the package page. If you like this or other open-source work I do, you can now sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

6 March 2023

Vincent Bernat: DDoS detection and remediation with Akvorado and Flowspec

Akvorado collects sFlow and IPFIX flows, stores them in a ClickHouse database, and presents them in a web console. Although it lacks built-in DDoS detection, it s possible to create one by crafting custom ClickHouse queries.

DDoS detection Let s assume we want to detect DDoS targeting our customers. As an example, we consider a DDoS attack as a collection of flows over one minute targeting a single customer IP address, from a single source port and matching one of these conditions:
  • an average bandwidth of 1 Gbps,
  • an average bandwidth of 200 Mbps when the protocol is UDP,
  • more than 20 source IP addresses and an average bandwidth of 100 Mbps, or
  • more than 10 source countries and an average bandwidth of 100 Mbps.
Here is the SQL query to detect such attacks over the last 5 minutes: 
SELECT *
FROM (
  SELECT
    toStartOfMinute(TimeReceived) AS TimeReceived,
    DstAddr,
    SrcPort,
    dictGetOrDefault('protocols', 'name', Proto, '???') AS Proto,
    SUM(((((Bytes * SamplingRate) * 8) / 1000) / 1000) / 1000) / 60 AS Gbps,
    uniq(SrcAddr) AS sources,
    uniq(SrcCountry) AS countries
  FROM flows
  WHERE TimeReceived > now() - INTERVAL 5 MINUTE
    AND DstNetRole = 'customers'
  GROUP BY
    TimeReceived,
    DstAddr,
    SrcPort,
    Proto
)
WHERE (Gbps > 1)
   OR ((Proto = 'UDP') AND (Gbps > 0.2)) 
   OR ((sources > 20) AND (Gbps > 0.1)) 
   OR ((countries > 10) AND (Gbps > 0.1))
ORDER BY
  TimeReceived DESC,
  Gbps DESC
Here is an example output1 where two of our users are under attack. One from what looks like an NTP amplification attack, the other from a DNS amplification attack:
TimeReceived DstAddr SrcPort Proto Gbps sources countries
2023-02-26 17:44:00 ::ffff:203.0.113.206 123 UDP 0.102 109 13
2023-02-26 17:43:00 ::ffff:203.0.113.206 123 UDP 0.130 133 17
2023-02-26 17:43:00 ::ffff:203.0.113.68 53 UDP 0.129 364 63
2023-02-26 17:43:00 ::ffff:203.0.113.206 123 UDP 0.113 129 21
2023-02-26 17:42:00 ::ffff:203.0.113.206 123 UDP 0.139 50 14
2023-02-26 17:42:00 ::ffff:203.0.113.206 123 UDP 0.105 42 14
2023-02-26 17:40:00 ::ffff:203.0.113.68 53 UDP 0.121 340 65

DDoS remediation Once detected, there are at least two ways to stop the attack at the network level:
  • blackhole the traffic to the targeted user (RTBH), or
  • selectively drop packets matching the attack patterns (Flowspec).

Traffic blackhole The easiest method is to sacrifice the attacked user. While this helps the attacker, this protects your network. It is a method supported by all routers. You can also offload this protection to many transit providers. This is useful if the attack volume exceeds your internet capacity. This works by advertising with BGP a route to the attacked user with a specific community. The border router modifies the next hop address of these routes to a specific IP address configured to forward the traffic to a null interface. RFC 7999 defines 65535:666 for this purpose. This is known as a remote-triggered blackhole (RTBH) and is explained in more detail in RFC 3882. It is also possible to blackhole the source of the attacks by leveraging unicast Reverse Path Forwarding (uRPF) from RFC 3704, as explained in RFC 5635. However, uRPF can be a serious tax on your router resources. See NCS5500 uRPF: Configuration and Impact on Scale for an example of the kind of restrictions you have to expect when enabling uRPF. On the advertising side, we can use BIRD. Here is a complete configuration file to allow any router to collect them: 
log stderr all;
router id 192.0.2.1;
protocol device  
  scan time 10;
 
protocol bgp exporter  
  ipv4  
    import none;
    export where proto = "blackhole4";
   ;
  ipv6  
    import none;
    export where proto = "blackhole6";
   ;
  local as 64666;
  neighbor range 192.0.2.0/24 external;
  multihop;
  dynamic name "exporter";
  dynamic name digits 2;
  graceful restart yes;
  graceful restart time 0;
  long lived graceful restart yes;
  long lived stale time 3600;  # keep routes for 1 hour!
 
protocol static blackhole4  
  ipv4;
  route 203.0.113.206/32 blackhole  
    bgp_community.add((65535, 666));
   ;
  route 203.0.113.68/32 blackhole  
    bgp_community.add((65535, 666));
   ;
 
protocol static blackhole6  
  ipv6;
We use BGP long-lived graceful restart to ensure routes are kept for one hour, even if the BGP connection goes down, notably during maintenance. On the receiver side, if you have a Cisco router running IOS XR, you can use the following configuration to blackhole traffic received on the BGP session. As the BGP session is dedicated to this usage, The community is not used, but you can also forward these routes to your transit providers. 
router static
 vrf public
  address-family ipv4 unicast
   192.0.2.1/32 Null0 description "BGP blackhole"
  !
  address-family ipv6 unicast
   2001:db8::1/128 Null0 description "BGP blackhole"
  !
 !
!
route-policy blackhole_ipv4_in_public
  if destination in (0.0.0.0/0 le 31) then
    drop
  endif
  set next-hop 192.0.2.1
  done
end-policy
!
route-policy blackhole_ipv6_in_public
  if destination in (::/0 le 127) then
    drop
  endif
  set next-hop 2001:db8::1
  done
end-policy
!
router bgp 12322
 neighbor-group BLACKHOLE_IPV4_PUBLIC
  remote-as 64666
  ebgp-multihop 255
  update-source Loopback10
  address-family ipv4 unicast
   maximum-prefix 100 90
   route-policy blackhole_ipv4_in_public in
   route-policy drop out
   long-lived-graceful-restart stale-time send 86400 accept 86400
  !
  address-family ipv6 unicast
   maximum-prefix 100 90
   route-policy blackhole_ipv6_in_public in
   route-policy drop out
   long-lived-graceful-restart stale-time send 86400 accept 86400
  !
 !
 vrf public
  neighbor 192.0.2.1
   use neighbor-group BLACKHOLE_IPV4_PUBLIC
   description akvorado-1
When the traffic is blackholed, it is still reported by IPFIX and sFlow. In Akvorado, use ForwardingStatus >= 128 as a filter. While this method is compatible with all routers, it makes the attack successful as the target is completely unreachable. If your router supports it, Flowspec can selectively filter flows to stop the attack without impacting the customer.

Flowspec Flowspec is defined in RFC 8955 and enables the transmission of flow specifications in BGP sessions. A flow specification is a set of matching criteria to apply to IP traffic. These criteria include the source and destination prefix, the IP protocol, the source and destination port, and the packet length. Each flow specification is associated with an action, encoded as an extended community: traffic shaping, traffic marking, or redirection. To announce flow specifications with BIRD, we extend our configuration. The extended community used shapes the matching traffic to 0 bytes per second. 
flow4 table flowtab4;
flow6 table flowtab6;
protocol bgp exporter  
  flow4  
    import none;
    export where proto = "flowspec4";
   ;
  flow6  
    import none;
    export where proto = "flowspec6";
   ;
  # [ ]
 
protocol static flowspec4  
  flow4;
  route flow4  
    dst 203.0.113.68/32;
    sport = 53;
    length >= 1476 && <= 1500;
    proto = 17;
   
    bgp_ext_community.add((generic, 0x80060000, 0x00000000));
   ;
  route flow4  
    dst 203.0.113.206/32;
    sport = 123;
    length = 468;
    proto = 17;
   
    bgp_ext_community.add((generic, 0x80060000, 0x00000000));
   ;
 
protocol static flowspec6  
  flow6;
If you have a Cisco router running IOS XR, the configuration may look like this: 
vrf public
 address-family ipv4 flowspec
 address-family ipv6 flowspec
!
router bgp 12322
 address-family vpnv4 flowspec
 address-family vpnv6 flowspec
 neighbor-group FLOWSPEC_IPV4_PUBLIC
  remote-as 64666
  ebgp-multihop 255
  update-source Loopback10
  address-family ipv4 flowspec
   long-lived-graceful-restart stale-time send 86400 accept 86400
   route-policy accept in
   route-policy drop out
   maximum-prefix 100 90
   validation disable
  !
  address-family ipv6 flowspec
   long-lived-graceful-restart stale-time send 86400 accept 86400
   route-policy accept in
   route-policy drop out
   maximum-prefix 100 90
   validation disable
  !
 !
 vrf public
  address-family ipv4 flowspec
  address-family ipv6 flowspec
  neighbor 192.0.2.1
   use neighbor-group FLOWSPEC_IPV4_PUBLIC
   description akvorado-1
Then, you need to enable Flowspec on all interfaces with: 
flowspec
 vrf public
  address-family ipv4
   local-install interface-all
  !
  address-family ipv6
   local-install interface-all
  !
 !
!
As with the RTBH setup, you can filter dropped flows with ForwardingStatus >= 128.

DDoS detection (continued) In the example using Flowspec, the flows were also filtered on the length of the packet: 
route flow4  
  dst 203.0.113.68/32;
  sport = 53;
  length >= 1476 && <= 1500;
  proto = 17;
 
  bgp_ext_community.add((generic, 0x80060000, 0x00000000));
 ;
This is an important addition: legitimate DNS requests are smaller than this and therefore not filtered.2 With ClickHouse, you can get the 10th and 90th percentiles of the packet sizes with quantiles(0.1, 0.9)(Bytes/Packets). The last issue we need to tackle is how to optimize the request: it may need several seconds to collect the data and it is likely to consume substantial resources from your ClickHouse database. One solution is to create a materialized view to pre-aggregate results: 
CREATE TABLE ddos_logs (
  TimeReceived DateTime,
  DstAddr IPv6,
  Proto UInt32,
  SrcPort UInt16,
  Gbps SimpleAggregateFunction(sum, Float64),
  Mpps SimpleAggregateFunction(sum, Float64),
  sources AggregateFunction(uniqCombined(12), IPv6),
  countries AggregateFunction(uniqCombined(12), FixedString(2)),
  size AggregateFunction(quantiles(0.1, 0.9), UInt64)
) ENGINE = SummingMergeTree
PARTITION BY toStartOfHour(TimeReceived)
ORDER BY (TimeReceived, DstAddr, Proto, SrcPort)
TTL toStartOfHour(TimeReceived) + INTERVAL 6 HOUR DELETE ;
CREATE MATERIALIZED VIEW ddos_logs_view TO ddos_logs AS
  SELECT
    toStartOfMinute(TimeReceived) AS TimeReceived,
    DstAddr,
    Proto,
    SrcPort,
    sum(((((Bytes * SamplingRate) * 8) / 1000) / 1000) / 1000) / 60 AS Gbps,
    sum(((Packets * SamplingRate) / 1000) / 1000) / 60 AS Mpps,
    uniqCombinedState(12)(SrcAddr) AS sources,
    uniqCombinedState(12)(SrcCountry) AS countries,
    quantilesState(0.1, 0.9)(toUInt64(Bytes/Packets)) AS size
  FROM flows
  WHERE DstNetRole = 'customers'
  GROUP BY
    TimeReceived,
    DstAddr,
    Proto,
    SrcPort
The ddos_logs table is using the SummingMergeTree engine. When the table receives new data, ClickHouse replaces all the rows with the same sorting key, as defined by the ORDER BY directive, with one row which contains summarized values using either the sum() function or the explicitly specified aggregate function (uniqCombined and quantiles in our example).3 Finally, we can modify our initial query with the following one: 
SELECT *
FROM (
  SELECT
    TimeReceived,
    DstAddr,
    dictGetOrDefault('protocols', 'name', Proto, '???') AS Proto,
    SrcPort,
    sum(Gbps) AS Gbps,
    sum(Mpps) AS Mpps,
    uniqCombinedMerge(12)(sources) AS sources,
    uniqCombinedMerge(12)(countries) AS countries,
    quantilesMerge(0.1, 0.9)(size) AS size
  FROM ddos_logs
  WHERE TimeReceived > now() - INTERVAL 60 MINUTE
  GROUP BY
    TimeReceived,
    DstAddr,
    Proto,
    SrcPort
)
WHERE (Gbps > 1)
   OR ((Proto = 'UDP') AND (Gbps > 0.2)) 
   OR ((sources > 20) AND (Gbps > 0.1)) 
   OR ((countries > 10) AND (Gbps > 0.1))
ORDER BY
  TimeReceived DESC,
  Gbps DESC

Gluing everything together To sum up, building an anti-DDoS system requires to following these steps:
  1. define a set of criteria to detect a DDoS attack,
  2. translate these criteria into SQL requests,
  3. pre-aggregate flows into SummingMergeTree tables,
  4. query and transform the results to a BIRD configuration file, and
  5. configure your routers to pull the routes from BIRD.
A Python script like the following one can handle the fourth step. For each attacked target, it generates both a Flowspec rule and a blackhole route. 
import socket
import types
from clickhouse_driver import Client as CHClient
# Put your SQL query here!
SQL_QUERY = " "
# How many anti-DDoS rules we want at the same time?
MAX_DDOS_RULES = 20
def empty_ruleset():
    ruleset = types.SimpleNamespace()
    ruleset.flowspec = types.SimpleNamespace()
    ruleset.blackhole = types.SimpleNamespace()
    ruleset.flowspec.v4 = []
    ruleset.flowspec.v6 = []
    ruleset.blackhole.v4 = []
    ruleset.blackhole.v6 = []
    return ruleset
current_ruleset = empty_ruleset()
client = CHClient(host="clickhouse.akvorado.net")
while True:
    results = client.execute(SQL_QUERY)
    seen =  
    new_ruleset = empty_ruleset()
    for (t, addr, proto, port, gbps, mpps, sources, countries, size) in results:
        if (addr, proto, port) in seen:
            continue
        seen[(addr, proto, port)] = True
        # Flowspec
        if addr.ipv4_mapped:
            address = addr.ipv4_mapped
            rules = new_ruleset.flowspec.v4
            table = "flow4"
            mask = 32
            nh = "proto"
        else:
            address = addr
            rules = new_ruleset.flowspec.v6
            table = "flow6"
            mask = 128
            nh = "next header"
        if size[0] == size[1]:
            length = f"length =  int(size[0]) "
        else:
            length = f"length >=  int(size[0])  && <=  int(size[1]) "
        header = f"""
# Time:  t 
# Source:  address , protocol:  proto , port:  port 
# Gbps/Mpps:  gbps:.3 / mpps:.3 , packet size:  int(size[0]) <=X<= int(size[1]) 
# Flows:  flows , sources:  sources , countries:  countries 
"""
        rules.append(
                f""" header 
route  table   
  dst  address / mask ;
  sport =  port ;
   length ;
   nh  =  socket.getprotobyname(proto) ;
 
  bgp_ext_community.add((generic, 0x80060000, 0x00000000));
 ;
"""
        )
        # Blackhole
        if addr.ipv4_mapped:
            rules = new_ruleset.blackhole.v4
        else:
            rules = new_ruleset.blackhole.v6
        rules.append(
            f""" header 
route  address / mask  blackhole  
  bgp_community.add((65535, 666));
 ;
"""
        )
        new_ruleset.flowspec.v4 = list(
            set(new_ruleset.flowspec.v4[:MAX_DDOS_RULES])
        )
        new_ruleset.flowspec.v6 = list(
            set(new_ruleset.flowspec.v6[:MAX_DDOS_RULES])
        )
        # TODO: advertise changes by mail, chat, ...
        current_ruleset = new_ruleset
        changes = False
        for rules, path in (
            (current_ruleset.flowspec.v4, "v4-flowspec"),
            (current_ruleset.flowspec.v6, "v6-flowspec"),
            (current_ruleset.blackhole.v4, "v4-blackhole"),
            (current_ruleset.blackhole.v6, "v6-blackhole"),
        ):
            path = os.path.join("/etc/bird/", f" path .conf")
            with open(f" path .tmp", "w") as f:
                for r in rules:
                    f.write(r)
            changes = (
                changes or not os.path.exists(path) or not samefile(path, f" path .tmp")
            )
            os.rename(f" path .tmp", path)
        if not changes:
            continue
        proc = subprocess.Popen(
            ["birdc", "configure"],
            stdin=subprocess.DEVNULL,
            stdout=subprocess.PIPE,
            stderr=subprocess.PIPE,
        )
        stdout, stderr = proc.communicate(None)
        stdout = stdout.decode("utf-8", "replace")
        stderr = stderr.decode("utf-8", "replace")
        if proc.returncode != 0:
            logger.error(
                "  error:\n \n ".format(
                    "birdc reconfigure",
                    "\n".join(
                        [" O:  ".format(line) for line in stdout.rstrip().split("\n")]
                    ),
                    "\n".join(
                        [" E:  ".format(line) for line in stderr.rstrip().split("\n")]
                    ),
                )
            )
Until Akvorado integrates DDoS detection and mitigation, the ideas presented in this blog post provide a solid foundation to get started with your own anti-DDoS system.
  1. ClickHouse can export results using Markdown format when appending FORMAT Markdown to the query.
  2. While most DNS clients should retry with TCP on failures, this is not always the case: until recently, musl libc did not implement this.
  3. The materialized view also aggregates the data at hand, both for efficiency and to ensure we work with the right data types.

Valhalla's Things: Bookbinding: photo album

Posted on March 6, 2023
an open book with a watercolour of a costume pasted from two corners on one page; near the spine there is a sliver of paper as a spacer. When I paint postcards I tend to start with a draft (usually on lightweight (250 g/m ) watercolour paper, then trace1 the drawing on blank postcards and paint it again. I keep the drafts for a number of reasons; for the views / architectural ones I m using a landscape photo album that I bought many years ago, but lately I ve also sent a few cards with my historical outfits to people who like to be kept updated on that, and I wanted a different book for those, both for better organization and to be able to keep them in the portrait direction. If you know me, you can easily guess that buying one wasn t considered as an option. A closed hardcover book in uniform dark grey. Since I m not going to be writing on the pages, I decided to use a relatively cheap 200 g / m linoprint paper with a nice feel, and I ve settled on a B6 size (before trimming) to hold A6 postcard drafts. For the binding I ve decided to use a technique I ve learned from a craft book ages ago that doesn t use tapes, and added a full hard cover in dark grey linen-feel2 paper. For the end-papers I ve used some random sheets of light blue paper (probably around 100-something g / m ), and that s the thing where I could have done better, but they work. Up to now there isn t anything I hadn t done before, what was new was the fact that this book was meant to hold things between the pages, and I needed to provide space for them. a book seen from the top: near the spine all signatures are made of 4 sheets, but two of them for each signature are just stubs, and leave open spaces between the pages. After looking on the internet for solutions, I settled on adding spacers by making a signature composed of paper - spacer - paper - spacer, with the spacers being 2 cm wide, folded in half. And then, between finishing binding the book and making the cover I utterly forgot to add the head bands. Argh. It s not the first time I make this error. The same book, open on an empty page. I m happy enough with the result. There are things that are easy to improve on in the next iteration (endpapers and head bands), and something in me is not 100% happy with the fact that the spacers aren t placed between every sheet, but there are places with no spacer and places with two of them, but I can t think of (and couldn t find) a way to make them otherwise with a sewn book, unless I sew each individual sheet, which sounds way too bulky (the album I m using for the landscapes was glued, but I didn t really want to go that way). The size is smaller than the other one I was using and doesn t leave a lot of room around the paintings, but that isn t necessarily a bad thing, because it also means less wasted space. I believe that one of my next project will be another similar book in a landscape format, for those postcard drafts that aren t landscapes nor clothing related. And then maybe another? or two? or 
Traceback (most recent call last):
TooManyProjectsError: project queue is full
  1. yes, trace. I can t draw. I have too many hobbies to spend the required amount of time every day to practice it. I m going to fake it. 85% of the time I m tracing from a photo I took myself, so I m not even going to consider it cheating.
  2. the description of which, on the online shop, made it look like fabric, even if the price was suspiciously low, so I bought a sheet to see what it was. It wasn t fabric. It feels and looks nice, but I m not sure how sturdy it s going to be.

5 March 2023

Enrico Zini: Generating MIDI events with JACK and Python

I had a go at trying to figure out how to generate arbitrary MIDI events and send them out over a JACK MIDI channel. Setting up JACK and Pipewire Pipewire has a JACK interface, which in theory means one could use JACK clients out of the box without extra setup. In practice, one need to tell JACK clients which set of libraries to use to communicate to servers, and Pipewire's JACK server is not the default choice. To tell JACK clients to use Pipewire's server, you can either: Programming with JACK Python has a JACK client library that worked flawlessly for me so far. Everything with JACK is designed around minimizing latency. Everything happens around a callback that gets called form a separate thread, and which gets a buffer to fill with events. All the heavy processing needs to happen outside the callback, and the callback is only there to do the minimal amount of work needed to shovel the data your application produced into JACK channels. Generating MIDI messages The Mido library can be used to parse and create MIDI messages and it also worked flawlessly for me so far. One needs to study a bit what kind of MIDI message one needs to generate (like "note on", "note off", "program change") and what arguments they get. It also helps to read about the General MIDI standard which defines mappings between well-known instruments and channels and instrument numbers in MIDI messages. A timed message queue To keep a queue of events that happen over time, I implemented a Delta List that indexes events by their future frame number. I called the humble container for my audio experiments pyeep and here's my delta list implementation. A JACK player The simple JACK MIDI player backend is also in pyeep. It needs to protect the delta list with a mutex since we are working across thread boundaries, but it tries to do as little work under lock as possible, to minimize the risk of locking the realtime thread for too long. The play method converts delays in seconds to frame counts, and the on_process callback moves events from the queue to the jack output. Here's an example script that plays a simple drum pattern: 
#!/usr/bin/python3
# Example JACK midi event generator
#
# Play a drum pattern over JACK
import time
from pyeep.jackmidi import MidiPlayer
# See:
# https://soundprogramming.net/file-formats/general-midi-instrument-list/
# https://www.pgmusic.com/tutorial_gm.htm
DRUM_CHANNEL = 9
with MidiPlayer("pyeep drums") as player:
    beat: int = 0
    while True:
        player.play("note_on", velocity=64, note=35, channel=DRUM_CHANNEL)
        player.play("note_off", note=38, channel=DRUM_CHANNEL, delay_sec=0.5)
        if beat == 0:
            player.play("note_on", velocity=100, note=38, channel=DRUM_CHANNEL)
            player.play("note_off", note=36, channel=DRUM_CHANNEL, delay_sec=0.3)
        if beat + 1 == 2:
            player.play("note_on", velocity=100, note=42, channel=DRUM_CHANNEL)
            player.play("note_off", note=42, channel=DRUM_CHANNEL, delay_sec=0.3)
        beat = (beat + 1) % 4
        time.sleep(0.3)
Running the example I ran the jack_drums script, and of course not much happened. First I needed a MIDI synthesizer. I installed fluidsynth, and ran it on the command line with no arguments. it registered with JACK, ready to do its thing. Then I connected things together. I used qjackctl, opened the graph view, and connected the MIDI output of "pyeep drums" to the "FLUID Synth input port". fluidsynth's output was already automatically connected to the audio card and I started hearing the drums playing!

Reproducible Builds: Reproducible Builds in February 2023

Welcome to the February 2023 report from the Reproducible Builds project. As ever, if you are interested in contributing to our project, please visit the Contribute page on our website.
FOSDEM 2023 was held in Brussels on the 4th & 5th of February and featured a number of talks related to reproducibility. In particular, Akihiro Suda gave a talk titled Bit-for-bit reproducible builds with Dockerfile discussing deterministic timestamps and deterministic apt-get (original announcement). There was also an entire track of talks on Software Bill of Materials (SBOMs). SBOMs are an inventory for software with the intention of increasing the transparency of software components (the US National Telecommunications and Information Administration (NTIA) published a useful Myths vs. Facts document in 2021).
On our mailing list this month, Larry Doolittle was puzzled why the Debian verilator package was not reproducible [ ], but Chris Lamb pointed out that this was due to the use of Python s datetime.fromtimestamp over datetime.utcfromtimestamp [ ].
James Addison also was having issues with a Debian package: in this case, the alembic package. Chris Lamb was also able to identify the Sphinx documentation generator as the cause of the problem, and provided a potential patch that might fix it. This was later filed upstream [ ].
Anthony Harrison wrote to our list twice, first by introducing himself and their background and later to mention the increasing relevance of Software Bill of Materials (SBOMs):
As I am sure everyone is aware, there is a growing interest in [SBOMs] as a way of improving software security and resilience. In the last two years, the US through the Exec Order, the EU through the proposed Cyber Resilience Act (CRA) and this month the UK has issued a consultation paper looking at software security and SBOMs appear very prominently in each publication. [ ]

Tim Retout wrote a blog post discussing AlmaLinux in the context of CentOS, RHEL and supply-chain security in general [ ]:
Alma are generating and publishing Software Bill of Material (SBOM) files for every package; these are becoming a requirement for all software sold to the US federal government. What s more, they are sending these SBOMs to a third party (CodeNotary) who store them in some sort of Merkle tree system to make it difficult for people to tamper with later. This should theoretically allow end users of the distribution to verify the supply chain of the packages they have installed?

Debian

F-Droid & Android

diffoscope diffoscope is our in-depth and content-aware diff utility. Not only can it locate and diagnose reproducibility issues, it can provide human-readable diffs from many kinds of binary formats. This month, Chris Lamb released versions 235 and 236; Mattia Rizzolo later released version 237. Contributions include:
  • Chris Lamb:
    • Fix compatibility with PyPDF2 (re. issue #331) [ ][ ][ ].
    • Fix compatibility with ImageMagick version 7.1 [ ].
    • Require at least version 23.1.0 to run the Black source code tests [ ].
    • Update debian/tests/control after merging changes from others [ ].
    • Don t write test data during a test [ ].
    • Update copyright years [ ].
    • Merged a large number of changes from others.
  • Akihiro Suda edited the .gitlab-ci.yml configuration file to ensure that versioned tags are pushed to the container registry [ ].
  • Daniel Kahn Gillmor provided a way to migrate from PyPDF2 to pypdf (#1029741).
  • Efraim Flashner updated the tool metadata for isoinfo on GNU Guix [ ].
  • FC Stegerman added support for Android resources.arsc files [ ], improved a number of file-matching regular expressions [ ][ ] and added support for Android dexdump [ ]; they also fixed a test failure (#1031433) caused by Debian s black package having been updated to a newer version.
  • Mattia Rizzolo:
    • updated the release documentation [ ],
    • fixed a number of Flake8 errors [ ][ ],
    • updated the autopkgtest configuration to only install aapt and dexdump on architectures where they are available [ ], making sure that the latest diffoscope release is in a good fit for the upcoming Debian bookworm freeze.

reprotest Reprotest version 0.7.23 was uploaded to both PyPI and Debian unstable, including the following changes:
  • Holger Levsen improved a lot of documentation [ ][ ][ ], tidied the documentation as well [ ][ ], and experimented with a new --random-locale flag [ ].
  • Vagrant Cascadian adjusted reprotest to no longer randomise the build locale and use a UTF-8 supported locale instead [ ] (re. #925879, #1004950), and to also support passing --vary=locales.locale=LOCALE to specify the locale to vary [ ].
Separate to this, Vagrant Cascadian started a thread on our mailing list questioning the future development and direction of reprotest.

Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

Testing framework The Reproducible Builds project operates a comprehensive testing framework (available at tests.reproducible-builds.org) in order to check packages and other artifacts for reproducibility. In February, the following changes were made by Holger Levsen:
  • Add three new OSUOSL nodes [ ][ ][ ] and decommission the osuosl174 node [ ].
  • Change the order of listed Debian architectures to show the 64-bit ones first [ ].
  • Reduce the frequency that the Debian package sets and dd-list HTML pages update [ ].
  • Sort Tested suite consistently (and Debian unstable first) [ ].
  • Update the Jenkins shell monitor script to only query disk statistics every 230min [ ] and improve the documentation [ ][ ].

Other development work disorderfs version 0.5.11-3 was uploaded by Holger Levsen, fixing a number of issues with the manual page [ ][ ][ ].
Bernhard M. Wiedemann published another monthly report about reproducibility within openSUSE.
If you are interested in contributing to the Reproducible Builds project, please visit the Contribute page on our website. You can get in touch with us via:

4 March 2023

Matt Brown: Retrospective: Feb 2023

February ended up being a very short work month as I made a last minute decision to travel to Adelaide for the first 2 weeks of the month to help my brother with some house renovations he was undertaking. I thought I might be able to keep up with some work and my writing goals in the evenings while I was there, but days of hard manual labour are such an unfamiliar routine for me that I didn t have any energy left to make good on that intention. The majority of my time and focus for the remaining one and half weeks of the month was catching up on the consulting work that I had pushed back while in Adelaide. So while it doesn t make for a thrilling first month to look back and report on, overall I m not unhappy with what I achieved given the time available. Next month, I hope to be able to report some more exciting progress on the product development front as well.

Monthly Scoring Rubric I m evaluating each goal using a 10 point scale based on execution velocity and risk level, rather than absolute success (which is what I will look at in the annual/mid-year review). If velocity is good and risk is low or well managed the score is high, if either the velocity is low, or risk is high then the score is low. E.g:
  • 10 - perfect execution with low-risk, on track for significantly overachieving the goal.
  • 7 - good execution with low or well managed risk, highly likely to achieve the goal.
  • 5 - execution and risk are OK, should achieve the goal if all goes well.
  • 3 - execution or risk have problems, goal is at risk.
  • 0 - stalled, with no obvious path to recovery or success.

Goals

Consulting - 6/10 Goal: Execute a series of successful consulting engagements, building a reputation for myself and leaving happy customers willing to provide testimonials that support a pipeline of future opportunities.
  • I have one active local engagement assisting a software team with migrating their application from a single to multi-region architecture.
  • Two promising international engagements which were close to starting both cancelled based on newly issued company policies freezing their staffing/outsourcing budgets due to the current economic climate.
I m happy with where this is at - I hit 90% of my target hours in February (taking into account 2 weeks off) and the feedback I m receiving is positive. The main risk is the future pipeline of engagements, particularly if the cancellations indicate a new pattern. I m not overly concerned yet, as all the opportunities to date have been from direct or referred contacts in my personal network, so there s plenty of potential to more actively solicit work to create a healthier pipeline.

Product Development - 3/10 Goal: Grow my product development skill set by taking several ideas to MVP stage with customer feedback received, and launch at least one product which generates revenue and has growth potential.
  • Accelerating electrification - I continued to keep up with industry news and added some interesting reports to my reading queue, but made no significant progress towards identifying a specific product opportunity.
  • Farm management SaaS - no activity or progress at all.
  • co2mon.nz - I put significant thought and planning into how to approach a second iteration of this product. I started writing and completed 80% of a post to communicate the revised business plan, but it s not ready for publication yet, and even if it was, the real work towards it would need to actually happen to score more points here.
I had high hopes to make at least some progress in all three areas in February, but it just didn t happen due to lack of time. The good news is that since the low score here is purely execution driven, there s no new risks or blockers that will hinder much better progress here in March.

Professional Network Development - 8/10 Goal: To build a professional relationship with at least 30 new people this year. This is off to a strong start, I made 4 brand new connections and re-established contact with 9 other existing people I d not talked to for a while. I ve found the conversations energising and challenging and I m looking forward to continuing to keep this up.

Writing - 2/10 Goal: To publish a high-quality piece of writing on this site at least once a week. Well off track as already noted. I am enjoying the writing process and I continue to find it useful in developing my thoughts and forcing me to challenge my assumptions, but coupling the writing process with the thinking/planning that is a prerequisite to get those benefits definitely makes my output a lot slower than I was expecting. The slower speed, combined with the obvious time constraints of this month are not a great doubly whammy to be starting with, but I think with some planning and preparation it should have been avoidable by having a backlog of pre-written content for use in weeks where I m on holiday or otherwise busy. It s worth noting that among all the useful feedback I received, this writing target was often called out as overly ambitious, or likely to be counterproductive to producing quality writing. The feedback makes sense - for now I m not planning to change the goal (I might at my 6-month review point), but I am going to be diligent about adhering to my quality standard, which in turn means I m choosing to accept missing a weekly post here and there and taking a lower score on the goal overall. I apologise if you ve been eagerly waiting for writing that never arrived over February!

Community - 5/10 Goal: To support the growth of my local technical community by volunteering my experience and knowledge with others through activities such as mentoring, conference talks and similar.
  • I was an invited participant of the monthly KiwiSRE meet-up which was discussing SRE team models, and in particular I was able to speak to my experiences as described in an old CRE blog post on this topic.
  • I joined the program committee for SREcon23 APAC which is scheduled for mid-June in Singapore. I also submitted two talk proposals of my own (not sharing the details for now, since the review process is intended to be blind) which I m hopeful might make the grade with my fellow PC members!

Feedback As always, I d love to hear from you if you have thoughts or feedback triggered by anything I ve written above. In particular, it would be useful to know whether you find this type of report interesting to read and/or what you d like to see added/removed or changed.

3 March 2023

Louis-Philippe V ronneau: Goodbye Bullseye report from the Montreal 2023 BSP

Hello World! I haven't really had time to blog here since the start of the semester, as I've been pretty busy at work1. All this to say, this report for the Bug Squashing Party we held in Montreal last weekend is a little late, sorry :) First of all, I'm pleased to announce our local community seems to be doing great and has recovered from the pandemic-induced lull. May COVID stay away from our bodies forever. This time around, a total of 9 people made it to what has become somewhat of a biennial tradition2. We worked on a grand total of 14 bugs and even managed to close some! It looks like I was too concentrated on bugs to take a picture of the event... To redeem myself, I hereby offer you a picture of a cute-but-hairless cat I met on Sunday morning: Picture of a curious sphinx cat on a table You should try to join an upcoming BSP or to organise one if you can. It's loads of fun and you'll be helping the project make the next release happen sooner! As always, thanks to Debian for granting us a budget for the food and to rent the venue. Goodbye Bullseye!
  1. Which I guess is a good thing, since it means I actually have work this semester :O
  2. See our previous BSPs in 2017, 2019 and 2021.

Russell Coker: Hyper Threading on the E5-2696v3

I just did some quick tests of hyper-threading on my new E5-2696v3 CPU. I compiled the Linux 6.0.10 kernel with and without hyper-threading enabled. Here s the times for make -j36 bzImage and make -j36 modules with HT enabled: 
real    2m26.540s
user    55m25.121s
sys     9m56.443s
real    10m57.374s
user    309m21.531s
sys     58m1.070s
Here s the times for make -j18 bzImage and make -j18 modules with HT disabled: 
real    2m40.501s
user    31m35.295s
sys     5m43.523s
real    11m39.313s
user    170m46.840s
sys     31m37.756s
That s 9.6% faster for bzImage and 6.4% faster for modules. So for a performance boost that s between 5% and 10% I get greater exposure to kernel security issues and more difficulty tracking CPU time. That doesn t seem like a good trade-off so I ve put the nosmt kernel command-line option back.
Related posts:
  1. Basics of Linux Kernel Debugging Firstly a disclaimer, I m not an expert on this and...
  2. Purging an RT Database I had a problem where the email address circsales@washpost.com spammed...
  3. Video Mode and KVM I recently changed my KVM servers to use the kernel...

2 March 2023

Ian Jackson: Never use git submodules

tl;dr git submodules are always the wrong solution. Yes, even the to the problem they were specifically invented to solve. ( Read more... )

comment count unavailable comments

Ben Hutchings: Debian LTS work, January/February 2023

In January I was assigned 24 hours by Freexian's Debian LTS initiative and worked 8 hours. In February I was assigned another 8 hours and worked 8 hours. I updated the linux (4.19) package to the latest stable update, but didn't upload it. I merged the latest bullseye security update into the linux-5.10 package and uploaded that.

1 March 2023

Russ Allbery: Small book haul

I'm a bit behind on both free software maintenance and on writing reviews, what with one thing and another, but hopefully will have time to catch up next month. Meanwhile, publishing continues and books keep catching my eye. Blake Crouch (ed.) Forward (sff anthology)
Kate Elliott The Keeper's Six (sff)
Ruthanna Emrys A Half-Built Garden (sff)
R.F. Kuang Babel (sff)
Seanan McGuire The Unkindest Tide (sff)
Seanan McGuire A Killing Frost (sff)
Seanan McGuire When Sorrows Come (sff)
Seanan McGuire Be the Serpent (sff)
Terry Pratchett Thief of Time (sff)
Terry Pratchett The Last Hero (sff)
Terry Pratchett The Amazing Maurice and His Educated Rodents (sff)
Terry Pratchett Night Watch (sff)
Terry Pratchett The Wee Free Men (sff)
Terry Pratchett Monstrous Regiment (sff) I keep hearing amazing things about Babel, so it's very high on the list.

Reproducible Builds (diffoscope): diffoscope 238 released

The diffoscope maintainers are pleased to announce the release of diffoscope version 238. This version includes the following changes: 
* autopkgtest: fix tool name in the skippable list.
You find out more by visiting the project homepage.

Debian XMPP Team: XMPP What's new in Debian 12 bookworm

On Tue 13 July 2021 there was a blog post of new XMPP related software releases which have been uploaded to Debian 11 (bullseye). Today, we will inform you about updates for the upcoming Debian release bookworm. A lot of new releases have been provided by the upstream projects. There were lot of changes to the XMPP clients like Dino, Gajim, Profanity, Poezio and others. Also the XMPP servers have been enhanced. Unfortunately, we can not provide a list of all the changes which have been done, but will try to highlight some of the changes and new features. BTW, feel free to join the Debian User Support on Jabber at xmpp:debian@conference.debian.org?join. You can find a list of 58 packages of the Debian XMPP team on the XMPP QA Page. Server Libs Others Happy chatting - keep in touch with your family and friends via Jabber / XMPP - XMPP is an open standard of the Internet Engineering Task Force (IETF) for instant messaging.

28 February 2023

Paul Wise: FLOSS Activities Feb 2023

Focus This month I didn't have any particular focus. I just worked on issues in my info bubble.

Changes

Issues

Review

Administration
  • Debian BTS: unarchive/reopen/triage bugs for reintroduced package servefile
  • Debian IRC: turn an old channel into a redirect to the right one
  • Debian wiki: unblock IP addresses, approve accounts

Communication
  • Respond to queries from Debian users and contributors on the mailing lists and IRC

Sponsors The pyemd/sptag work was sponsored. All other work was done on a volunteer basis.

Shirish Agarwal: Cutting off body parts and Lenovo

I would suggest that this blog post would be slightly unpleasant and I do wish that there was a way, a standardized way just like movies where you can put General, 14+, 16+, Adult and whatnot. so people could share without getting into trouble. I would suggest to consider this blog as for somewhat mature and perhaps disturbing.

Cutting off body parts From last couple of months or so we have been getting daily reports of either men or women killed and then being chopped into pieces and this is being normalized . During my growing up years, the only such case I remember was the 1995 Tandoor case and it jolted the conscience of the nation. But it seems lot of water has passe under the bridge. as no one seems to be shocked anymore Also shocking are the number of heart attacks that young people are getting. Dunno the reason for either. Just saw this yesterday, The first thing to my mind was, at least she wasn t chopped. It was only latter I realized that the younger sister may have wanted to educate herself or have some other drreams, but because of some evil customs had to give hand in marriage. No outrage here for anything, not even child marriage :(. How have we become so insensitive. And it s mostly Hindus killing Hindus but still no outrage. We have been killing Muslims and Christians so that I guess is just par for the course :(. I wish I could say there is a solution but there seems to be not Even Child abuse cases have been going up but sad to say even they are being normalised. It s only when a US agency or somebody who feels shocked, then we feel shocked otherwise we have become numb

AMD and Lenovo Lappies About couple of months ago I had made a blog post about lappies. Then Russel reached out to me on Twitter and we engaged. One thing lead to other and soon I saw on some other topic somewhere came across this
The above is a video presentation given by Mark Pearson. Sad to say it was not illuminating enough. Especially the whole boothole thing. I did see three blog posts to get some more insight. The security entry did also share some news. I also reached out to Mr. Pearson to know both the status and also to enquire if there are any new lappies without an OS that I can buy from Lenovo. Sadly, both these e-mails went unanswered. Maybe they went to spam or something else, have no clue. While other organizations did work on it, Debian was kinda side-lined. Hence the annoyance from the Debian Maintainers that the whole thing came from the left field. And this doesn t just effect Debian but all those downstream distributions that rely on Debian . Now while it s almost a year since then and probably all has been fixed but there haven t been any instructions that I could find that tellls me if there is any new way or just the old way works. In any case, I do think bookworm release probably would have all the fixes needed. IIRC, we just entered soft freeze just couple of weeks back. I have to admit something though, I have never used secure-boot as it has been designed, partially because I always run testing, irrespective of whatever device I use. And AFAIK the whole idea of Secure Boot is to have few updates unlike Testing which is kinda a rolling release thing. While Secure Boot wants same bits, all underlying bits, in Testing it s hard to ensure that as the idea is to test new releases of software and see what works and what breaks till we send it to final release (something like Bookworm ). FWIW, currently bookworm and Testing is one and the same till Bookworm releases, and then Testing would have its own updates from the next hour/day after.

Russell Coker: Links February 2023

Vox has an insightful interview with the author of Slouching Towards Utopia: An Economic History of the Twentieth Century [1]. The main claim of that book is that The 140 years from 1870 to 2010 of the long twentieth century were, I strongly believe, the most consequential years of all humanity s centuries . A claim that seems well supported. PostMarketOS is an interesting OS for hardware designed for Android [2]. It is based on Alpine Linux, is small, and modular. If you want to change something just change that package not the entire image. Also an aim is to have as much commonality between devices as possible, all phones with the same CPU family can run the same packages apart from the kernel and maybe some utilities related to hardware. Abhijithpa blogged about getting started with pmOS, it seems easy to do [3]. Interesting article about gay samurai [4]. Regarding sex with men or women an elderly arbiter, after hearing the impassioned arguments of the two sides, counsels that the wisest course is to follow both paths in moderation, thereby helping to prevent overindulgence in either . Wow. The SCP project is an interesting collaborative SciFi/horror fiction project [5] based on an organisation that aims to Secure and Contain dangerous objects and beings and Protect the world from them. The series of stories about the Anti-Memetics Division [6] is a good place to start reading.
Related posts:
  1. Links January 2023 The Intercept has an amusing and interesting article about senior...
  2. Links February 2020 Truthout has an interesting summary of the US Wars Without...
  3. Links February 2021 Elestic Search gets a new license to deal with AWS...

Utkarsh Gupta: FOSS Activites in February 2023

Here s my (forty-first) monthly but brief update about the activities I ve done in the F/L/OSS world.

Debian
This was my 50th month of actively contributing to Debian. I became a DM in late March 2019 and a DD on Christmas 19! \o/ There s a bunch of things I do, both, technical and non-technical. Here are the things I did this month:

Uploads

Others
  • Looked up some Release team documentation.
  • Sponsored php-font-lib and php-dompdf-svg-lib for William.
  • Granted DM rights for php-dompdf.
  • Mentoring for newcomers.
  • Reviewed micro bits for Nilesh, new uploads and changes.
  • Ruby sprints.
  • Bug work (on BTS and #debian-ruby) for rails and redmine.
  • Moderation of -project mailing list.
A huge thanks to Freexian for sponsoring my Debian work and Entrouvert for sponsoring the Redmine backports. :D

Ubuntu
This was my 25th month of actively contributing to Ubuntu. Now that I joined Canonical to work on Ubuntu full-time, there s a bunch of things I do! \o/ I mostly worked on different things, I guess. I was too lazy to maintain a list of things I worked on so there s no concrete list atm. Maybe I ll get back to this section later or will start to list stuff from the fall, as I was doing before. :D

Debian (E)LTS
Debian Long Term Support (LTS) is a project to extend the lifetime of all Debian stable releases to (at least) 5 years. Debian LTS is not handled by the Debian security team, but by a separate group of volunteers and companies interested in making it a success. And Debian Extended LTS (ELTS) is its sister project, extending support to the stretch and jessie release (+2 years after LTS support). This was my forty-first month as a Debian LTS and thirty-second month as a Debian ELTS paid contributor.
I worked for 24.25 hours for LTS and 28.50 hours for ELTS.

LTS CVE Fixes and Announcements:
  • Fixed CVE-2022-47016 for tmux and uploaded to buster via 2.8-3+deb10u1.
    But decided to not roll the DLA for the package as the CVE got rejected upstream.
  • Issued DLA 3359-1, fixing CVE-2019-13038 and CVE-2021-3639, for libapache2-mod-auth-mellon.
    For Debian 10 buster, these problems have been fixed in version 0.14.2-1+deb10u1.
  • Issued DLA 3360-1, fixing CVE-2021-30151 and CVE-2022-23837, for ruby-sidekiq.
    For Debian 10 buster, these problems have been fixed in version 5.2.3+dfsg-1+deb10u1.
  • Worked on ruby-rails-html-sanitize and added notes to the security-tracker.
    TL;DR: we need newer methods in ruby-loofah to make the patches for ruby-rails-html-sanitize backportable.
  • Started to look at other set of packages meanwhile.

ELTS CVE Fixes and Announcements:
  • Issued ELA 813-1, fixing CVE-2017-12618 and CVE-2022-25147, for apr-util.
    For Debian 8 jessie, these problems have been fixed in version 1.5.4-1+deb8u1.
    For Debian 9 stretch, these problems have been fixed in version 1.5.4-3+deb9u1.
  • Issued ELA 814-1, fixing CVE-2022-39286, for jupyter-core.
    For Debian 9 stretch, these problems have been fixed in version 4.2.1-1+deb9u1.
  • Issued ELA 815-1, fixing CVE-2022-44792 and CVE-2022-44793, for net-snmp.
    For Debian 8 jessie, these problems have been fixed in version 5.7.2.1+dfsg-1+deb8u6.
    For Debian 9 stretch, these problems have been fixed in version 5.7.3+dfsg-1.7+deb9u5.
  • Helped facilitate RabbitMQ s update queries by one of our customers.
  • Started to look at other set of packages meanwhile.

Other (E)LTS Work:
Until next time.
:wq for today.

Reproducible Builds (diffoscope): diffoscope 237 released

The diffoscope maintainers are pleased to announce the release of diffoscope version 237. This version includes the following changes: 
* autopkgtest: only install aapt and dexdump on architectures where they are
  available. (Closes: #1031297)
* compartors/pdf:
  + Drop backward compatibility assignment.
  + Fix flake warnings, potentially reinstating PyPDF 1.x support (untested).
You find out more by visiting the project homepage.

27 February 2023

Daniel Lange: Thunderbird gpg key import

Thunderbird, srsly? 5MB (or 4.8MiB) import limit. Sure. My modest pubring (111 keys) is 18MB. The Debian keyring is 28MB. May be, just may be, add another 0 to that if statement? So, until that happens, workarounds ... Option 1: Export each pubkey into a separate file. The import dialog allows to select them all in one go. But - of course - it will ask confirmation for each. So prepare some valerian tea.
gpg --with-colons --list-public-keys grep ^pub cut -d : -f 5 xargs -I -n 1 gpg -ao .pub --export ;
Option 2: Strip all the signatures, so Thunderbird gets a smaller file to chew on. This uses pgp-clean from signing-party.
gpg --with-colons --list-public-keys grep ^pub cut -d : -f 5 xargs pgp-clean -s >> there_you_go_thunderbird.pub
Option 1 will retain the signatures on individual keys, Option 2 will not.

Daniel Lange: Getting gpg to import signatures again

The GnuPG (gpg) ecosystem has been played with a bit in 2019 by adding fake signatures en masse to well known keys. The main result is that the SKS Keyserver network based on the OCaml software of the same name is basically history. A few other keyservers have come up like Hagrid (Rust) and Hockeypuck (Go) but there seems to be no clear winner yet. In case you missed it in 2019, see my take on cleaning these polluted keys. Now the changed defaults in gpg to "mitigate" this issue are trickling down to even the conservative distributions. Debian Bullseye has self-sigs-only on gpg 2.2.27 and it looks like Debian Bookworm will get gpg 2.2.40. This would add import-clean but Daniel Kahn Gillmor patched it out. He argues correctly that this new default could delete data from good locally store pubkeys. This all ends in you getting some random combination of self-sigs-only and / or import-clean depending on which Linux distribution and version you happen to use. Better be explicit. I recommend to add:
# disable new gpg defaults
keyserver-options no-self-sigs-only
keyserver-options no-import-clean
to your ~/.gnupg/gpg.conf to make sure you can manage signatures yourself and receive them from keyservers or local imports as intended. In case you care: See info gnupg --index-search=keyserver-options for the fine documentation. Of course apt install info first to be able to read info pages. 'cause who still used them in 2023? Oh, wait...

Next.

Previous.