Search Results: "sergiodj"

18 May 2026

Sergio Durigan Junior: Fixing a 20+ year old bug in Debian curl

I have been helping co-maintain the Debian curl package for a few years now, and even though Samuel and Charles do most of the work, I'm happy to jump in and help when needed. This is one of those cases. Nowadays the package is maintained by 3 people (with help from others occasionally), but it hasn't always been like this. Samuel adopted the package back in 2021, and since then it has received a lot of love and care to make sure it lives up to Debian's standards. Again, kudos to both him and Charles who have been doing great work on this front. But a little more than 20 years ago, the situation in Debian (and curl!) was "a bit" different.

Once upon a time... According to d/changelog, the Debian curl maintainer in 2005 introduced changes to the packaging that allowed it to generate a version of libcurl for each TLS backend available: OpenSSL and GnuTLS. This meant that curl would have two binary library packages:
  • libcurl3-openssl and its respective -dev variant, for libcurl linked against OpenSSL; and
  • libcurl3-gnutls and its respective -dev variant, for libcurl linked against GnuTLS.
But then, around 2006/2007 or so, upstream curl decided to bump the SONAME version of libcurl from 3 to 4. At the time, they apparently did not version their library symbols like they do now, which was... less than ideal. I don't judge them: curl and a lot of other important projects have come a long way when we consider best practices to write shared libraries. Meanwhile, on Debian land, the release team was having trouble with other transitions going on at the time. For those who are not versed in Debian's vocabulary, a transition happens when a shared library gets its SONAME version bumped: when this happens, we have to make sure that all reverse dependencies of that library still build with the new version, and fix things that fail. The more reverse dependencies the library has, the harder this work gets. When upstream curl bumping the SONAME version of libcurl, the Debian curl maintainer at the time correctly renamed the binary packages from libcurl3- openssl,gnutls (and their -dev variants) to libcurl4- openssl,gnutls (and their -dev variants), which obviously triggered a transition. And a big one, because libcurl is used by several projects. Long story short, the Debian release team found themselves between a rock and a hard place. According to the late Steve Langasek at the time:
We talked a while back about the curl transition, and about how upstream's change from libcurl.so.3 to libcurl.so.4 is gratuitously painful for us in light of the large number of reverse dependencies. The libcurl transition has at this point gotten tangled with soname transitions in jasper, exiv2, kexiv2, and God only knows what else. So I'd like to revisit this question, because tracking this transition is costing the release team a lot of time that would be better spent elsewhere, and removing the need for a libcurl transition promises to reduce the complexity of the other components by an order of magnitude. On looking at the curl package, I've come to understand that the symbol versioning in place in this library is the result of a Debian-local patch. That's great news, because it suggests a solution to this quandary that doesn't require an unreasonable amount of developer time.
Yeah, it wasn't pretty. Here's what was proposed:
I am proposing the following:
  • Keep the library soname the same as it currently is upstream. Because upstream uses unversioned symbols, our package will be binary-compatible with applications built against the upstream libcurl regardless of what we do with symbol versioning, so leaving the soname alone minimizes the amount of patching to be done against upstream code here.
  • Revert the Debian symbol versioning to the libcurl3 version, and make libcurl.so.3 a symlink to libcurl.so.4. We have already established that libcurl.so.4 is still API-compatible with libcurl.so.3, in spite of the soname change upstream; reverting the symbol versioning will make it fully ABI-compatible with libcurl.so.3, and adding the symlink lets previously-built binaries find it.
  • Revert the Debian package names to the curl 7.15.5 versions. Because compatibility has been restored with libcurl3 and libcurl3-gnutls, restoring the package names provides the best upgrade path from etch to lenny; and because the symbol versions have been reverted, the libraries are not binary-compatible with the Debian packages currently named libcurl4/libcurl4-gnutls/libcurl4-openssl (in spite of being binary-compatible with upstream), so it would be wrong to keep the current names regardless.
  • Drop the SSL-less variant of the library, which was not present in curl 7.15.5; AFAICS, there is no use case where a user of curl needs to not have SSL support, so this split seems to be unnecessary overhead. Please correct me if I'm mistaken.
  • Leave the -dev package names alone otherwise, to simplify binNMUing of the reverse-dependencies (some packages have already added versioned build-deps on libcurl4.*-dev -- I have no idea why -- so reverting the names would mean more work to chase down those packages). Drop libcurl4-dev as a binary package, though, in favor of being Provided by libcurl4-gnutls-dev. Many of the packages currently build-depending on libcurl4-dev -- including some that wrongly used libcurl3-dev before -- are GPL, and these are apparently all packages where having SSL support missing in libcurl4 wasn't hurting them, so libcurl4-gnutls-dev seems to be the reasonable "default" here.
  • Schedule binNMUs for all reverse-dependencies.
Again, no judgement here: this was what needed to be done at the time, and I believe it was a good solution given the circumstances. In the end, the binary library packages got renamed again: from libcurl4- openssl,gnutls back to libcurl3- openssl,gnutls (but not their -dev variants!), but they continued shipping libcurl libraries whose SONAME version was 4. This solved the immediate problem of untangling the transitions mentioned by Steve, but introduced a technical debt that would stick with the package literally for decades. The situation at the end of 2007 was:
  • libcurl3-openssl with libcurl4-openssl-dev; and
  • libcurl3-gnutls with libcurl4-gnutls-dev.

More discrepancy is added Eventually the libcurl3-openssl package got renamed to libcurl3, but aside from that the situation with mismatched library names vs. SONAME versions stayed relatively unchanged until around 2018, when the Debian curl maintainer at the time (a different person) renamed libcurl3 to libcurl4 to fix a bug. This was the right thing to do for libcurl3, and at the time upstream curl was already properly versioning their symbols, but for some reason libcurl3-gnutls got left behind. So now we had:
  • libcurl4 with libcurl4-dev; and
  • libcurl3-gnutls with libcurl4-gnutls-dev.
In other words, we now have a discrepancy between the OpenSSL and GnuTLS variants' names. Yeah, confusing. And this is the situation right now, on May 2026, while I write this post. To make matters worse, the Debian curl package has been carrying a patch to facilitate the split of OpenSSL and GnuTLS flavours for decades now, and, for some reason I didn't bother to investigate, the patch pins the SONAME version of libcurl3-gnutls to CURL_GNUTLS_3, effectively overriding upstream's decision to version the symbols as CURL_GNUTLS_4.

A call to make things right Back in 2022, Simon McVittie filed a Debian bug to try and call our attention to the fact that we were shipping this messy set of curl packages. I had just started to get involved in the package maintenance and Samuel asked me to take a look at the bug. I noticed it was going to take more time than I had available, so I decided to put it in my TODO list (TM). Simon was generous enough to lay out a possible plan to tackle the problem, but I had a feeling that this was going to be harder than it looked. I kept postponing working on the bug, but also kept thinking about it now and then because it's an interesting thing to solve. Then, a month or so ago the Debian Brasil community got together for MiniDebConf Campinas 2026 and we decided to do a bug squashing party there. I started working on a few FTBFS bugs with GCC 16, but then got remembered about the curl bug and thought that that was the perfect time and place to start working on it, for a few reasons:
  • Samuel and Charles were also attending the conference, so I could talk to them about my plans and show them a PoC.
  • I was going to give a presentation about symbols (in pt_BR), so I could use this bug as an example of symbol versioning.
  • I wanted to have fun.

The initial plan The plan I had in mind was a variant of Simon's proposed plan:
  • I would have to adjust our GnuTLS-specific patch so that it did not override the SONAME version for libcurl-gnutls. Then,
  • For each symbol from libcurl3-gnutls I would have to:
    • Explicitly version it as curl_symbol_name@@CURL_GNUTLS_4.
    • Create an alias for the symbol (let's call it __curl_compat_symbol_name).
    • Explicitly version this alias as __curl_compat_symbol_name@CURL_GNUTLS_3.
  • Have a separate version of curl's linker script to make it possible to create a hierarchy between CURL_GNUTLS_3 and CURL_GNUTLS_4 symbols.
Note that this whole dance is needed because it is a hard requirement that programs linked against libcurl3-gnutls keep working when we ship libcurl4-gnutls, without needing to recompile them. Due to the fact that we will not really bump the SONAME of libcurl-gnutls (but instead fix the symbol versions shipped by it), we cannot expect programs to break given that they are actually using the exact same ABI as before. Unfortunately (as it is common with low level tools) the documentation for ld's versioning syntax is quite incomplete and hard to find. One of the best sources I found was this blog post. For this reason, let me quickly explain the different notations for symbol versioning used above.

curl_symbol_name@@CURL_GNUTLS_4 When we use curl_symbol_name@@CURL_GNUTLS_4 (note the @@) we are telling the linker that this should be considered the default version of curl_symbol_name. In other words, when a binary that links against libcurl-gnutls calls curl_symbol_name, the linker should use curl_symbol_name@@CURL_GNUTLS_4 to resolve the symbol. There are a few ways to specify a symbol version in C/C++:
__attribute__((__symver__("curl_symbol_name@@CURL_GNUTLS_4")))
void curl_symbol_name()
 
  /* ... */
 

/* or... */
void curl_symbol_name()
 
  /* ... */
 
__asm__(".symver curl_symbol_name, curl_symbol_name@@CURL_GNUTLS_4");

Function alias Creating an alias for a function is basically saying that a function can be called by another name. You can do that in C/C++ like:
void curl_symbol_name()
 
  /* ... */
 

void __curl_compat_symbol_name()
  __attribute__((alias("curl_symbol_name")));

__curl_compat_symbol_name@CURL_GNUTLS_3 Finally, when we use __curl_compat_symbol_name@CURL_GNUTL_3 (note the single @) we are telling the linker that this symbol exists, but it should not be used as the default symbol. In fact, this notation will basically hide the symbol and make it only available for those programs that have already been linked against it. It's a way of saying "don't offer this symbol when linking, but it's here in case a program needs it to run" (it's a bit more complicated than that, but you get the point). The reason I had to create an alias to the function before versioning the symbol with @CURL_GNUTLS_3 is because, once I've versioned the main symbol as @@CURL_GNUTLS_4, I can't create another version of it. It's also important to mention that to be able to create a version for the alias I also had to change its visibility to default. In the end, the alias ended up being defined as:
extern void __curl_compat_symbol_name()
  __attribute__((alias("curl_symbol_name"), visibility("default")));

First attempt and lessons learned For my PoC I decided to tackle a small subset of the problem. The symbols file for libcurl3-gnutls contains around 100 symbols that need to be fixed, so I chose two of them and started trying to write a patch to see if I could make things work. And after some time struggling with GCC's syntax and inspecting nm -D's output I finally got something that looked like it was going to work. The two symbols I had chosen to work with got correctly versioned (both as @@CURL_GNUTLS_4 and @CURL_GNUTLS_3), and a quick-and-dirty C program that used those symbols correctly compiled and ran with the expected symbols. I showed the results to Samuel and Charles, we got excited about what we saw, and then the conference ended.

Second attempt and some adjustments After getting back home I resumed the work on my branch and wrote an Emacs function that semi-automatically adjusted all 100+ symbols listed in the symbols file so that they all looked like:
__attribute__((__symver__("curl_symbol_name@@CURL_GNUTLS_4")))
void curl_symbol_name()
 
  /* ... */
 

extern void __curl_compat_symbol_name()
  __attribute__((alias("curl_symbol_name"), visibility("default"),
                 symver("__curl_compat_symbol_name@CURL_GNUTLS_3")));
The patch was big but mostly repetitive, and I was happy to have come up with a solution that looked clean. Until I tried to build the package, that is. I started seeing some strange errors that happened when ld was trying to link the final libcurl4-gnutls object (yes, at that point I had already renamed the binary package). This is one of the errors I was getting from ld (I got variants of this error as I was trying to fix the approach):
/usr/bin/x86_64-linux-gnu-ld.bfd: .libs/libcurl_gnutls_la-easy.o: in function  dupeasy_meta_freeentry':
./debian/build-gnutls/lib/./debian/build-gnutls/lib/easy.c:1024: multiple definition of  curl_easy_cleanup'; .libs/libcurl_gnutls_la-easy.o:./debian/build-gnutls/lib/./debian/build-gnutls/lib/easy.c:908: first defined here
/usr/bin/x86_64-linux-gnu-ld.bfd: .libs/libcurl-gnutls.so.4.8.0: version node not found for symbol curl_easy_duphandle@CURL_GNUTLS3
/usr/bin/x86_64-linux-gnu-ld.bfd: failed to set dynamic section sizes: bad value
This was strange. I did some tests with very simple versions of a shared library using the versioning mechanism I had implemented and it all worked. I could not reproduce the problem, and that's not a great feeling to have. Then, after reading a lot of documentation and blog posts throughout the internet I found something interesting. Apparently ld has a limitation when it comes to dealing with symbols versioned with @@. If there is a single symbol versioned like that in a source file (the actual term is TU, which means Translation Unit, but let's simplify), then ld is happy and generates the expected version without issues. But when we're dealing with multiple definitions of @@ symbols in a source file (which is exactly what happens in curl), then ld can get confused and start giving errors during the link stage. To solve that limitation, we have to resort to yet another symbol versioning notation: @@@. Yes, three at signs. For example:
void curl_symbol_name()
 
  /* ... */
 
__asm__(".symver curl_symbol_name, curl_symbol_name@@@CURL_GNUTLS_4");
Note that we have to use __asm__ because GCC's __attribute__ doesn't support the triple-at notation. What this does is tell the linker to create a versioned symbol for curl_symbol_name, set it as the default symbol when linking, but also remove the unversioned curl_symbol_name symbol. This makes ld happy and allows it to successfully link libcurl-gnutls. As usual, you won't find any mention of the @@@ notation inside ld's documentation. With libcurl-gnutls compiling again, I had to adjust libcurl's linker script to create a hierarchy between CURL_GNUTLS_3 and CURL_GNUTLS_4 symbols. Here's the final version of the file:
CURL_GNUTLS_3
 
  global:
    curl_easy_cleanup;
    /* lots of other symbols here */
  local: *;
 ;

CURL_GNUTLS_4
 
  global: curl_*;
  local: *;
  CURL_GNUTLS_3;

Debian package adjustments After getting the hard part out of the way, the rest was easy. It was time to finally rename libcurl3-gnutls to libcurl4-gnutls. Initially I was thinking that I'd need to ask the release team for a transition to happen, but as it turns out that won't be necessary. Because we are effectively shipping the same exact library/ABI and the only difference is the inclusion of the extra CURL_GNUTLS_4 versioned symbols, and given that we will be shipping CURL_GNUTLS_3 versioned symbols to guarantee backwards compatibility, packages won't need to get rebuild just to pick up the new dependency. Instead, we can safely turn libcurl3-gnutls into a transitional package that depends on libcurl4-gnutls.

Merge request and next steps This is the merge request where I am working on the fix. As of this writing it is in a draft state, but I expect to merge in the next couple of days. Once the fixed curl package is uploaded, we should keep an eye on the archive to make sure no unexpected bugs happen. I would like to carry this patch downstream at least until forky is released. It doesn't make sense to propose it upstream because this problem is Debian-specific and should be fixed there. We will need to make sure that all reverse dependencies of libcurl3-gnutls are recompiled before we can get rid of the transitional package, too. This was a fun bug to investigate and fix, and I am happy that we will finally have sensible names (and symbol versions!) for both of our libcurl variants. Stay tuned for the next challenge!

Sergio Durigan Junior: Fixing a 20+ year old bug in Debian curl

I have been helping co-maintain the Debian curl package for a few years now, and even though Samuel and Charles do most of the work, I'm happy to jump in and help when needed. This is one of those cases. Nowadays the package is maintained by 3 people (with help from others occasionally), but it hasn't always been like this. Samuel adopted the package back in 2021, and since then it has received a lot of love and care to make sure it lives up to Debian's standards. Again, kudos to both him and Charles who have been doing great work on this front. But a little more than 20 years ago, the situation in Debian (and curl!) was "a bit" different.

Once upon a time... According to d/changelog, the Debian curl maintainer in 2005 introduced changes to the packaging that allowed it to generate a version of libcurl for each TLS backend available: OpenSSL and GnuTLS. This meant that curl would have two binary library packages:
  • libcurl3-openssl and its respective -dev variant, for libcurl linked against OpenSSL; and
  • libcurl3-gnutls and its respective -dev variant, for libcurl linked against GnuTLS.
But then, around 2006/2007 or so, upstream curl decided to bump the SONAME version of libcurl from 3 to 4. At the time, they apparently did not version their library symbols like they do now, which was... less than ideal. I don't judge them: curl and a lot of other important projects have come a long way when we consider best practices to write shared libraries. Meanwhile, on Debian land, the release team was having trouble with other transitions going on at the time. For those who are not versed in Debian's vocabulary, a transition happens when a shared library gets its SONAME version bumped: when this happens, we have to make sure that all reverse dependencies of that library still build with the new version, and fix things that fail. The more reverse dependencies the library has, the harder this work gets. When upstream curl bumping the SONAME version of libcurl, the Debian curl maintainer at the time correctly renamed the binary packages from libcurl3- openssl,gnutls (and their -dev variants) to libcurl4- openssl,gnutls (and their -dev variants), which obviously triggered a transition. And a big one, because libcurl is used by several projects. Long story short, the Debian release team found themselves between a rock and a hard place. According to the late Steve Langasek at the time:
We talked a while back about the curl transition, and about how upstream's change from libcurl.so.3 to libcurl.so.4 is gratuitously painful for us in light of the large number of reverse dependencies. The libcurl transition has at this point gotten tangled with soname transitions in jasper, exiv2, kexiv2, and God only knows what else. So I'd like to revisit this question, because tracking this transition is costing the release team a lot of time that would be better spent elsewhere, and removing the need for a libcurl transition promises to reduce the complexity of the other components by an order of magnitude. On looking at the curl package, I've come to understand that the symbol versioning in place in this library is the result of a Debian-local patch. That's great news, because it suggests a solution to this quandary that doesn't require an unreasonable amount of developer time.
Yeah, it wasn't pretty. Here's what was proposed:
I am proposing the following:
  • Keep the library soname the same as it currently is upstream. Because upstream uses unversioned symbols, our package will be binary-compatible with applications built against the upstream libcurl regardless of what we do with symbol versioning, so leaving the soname alone minimizes the amount of patching to be done against upstream code here.
  • Revert the Debian symbol versioning to the libcurl3 version, and make libcurl.so.3 a symlink to libcurl.so.4. We have already established that libcurl.so.4 is still API-compatible with libcurl.so.3, in spite of the soname change upstream; reverting the symbol versioning will make it fully ABI-compatible with libcurl.so.3, and adding the symlink lets previously-built binaries find it.
  • Revert the Debian package names to the curl 7.15.5 versions. Because compatibility has been restored with libcurl3 and libcurl3-gnutls, restoring the package names provides the best upgrade path from etch to lenny; and because the symbol versions have been reverted, the libraries are not binary-compatible with the Debian packages currently named libcurl4/libcurl4-gnutls/libcurl4-openssl (in spite of being binary-compatible with upstream), so it would be wrong to keep the current names regardless.
  • Drop the SSL-less variant of the library, which was not present in curl 7.15.5; AFAICS, there is no use case where a user of curl needs to not have SSL support, so this split seems to be unnecessary overhead. Please correct me if I'm mistaken.
  • Leave the -dev package names alone otherwise, to simplify binNMUing of the reverse-dependencies (some packages have already added versioned build-deps on libcurl4.*-dev -- I have no idea why -- so reverting the names would mean more work to chase down those packages). Drop libcurl4-dev as a binary package, though, in favor of being Provided by libcurl4-gnutls-dev. Many of the packages currently build-depending on libcurl4-dev -- including some that wrongly used libcurl3-dev before -- are GPL, and these are apparently all packages where having SSL support missing in libcurl4 wasn't hurting them, so libcurl4-gnutls-dev seems to be the reasonable "default" here.
  • Schedule binNMUs for all reverse-dependencies.
Again, no judgement here: this was what needed to be done at the time, and I believe it was a good solution given the circumstances. In the end, the binary library packages got renamed again: from libcurl4- openssl,gnutls back to libcurl3- openssl,gnutls (but not their -dev variants!), but they continued shipping libcurl libraries whose SONAME version was 4. This solved the immediate problem of untangling the transitions mentioned by Steve, but introduced a technical debt that would stick with the package literally for decades. The situation at the end of 2007 was:
  • libcurl3-openssl with libcurl4-openssl-dev; and
  • libcurl3-gnutls with libcurl4-gnutls-dev.

More discrepancy is added Eventually the libcurl3-openssl package got renamed to libcurl3, but aside from that the situation with mismatched library names vs. SONAME versions stayed relatively unchanged until around 2018, when the Debian curl maintainer at the time (a different person) renamed libcurl3 to libcurl4 to fix a bug. This was the right thing to do for libcurl3, and at the time upstream curl was already properly versioning their symbols, but for some reason libcurl3-gnutls got left behind. So now we had:
  • libcurl4 with libcurl4-dev; and
  • libcurl3-gnutls with libcurl4-gnutls-dev.
In other words, we now have a discrepancy between the OpenSSL and GnuTLS variants' names. Yeah, confusing. And this is the situation right now, on May 2026, while I write this post. To make matters worse, the Debian curl package has been carrying a patch to facilitate the split of OpenSSL and GnuTLS flavours for decades now, and, for some reason I didn't bother to investigate, the patch pins the SONAME version of libcurl3-gnutls to CURL_GNUTLS_3, effectively overriding upstream's decision to version the symbols as CURL_GNUTLS_4.

A call to make things right Back in 2022, Simon McVittie filed a Debian bug to try and call our attention to the fact that we were shipping this messy set of curl packages. I had just started to get involved in the package maintenance and Samuel asked me to take a look at the bug. I noticed it was going to take more time than I had available, so I decided to put it in my TODO list (TM). Simon was generous enough to lay out a possible plan to tackle the problem, but I had a feeling that this was going to be harder than it looked. I kept postponing working on the bug, but also kept thinking about it now and then because it's an interesting thing to solve. Then, a month or so ago the Debian Brasil community got together for MiniDebConf Campinas 2026 and we decided to do a bug squashing party there. I started working on a few FTBFS bugs with GCC 16, but then got remembered about the curl bug and thought that that was the perfect time and place to start working on it, for a few reasons:
  • Samuel and Charles were also attending the conference, so I could talk to them about my plans and show them a PoC.
  • I was going to give a presentation about symbols (in pt_BR), so I could use this bug as an example of symbol versioning.
  • I wanted to have fun.

The initial plan The plan I had in mind was a variant of Simon's proposed plan:
  • I would have to adjust our GnuTLS-specific patch so that it did not override the SONAME version for libcurl-gnutls. Then,
  • For each symbol from libcurl3-gnutls I would have to:
    • Explicitly version it as curl_symbol_name@@CURL_GNUTLS_4.
    • Create an alias for the symbol (let's call it __curl_compat_symbol_name).
    • Explicitly version this alias as __curl_compat_symbol_name@CURL_GNUTLS_3.
  • Have a separate version of curl's linker script to make it possible to create a hierarchy between CURL_GNUTLS_3 and CURL_GNUTLS_4 symbols.
Note that this whole dance is needed because it is a hard requirement that programs linked against libcurl3-gnutls keep working when we ship libcurl4-gnutls, without needing to recompile them. Due to the fact that we will not really bump the SONAME of libcurl-gnutls (but instead fix the symbol versions shipped by it), we cannot expect programs to break given that they are actually using the exact same ABI as before. Unfortunately (as it is common with low level tools) the documentation for ld's versioning syntax is quite incomplete and hard to find. One of the best sources I found was this blog post. For this reason, let me quickly explain the different notations for symbol versioning used above.

curl_symbol_name@@CURL_GNUTLS_4 When we use curl_symbol_name@@CURL_GNUTLS_4 (note the @@) we are telling the linker that this should be considered the default version of curl_symbol_name. In other words, when a binary that links against libcurl-gnutls calls curl_symbol_name, the linker should use curl_symbol_name@@CURL_GNUTLS_4 to resolve the symbol. There are a few ways to specify a symbol version in C/C++:
__attribute__((__symver__("curl_symbol_name@@CURL_GNUTLS_4")))
void curl_symbol_name()
 
  /* ... */
 

/* or... */
void curl_symbol_name()
 
  /* ... */
 
__asm__(".symver curl_symbol_name, curl_symbol_name@@CURL_GNUTLS_4");

Function alias Creating an alias for a function is basically saying that a function can be called by another name. You can do that in C/C++ like:
void curl_symbol_name()
 
  /* ... */
 

void __curl_compat_symbol_name()
  __attribute__((alias("curl_symbol_name")));

__curl_compat_symbol_name@CURL_GNUTLS_3 Finally, when we use __curl_compat_symbol_name@CURL_GNUTL_3 (note the single @) we are telling the linker that this symbol exists, but it should not be used as the default symbol. In fact, this notation will basically hide the symbol and make it only available for those programs that have already been linked against it. It's a way of saying "don't offer this symbol when linking, but it's here in case a program needs it to run" (it's a bit more complicated than that, but you get the point). The reason I had to create an alias to the function before versioning the symbol with @CURL_GNUTLS_3 is because, once I've versioned the main symbol as @@CURL_GNUTLS_4, I can't create another version of it. It's also important to mention that to be able to create a version for the alias I also had to change its visibility to default. In the end, the alias ended up being defined as:
extern void __curl_compat_symbol_name()
  __attribute__((alias("curl_symbol_name"), visibility("default")));

First attempt and lessons learned For my PoC I decided to tackle a small subset of the problem. The symbols file for libcurl3-gnutls contains around 100 symbols that need to be fixed, so I chose two of them and started trying to write a patch to see if I could make things work. And after some time struggling with GCC's syntax and inspecting nm -D's output I finally got something that looked like it was going to work. The two symbols I had chosen to work with got correctly versioned (both as @@CURL_GNUTLS_4 and @CURL_GNUTLS_3), and a quick-and-dirty C program that used those symbols correctly compiled and ran with the expected symbols. I showed the results to Samuel and Charles, we got excited about what we saw, and then the conference ended.

Second attempt and some adjustments After getting back home I resumed the work on my branch and wrote an Emacs function that semi-automatically adjusted all 100+ symbols listed in the symbols file so that they all looked like:
__attribute__((__symver__("curl_symbol_name@@CURL_GNUTLS_4")))
void curl_symbol_name()
 
  /* ... */
 

extern void __curl_compat_symbol_name()
  __attribute__((alias("curl_symbol_name"), visibility("default"),
                 symver("__curl_compat_symbol_name@CURL_GNUTLS_3")));
The patch was big but mostly repetitive, and I was happy to have come up with a solution that looked clean. Until I tried to build the package, that is. I started seeing some strange errors that happened when ld was trying to link the final libcurl4-gnutls object (yes, at that point I had already renamed the binary package). This is one of the errors I was getting from ld (I got variants of this error as I was trying to fix the approach):
/usr/bin/x86_64-linux-gnu-ld.bfd: .libs/libcurl_gnutls_la-easy.o: in function  dupeasy_meta_freeentry':
./debian/build-gnutls/lib/./debian/build-gnutls/lib/easy.c:1024: multiple definition of  curl_easy_cleanup'; .libs/libcurl_gnutls_la-easy.o:./debian/build-gnutls/lib/./debian/build-gnutls/lib/easy.c:908: first defined here
/usr/bin/x86_64-linux-gnu-ld.bfd: .libs/libcurl-gnutls.so.4.8.0: version node not found for symbol curl_easy_duphandle@CURL_GNUTLS3
/usr/bin/x86_64-linux-gnu-ld.bfd: failed to set dynamic section sizes: bad value
This was strange. I did some tests with very simple versions of a shared library using the versioning mechanism I had implemented and it all worked. I could not reproduce the problem, and that's not a great feeling to have. Then, after reading a lot of documentation and blog posts throughout the internet I found something interesting. Apparently ld has a limitation when it comes to dealing with symbols versioned with @@. If there is a single symbol versioned like that in a source file (the actual term is TU, which means Translation Unit, but let's simplify), then ld is happy and generates the expected version without issues. But when we're dealing with multiple definitions of @@ symbols in a source file (which is exactly what happens in curl), then ld can get confused and start giving errors during the link stage. To solve that limitation, we have to resort to yet another symbol versioning notation: @@@. Yes, three at signs. For example:
void curl_symbol_name()
 
  /* ... */
 
__asm__(".symver curl_symbol_name, curl_symbol_name@@@CURL_GNUTLS_4");
Note that we have to use __asm__ because GCC's __attribute__ doesn't support the triple-at notation. What this does is tell the linker to create a versioned symbol for curl_symbol_name, set it as the default symbol when linking, but also remove the unversioned curl_symbol_name symbol. This makes ld happy and allows it to successfully link libcurl-gnutls. As usual, you won't find any mention of the @@@ notation inside ld's documentation. With libcurl-gnutls compiling again, I had to adjust libcurl's linker script to create a hierarchy between CURL_GNUTLS_3 and CURL_GNUTLS_4 symbols. Here's the final version of the file:
CURL_GNUTLS_3
 
  global:
    curl_easy_cleanup;
    /* lots of other symbols here */
  local: *;
 ;

CURL_GNUTLS_4
 
  global: curl_*;
  local: *;
  CURL_GNUTLS_3;

Debian package adjustments After getting the hard part out of the way, the rest was easy. It was time to finally rename libcurl3-gnutls to libcurl4-gnutls. Initially I was thinking that I'd need to ask the release team for a transition to happen, but as it turns out that won't be necessary. Because we are effectively shipping the same exact library/ABI and the only difference is the inclusion of the extra CURL_GNUTLS_4 versioned symbols, and given that we will be shipping CURL_GNUTLS_3 versioned symbols to guarantee backwards compatibility, packages won't need to get rebuild just to pick up the new dependency. Instead, we can safely turn libcurl3-gnutls into a transitional package that depends on libcurl4-gnutls.

Merge request and next steps This is the merge request where I am working on the fix. As of this writing it is in a draft state, but I expect to merge in the next couple of days. Once the fixed curl package is uploaded, we should keep an eye on the archive to make sure no unexpected bugs happen. I would like to carry this patch downstream at least until forky is released. It doesn't make sense to propose it upstream because this problem is Debian-specific and should be fixed there. We will need to make sure that all reverse dependencies of libcurl3-gnutls are recompiled before we can get rid of the transitional package, too. This was a fun bug to investigate and fix, and I am happy that we will finally have sensible names (and symbol versions!) for both of our libcurl variants. Stay tuned for the next challenge!

27 March 2026

Samuel Henrique: I use curl with ECH btw (in Debian)

tl;dr This is an experimental feature that, for the first time, brings full ECH support to curl on Debian using OpenSSL. Starting with curl 8.19.0-3+exp2 (Debian Experimental), you can now use ECH, with HTTPS-RR and DoH for maximum privacy. curl 8.19.0-3+exp2 is quite fresh at the time of writing, bear in mind that your repository might not have synced the package yet, all mirrors should have it by March 27th 14:00 UTC.
# defo.ie is a test server that confirms whether ECH was successfully used
curl -v --ech hard https://defo.ie/ech-check.php
# For Encrypted Client Hello (ECH) + DNS over HTTPS (DoH)
curl -v --ech hard --doh-url https://1.1.1.1/dns-query https://defo.ie/ech-check.php
"--ech hard" tells curl to refuse the connection entirely if ECH cannot be negotiated. Or, if you would like to try it out in a container:
podman run debian:experimental /bin/bash -c 'apt install --update -t experimental -y curl && curl -v --ech hard --doh-url https://1.1.1.1/dns-query https://defo.ie/ech-check.php'
(in case you haven't noticed, apt now has the --update option for the upgrade and install commands)

For Privacy CloudFlare calls it "the last puzzle piece to privacy" in their must-read announcement: https://blog.cloudflare.com/announcing-encrypted-client-hello/. Encrypted Client Hello (rfc9849) encrypts the "which website are you connecting to?" part of the TLS handshake that was previously visible in plaintext. HTTPS-RR (rfc9460) is a DNS record type that publishes connection parameters for a service, including the public key clients need to perform ECH. DNS Over HTTPS (rfc8484) encrypts DNS queries by tunneling them over HTTPS, hiding what domains you're looking up from network observers. When all three operate together over a CDN with shared IP space, the target domain name is hidden from passive observers; the HTTPS-RR record is queried over DoH in order to retrieve the ECH key (rfc9848) for the TLS handshake. Seems like quite an important feature, and in fact the major browsers have it enabled for some time now, the trick is that they do not use OpenSSL (Chrome uses BoringSSL and Firefox uses NSS). For everyone else, the only option is to patch OpenSSL or wait until 4.0.0 is released, and so part of the reason Debian is the first distro to enable it (curl + OpenSSL + ECH) is that the OpenSSL maintainer (Sebastian Andrzej Siewior) packaged the alpha release just 3 days after it was published. Do not forget that ECH support is experimental and currently relies on the alpha release of OpenSSL.

wcurl Gets It Too Considering wcurl is just a wrapper on curl, it gets the feature for free:
wcurl --curl-options="--ech hard --doh-url https://1.1.1.1/dns-query" $URL
If you're using wcurl, you don't want to have to set parameters, this is just to show that the feature is there and if you have a .curlrc file, it can enable the feature seamlessly.

Other Debian Releases Given the ECH feature requires OpenSSL >= 4, it will not make it to Debian 13, having a small chance of going to Debian 13 Backports (emphasis on small). It should get to Debian Unstable and Debian Testing within the next couple of months as the OpenSSL GA release happens and gets packaged, but you should be able to install the package from Experimental in your Unstable and Testing systems without issues. It will also be in Debian 14 once it becomes the new Stable.

Shoulders of Giants Stephen Farrell's presentation from OpenSSL Conference 2025 has a lot of background on the work involved: Encrypted Client Hello Lessons learned from trying to do something that was probably too complicated They have been working on implementing ECH in open-source projects for years, something as big as this doesn't happen without lots of people dedicating both their paid and free times over it. I ended up being the person who enabled it on Debian, which was pretty much the least amount of work between everyone involved, but hey it's fun flipping the switch and telling you about it.

Background Since 2025, the curl developers started organizing an yearly meeting with all maintainers of curl in Operating Systems. The 2026 edition happened in March 26th: https://github.com/curl/curl/wiki/curl-distro-discussion-2026. Attendance was really good, and as you can imagine one of the topics of discussion was ECH, in which it was pointed out that having OpenSSL 4 was the main requirement but besides it nothing unusual was needed. In Debian Experimental, we have been enabling HTTPS-RR since March 2025, and OpenSSL 4.0.0 alpha was packaged just recently (2026-03-13) by Sebastian Andrzej Siewior, it's time for the next step. The curl distro meeting was just the motivation I needed to go ahead and enable it in Debian Experimental, so as part of our Debian Brasil Weekly Meetings I've prepared and uploaded the changes, while Carlos Henrique Lima Melara worked on addressing a recent test regression for Debian Unstable. Unfortunately sergiodj couldn't join and I'm sure he's jealous of the hacking session now.

Appendix While writing this, I've noticed one of the authors of the CloudFlare blogpost is the previous curl maintainer on Debian; Alessandro Ghedini let me take over the maintenance back in 2021 and today curl is maintained by a team of 4 people, it's nice to see Alessandro's involvement.

10 October 2025

Louis-Philippe V ronneau: Montreal's Debian & Stuff - September 2025

Our Debian User Group met on September 27th for our first meeting since our summer hiatus. As always, it was fun and productive! Here's what we did: pollo: sergiodj: LeLutin: tvaz: viashimo: tassia: Pictures This time again, we were hosted at La Balise (formely ATS ). It's nice to see this community project continuing to improve: the social housing apartments on the top floors should be opening this month! Lots of construction work was also ongoing to make the Espace des Possibles more accessible from the street level. Group photo Some of us ended up grabbing a drink after the event at l'Isle de Garde, a pub right next to the venue, but I didn't take any pictures.

18 June 2025

Sergio Durigan Junior: GCC, glibc, stack unwinding and relocations A war story

I ve been meaning to write a post about this bug for a while, so here it is (before I forget the details!). First, I d like to thank a few people: I ll probably forget some details because it s been more than a week (and life at $DAYJOB moves fast), but we ll see.

The background story Wolfi OS takes security seriously, and one of the things we have is a package which sets the hardening compiler flags for C/C++ according to the best practices recommended by OpenSSF. At the time of this writing, these flags are (in GCC s spec file parlance):
*self_spec:
+ % !O:% !O1:% !O2:% !O3:% !O0:% !Os:% !0fast:% !0g:% !0z:-O2  -fhardened -Wno-error=hardened -Wno-hardened % !fdelete-null-pointer-checks:-fno-delete-null-pointer-checks  -fno-strict-overflow -fno-strict-aliasing % !fomit-frame-pointer:-fno-omit-frame-pointer  -mno-omit-leaf-frame-pointer
*link:
+ --as-needed -O1 --sort-common -z noexecstack -z relro -z now
The important part for our bug is the usage of -z now and -fno-strict-aliasing. As I was saying, these flags are set for almost every build, but sometimes things don t work as they should and we need to disable them. Unfortunately, one of these problematic cases has been glibc. There was an attempt to enable hardening while building glibc, but that introduced a strange breakage to several of our packages and had to be reverted. Things stayed pretty much the same until a few weeks ago, when I started working on one of my roadmap items: figure out why hardening glibc wasn t working, and get it to work as much as possible.

Reproducing the bug I started off by trying to reproduce the problem. It s important to mention this because I often see young engineers forgetting to check if the problem is even valid anymore. I don t blame them; the anxiety to get the bug fixed can be really blinding. Fortunately, I already had one simple test to trigger the failure. All I had to do was install the py3-matplotlib package and then invoke:
$ python3 -c 'import matplotlib'
This would result in an abortion with a coredump. I followed the steps above, and readily saw the problem manifesting again. OK, first step is done; I wasn t getting out easily from this one.

Initial debug The next step is to actually try to debug the failure. In an ideal world you get lucky and are able to spot what s wrong after just a few minutes. Or even better: you also can devise a patch to fix the bug and contribute it to upstream. I installed GDB, and then ran the py3-matplotlib command inside it. When the abortion happened, I issued a backtrace command inside GDB to see where exactly things had gone wrong. I got a stack trace similar to the following:
#0  0x00007c43afe9972c in __pthread_kill_implementation () from /lib/libc.so.6
#1  0x00007c43afe3d8be in raise () from /lib/libc.so.6
#2  0x00007c43afe2531f in abort () from /lib/libc.so.6
#3  0x00007c43af84f79d in uw_init_context_1[cold] () from /usr/lib/libgcc_s.so.1
#4  0x00007c43af86d4d8 in _Unwind_RaiseException () from /usr/lib/libgcc_s.so.1
#5  0x00007c43acac9014 in __cxxabiv1::__cxa_throw (obj=0x5b7d7f52fab0, tinfo=0x7c429b6fd218 <typeinfo for pybind11::attribute_error>, dest=0x7c429b5f7f70 <pybind11::reference_cast_error::~reference_cast_error() [clone .lto_priv.0]>)
    at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:93
#6  0x00007c429b5ec3a7 in ft2font__getattr__(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) [clone .lto_priv.0] [clone .cold] () from /usr/lib/python3.13/site-packages/matplotlib/ft2font.cpython-313-x86_64-linux-gnu.so
#7  0x00007c429b62f086 in pybind11::cpp_function::initialize<pybind11::object (*&)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >), pybind11::object, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, pybind11::name, pybind11::scope, pybind11::sibling>(pybind11::object (*&)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >), pybind11::object (*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >), pybind11::name const&, pybind11::scope const&, pybind11::sibling const&):: lambda(pybind11::detail::function_call&)#1 ::_FUN(pybind11::detail::function_call&) [clone .lto_priv.0] ()
   from /usr/lib/python3.13/site-packages/matplotlib/ft2font.cpython-313-x86_64-linux-gnu.so
#8  0x00007c429b603886 in pybind11::cpp_function::dispatcher(_object*, _object*, _object*) () from /usr/lib/python3.13/site-packages/matplotlib/ft2font.cpython-313-x86_64-linux-gnu.so
...
Huh. Initially this didn t provide me with much information. There was something strange seeing the abort function being called right after _Unwind_RaiseException, but at the time I didn t pay much attention to it. OK, time to expand our horizons a little. Remember when I said that several of our packages would crash with a hardened glibc? I decided to look for another problematic package so that I could make it crash and get its stack trace. My thinking here is that maybe if I can compare both traces, something will come up. I happened to find an old discussion where Dann Frazier mentioned that Emacs was also crashing for him. He and I share the Emacs passion, and I totally agreed with him when he said that Emacs crashing is priority -1! (I m paraphrasing). I installed Emacs, ran it, and voil : the crash happened again. OK, that was good. When I ran Emacs inside GDB and asked for a backtrace, here s what I got:
#0  0x00007eede329972c in __pthread_kill_implementation () from /lib/libc.so.6
#1  0x00007eede323d8be in raise () from /lib/libc.so.6
#2  0x00007eede322531f in abort () from /lib/libc.so.6
#3  0x00007eede262879d in uw_init_context_1[cold] () from /usr/lib/libgcc_s.so.1
#4  0x00007eede2646e7c in _Unwind_Backtrace () from /usr/lib/libgcc_s.so.1
#5  0x00007eede3327b11 in backtrace () from /lib/libc.so.6
#6  0x000059535963a8a1 in emacs_backtrace ()
#7  0x000059535956499a in main ()
Ah, this backtrace is much simpler to follow. Nice. Hmmm. Now the crash is happening inside _Unwind_Backtrace. A pattern emerges! This must have something to do with stack unwinding (or so I thought keep reading to discover the whole truth). You see, the backtrace function (yes, it s a function) and C++ s exception handling mechanism use similar techniques to do their jobs, and it pretty much boils down to unwinding frames from the stack. I looked into Emacs source code, specifically the emacs_backtrace function, but could not find anything strange over there. This bug was probably not going to be an easy fix

The quest for a minimal reproducer Being able to easily reproduce the bug is awesome and really helps with debugging, but even better is being able to have a minimal reproducer for the problem. You see, py3-matplotlib is a huge package and pulls in a bunch of extra dependencies, so it s not easy to ask other people to just install this big package plus these other dependencies, and then run this command , especially if we have to file an upstream bug and talk to people who may not even run the distribution we re using. So I set up to try and come up with a smaller recipe to reproduce the issue, ideally something that s not tied to a specific package from the distribution. Having all the information gathered from the initial debug session, especially the Emacs backtrace, I thought that I could write a very simple program that just invoked the backtrace function from glibc in order to trigger the code path that leads to _Unwind_Backtrace. Here s what I wrote:
#include <execinfo.h>

int
main(int argc, char *argv[])
 
  void *a[4096];
  backtrace (a, 100);
  return 0;
 
After compiling it, I determined that yes, the problem did happen with this small program as well. There was only a small nuisance: the manifestation of the bug was not deterministic, so I had to execute the program a few times until it crashed. But that s much better than what I had before, and a small price to pay. Having a minimal reproducer pretty much allows us to switch our focus to what really matters. I wouldn t need to dive into Emacs or Python s source code anymore. At the time, I was sure this was a glibc bug. But then something else happened.

GCC 15 I had to stop my investigation efforts because something more important came up: it was time to upload GCC 15 to Wolfi. I spent a couple of weeks working on this (it involved rebuilding the whole archive, filing hundreds of FTBFS bugs, patching some programs, etc.), and by the end of it the transition went smooth. When the GCC 15 upload was finally done, I switched my focus back to the glibc hardening problem. The first thing I did was to yes, reproduce the bug again. It had been a few weeks since I had touched the package, after all. So I built a hardened glibc with the latest GCC and the bug did not happen anymore! Fortunately, the very first thing I thought was this must be GCC , so I rebuilt the hardened glibc with GCC 14, and the bug was there again. Huh, unexpected but very interesting.

Diving into glibc and libgcc At this point, I was ready to start some serious debugging. And then I got a message on Signal. It was one of those moments where two minds think alike: Gabriel decided to check how I was doing, and I was thinking about him because this involved glibc, and Gabriel contributed to the project for many years. I explained what I was doing, and he promptly offered to help. Yes, there are more people who love low level debugging! We spent several hours going through disassembles of certain functions (because we didn t have any debug information in the beginning), trying to make sense of what we were seeing. There was some heavy GDB involved; unfortunately I completely lost the session s history because it was done inside a container running inside an ephemeral VM. But we learned a lot. For example:
  • It was hard to actually understand the full stack trace leading to uw_init_context_1[cold]. _Unwind_Backtrace obviously didn t call it (it called uw_init_context_1, but what was that [cold] doing?). We had to investigate the disassemble of uw_init_context_1 in order to determined where uw_init_context_1[cold] was being called.
  • The [cold] suffix is a GCC function attribute that can be used to tell the compiler that the function is unlikely to be reached. When I read that, my mind immediately jumped to this must be an assertion , so I went to the source code and found the spot.
  • We were able to determine that the return code of uw_frame_state_for was 5, which means _URC_END_OF_STACK. That s why the assertion was triggering.
After finding these facts without debug information, I decided to bite the bullet and recompiled GCC 14 with -O0 -g3, so that we could debug what uw_frame_state_for was doing. After banging our heads a bit more, we found that fde is NULL at this excerpt:
// ...
  fde = _Unwind_Find_FDE (context->ra + _Unwind_IsSignalFrame (context) - 1,
                          &context->bases);
  if (fde == NULL)
     
#ifdef MD_FALLBACK_FRAME_STATE_FOR
      /* Couldn't find frame unwind info for this function.  Try a
         target-specific fallback mechanism.  This will necessarily
         not provide a personality routine or LSDA.  */
      return MD_FALLBACK_FRAME_STATE_FOR (context, fs);
#else
      return _URC_END_OF_STACK;
#endif
     
// ...
We re debugging on amd64, which means that MD_FALLBACK_FRAME_STATE_FOR is defined and therefore is called. But that s not really important for our case here, because we had established before that _Unwind_Find_FDE would never return NULL when using a non-hardened glibc (or a glibc compiled with GCC 15). So we decided to look into what _Unwind_Find_FDE did. The function is complex because it deals with .eh_frame , but we were able to pinpoint the exact location where find_fde_tail (one of the functions called by _Unwind_Find_FDE) is returning NULL:
if (pc < table[0].initial_loc + data_base)
  return NULL;
We looked at the addresses of pc and table[0].initial_loc + data_base, and found that the former fell within libgcc s text section, which the latter fell within /lib/ld-linux-x86-64.so.2 text. At this point, we were already too tired to continue. I decided to keep looking at the problem later and see if I could get any further.

Bisecting GCC The next day, I woke up determined to find what changed in GCC 15 that caused the bug to disappear. Unless you know GCC s internals like they are your own home (which I definitely don t), the best way to do that is to git bisect the commits between GCC 14 and 15. I spent a few days running the bisect. It took me more time than I d have liked to find the right range of commits to pass git bisect (because of how branches and tags are done in GCC s repository), and I also had to write some helper scripts that:
  • Modified the gcc.yaml package definition to make it build with the commit being bisected.
  • Built glibc using the GCC that was just built.
  • Ran tests inside a docker container (with the recently built glibc installed) to determine whether the bug was present.
At the end, I had a commit to point to:
commit 99b1daae18c095d6c94d32efb77442838e11cbfb
Author: Richard Biener <rguenther@suse.de>
Date:   Fri May 3 14:04:41 2024 +0200
    tree-optimization/114589 - remove profile based sink heuristics
Makes sense, right?! No? Well, it didn t for me either. Even after reading what was changed in the code and the upstream bug fixed by the commit, I was still clueless as to why this change fixed the problem (I say fixed because it may very well be an unintended consequence of the change, and some other problem might have been introduced).

Upstream takes over After obtaining the commit that possibly fixed the bug, while talking to Dann and explaining what I did, he suggested that I should file an upstream bug and check with them. Great idea, of course. I filed the following upstream bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=120653 It s a bit long, very dense and complex, but ultimately upstream was able to find the real problem and have a patch accepted in just two days. Nothing like knowing the code base. The initial bug became: https://sourceware.org/bugzilla/show_bug.cgi?id=33088 In the end, the problem was indeed in how the linker defines __ehdr_start, which, according to the code (from elf/dl-support.c):
if (_dl_phdr == NULL)
   
    /* Starting from binutils-2.23, the linker will define the
       magic symbol __ehdr_start to point to our own ELF header
       if it is visible in a segment that also includes the phdrs.
       So we can set up _dl_phdr and _dl_phnum even without any
       information from auxv.  */


    extern const ElfW(Ehdr) __ehdr_start attribute_hidden;
    assert (__ehdr_start.e_phentsize == sizeof *GL(dl_phdr));
    _dl_phdr = (const void *) &__ehdr_start + __ehdr_start.e_phoff;
    _dl_phnum = __ehdr_start.e_phnum;
   
But the following definition is the problematic one (from elf/rtld.c):
extern const ElfW(Ehdr) __ehdr_start attribute_hidden;
This symbol (along with its counterpart, __ehdr_end) was being run-time relocated when it shouldn t be. The fix that was pushed added optimization barriers to prevent the compiler from doing the relocations. I don t claim to fully understand what was done here, and Jakub s analysis is a thing to behold, but in the end I was able to confirm that the patch fixed the bug. And in the end, it was indeed a glibc bug.

Conclusion This was an awesome bug to investigate. It s one of those that deserve a blog post, even though some of the final details of the fix flew over my head. I d like to start blogging more about these sort of bugs, because I ve encountered my fair share of them throughout my career. And it was great being able to do some debugging with another person, exchange ideas, learn things together, and ultimately share that deep satisfaction when we find why a crash is happening. I have at least one more bug in my TODO list to write about (another one with glibc, but this time I was able to get to the end of it and come up with a patch). Stay tunned. P.S.: After having published the post I realized that I forgot to explain why the -z now and -fno-strict-aliasing flags were important. -z now is the flag that I determined to be the root cause of the breakage. If I compiled glibc with every hardening flag except -z now, everything worked. So initially I thought that the problem had to do with how ld.so was resolving symbols at runtime. As it turns out, this ended up being more a symptom than the real cause of the bug. As for -fno-strict-aliasing, a Gentoo developer who commented on the GCC bug above mentioned that this OpenSSF bug had a good point against using this flag for hardening. I still have to do a deep dive on what was discussed in the issue, but this is certainly something to take into consideration. There s this very good write-up about strict aliasing in general if you re interested in understanding it better.

29 May 2025

Arthur Diniz: Bringing Kubernetes Back to Debian

I ve been part of the Debian Project since 2019, when I attended DebConf held in Curitiba, Brazil. That event sparked my interest in the community, packaging, and how Debian works as a distribution. In the early years of my involvement, I contributed to various teams such as the Python, Golang and Cloud teams, packaging dependencies and maintaining various tools. However, I soon felt the need to focus on packaging software I truly enjoyed, tools I was passionate about using and maintaining. That s when I turned my attention to Kubernetes within Debian.

A Broken Ecosystem The Kubernetes packaging situation in Debian had been problematic for some time. Given its large codebase and complex dependency tree, the initial packaging approach involved vendorizing all dependencies. While this allowed a somewhat functional package to be published, it introduced several long-term issues, especially security concerns. Vendorized packages bundle third-party dependencies directly into the source tarball. When vulnerabilities arise in those dependencies, it becomes difficult for Debian s security team to patch and rebuild affected packages system-wide. This approach broke Debian s best practices, and it eventually led to the abandonment of the Kubernetes source package, which had stalled at version 1.20.5. Due to this abandonment, critical bugs emerged and the package was removed from Debian s testing channel, as we can see in the package tracker.

New Debian Kubernetes Team Around this time, I became a Debian Maintainer (DM), with permissions to upload certain packages. I saw an opportunity to both contribute more deeply to Debian and to fix Kubernetes packaging. In early 2024, just before DebConf Busan in South Korea, I founded the Debian Kubernetes Team. The mission of the team was to repackage Kubernetes in a maintainable, security-conscious, and Debian-compliant way. At DebConf, I shared our progress with the broader community and received great feedback and more visibility, along with people interested in contributing to the team. Our first tasks was to migrate existing Kubernetes-related tools such as kubectx, kubernetes-split-yaml and kubetail into a dedicated namespace on Salsa, Debian s GitLab instance. Many of these tools were stored across different teams (like the Go team), and consolidating them helped us organize development and focus our efforts.

De-vendorizing Kubernetes Our main goal was to un-vendorize Kubernetes and bring it up-to-date with upstream releases. This meant:
  • Removing the vendor directory and all embedded third-party code.
  • Trimming the build scope to focus solely on building kubectl, Kubernetes CLI.
  • Using Files-Excluded in debian/copyright to cleanly drop unneeded files during source imports.
  • Rebuilding the dependency tree, ensuring all Go modules were separately packaged in Debian.
We used uscan, a standard Debian packaging tool that fetches upstream tarballs and prepares them accordingly. The Files-Excluded directive in our debian/copyright file instructed uscan to automatically remove unnecessary files during the repackaging process:
$ uscan
Newest version of kubernetes on remote site is 1.32.3, specified download version is 1.32.3
Successfully repacked ../v1.32.3 as ../kubernetes_1.32.3+ds.orig.tar.gz, deleting 30616 files from it.
The results were dramatic. By comparing the original upstream tarball with our repackaged version, we can see that our approach reduced the tarball size by over 75%:
$ du -h upstream-v1.32.3.tar.gz kubernetes_1.32.3+ds.orig.tar.gz
14M	upstream-v1.32.3.tar.gz
3.2M	kubernetes_1.32.3+ds.orig.tar.gz
This significant reduction wasn t just about saving space. By removing over 30,000 files, we simplified the package, making it more maintainable. Each dependency could now be properly tracked, updated, and patched independently, resolving the security concerns that had plagued the previous packaging approach.

Dependency Graph To give you an idea of the complexity involved in packaging Kubernetes for Debian, the image below is a dependency graph generated with debtree, visualizing all the Go modules and other dependencies required to build the kubectl binary. kubectl-depgraph This web of nodes and edges represents every module and its relationship during the compilation process of kubectl. Each box is a Debian package, and the lines connecting them show how deeply intertwined the ecosystem is. What might look like a mess of blue spaghetti is actually a clear demonstration of the vast and interconnected upstream world that tools like kubectl rely on. But more importantly, this graph is a testament to the effort that went into making kubectl build entirely using Debian-packaged dependencies only, no vendoring, no downloading from the internet, no proprietary blobs.

Upstream Version 1.32.3 and Beyond After nearly two years of work, we successfully uploaded version 1.32.3+ds of kubectl to Debian unstable. kubernetes/-/merge_requests/1 The new package also includes:
  • Zsh, Fish, and Bash completions installed automatically
  • Man pages and metadata for improved discoverability
  • Full integration with kind and docker for testing purposes

Integration Testing with Autopkgtest To ensure the reliability of kubectl in real-world scenarios, we developed a new autopkgtest suite that runs integration tests using real Kubernetes clusters created via Kind. Autopkgtest is a Debian tool used to run automated tests on binary packages. These tests are executed after the package is built but before it s accepted into the Debian archive, helping catch regressions and integration issues early in the packaging pipeline. Our test workflow validates kubectl by performing the following steps:
  • Installing Kind and Docker as test dependencies.
  • Spinning up two local Kubernetes clusters.
  • Switching between cluster contexts to ensure multi-cluster support.
  • Deploying and scaling a sample nginx application using kubectl.
  • Cleaning up the entire test environment to avoid side effects.
  • debian/tests/kubectl.sh

Popcon: Measuring Adoption To measure real-world usage, we rely on data from Debian s popularity contest (popcon), which gives insight into how many users have each binary installed. popcon-graph popcon-table Here s what the data tells us:
  • kubectl (new binary): Already installed on 2,124 systems.
  • golang-k8s-kubectl-dev: This is the Go development package (a library), useful for other packages and developers who want to interact with Kubernetes programmatically.
  • kubernetes-client: The legacy package that kubectl is replacing. We expect this number to decrease in future releases as more systems transition to the new package.
Although the popcon data shows activity for kubectl before the official Debian upload date, it s important to note that those numbers represent users who had it installed from upstream source-lists, not from the Debian repositories. This distinction underscores a demand that existed even before the package was available in Debian proper, and it validates the importance of bringing it into the archive.
Also worth mentioning: this number is not the real total number of installations, since users can choose not to participate in the popularity contest. So the actual adoption is likely higher than what popcon reflects.

Community and Documentation The team also maintains a dedicated wiki page which documents:
  • Maintained tools and packages
  • Contribution guidelines
  • Our roadmap for the upcoming Debian releases
https://debian-kubernetes.org

Looking Ahead to Debian 13 (Trixie) The next stable release of Debian will ship with kubectl version 1.32.3, built from a clean, de-vendorized source. This version includes nearly all the latest upstream features, and will be the first time in years that Debian users can rely on an up-to-date, policy-compliant kubectl directly from the archive. By comparing with upstream, our Debian package even delivers more out of the box, including shell completions, which the upstream still requires users to generate manually. In 2025, the Debian Kubernetes team will continue expanding our packaging efforts for the Kubernetes ecosystem. Our roadmap includes:
  • kubelet: The primary node agent that runs on each node. This will enable Debian users to create fully functional Kubernetes nodes without relying on external packages.
  • kubeadm: A tool for creating Kubernetes clusters. With kubeadm in Debian, users will then be able to bootstrap minimum viable clusters directly from the official repositories.
  • helm: The package manager for Kubernetes that helps manage applications through Kubernetes YAML files defined as charts.
  • kompose: A conversion tool that helps users familiar with docker-compose move to Kubernetes by translating Docker Compose files into Kubernetes resources.

Final Thoughts This journey was only possible thanks to the amazing support of the debian-devel-br community and the collective effort of contributors who stepped up to package missing dependencies, fix bugs, and test new versions. Special thanks to:
  • Carlos Henrique Melara (@charles)
  • Guilherme Puida (@puida)
  • Jo o Pedro Nobrega (@jnpf)
  • Lucas Kanashiro (@kanashiro)
  • Matheus Polkorny (@polkorny)
  • Samuel Henrique (@samueloph)
  • Sergio Cipriano (@cipriano)
  • Sergio Durigan Junior (@sergiodj)
I look forward to continuing this work, bringing more Kubernetes tools into Debian and improving the developer experience for everyone.

Arthur Diniz: Bringing Kubernetes Back to Debian

I ve been part of the Debian Project since 2019, when I attended DebConf held in Curitiba, Brazil. That event sparked my interest in the community, packaging, and how Debian works as a distribution. In the early years of my involvement, I contributed to various teams such as the Python, Golang and Cloud teams, packaging dependencies and maintaining various tools. However, I soon felt the need to focus on packaging software I truly enjoyed, tools I was passionate about using and maintaining. That s when I turned my attention to Kubernetes within Debian.

A Broken Ecosystem The Kubernetes packaging situation in Debian had been problematic for some time. Given its large codebase and complex dependency tree, the initial packaging approach involved vendorizing all dependencies. While this allowed a somewhat functional package to be published, it introduced several long-term issues, especially security concerns. Vendorized packages bundle third-party dependencies directly into the source tarball. When vulnerabilities arise in those dependencies, it becomes difficult for Debian s security team to patch and rebuild affected packages system-wide. This approach broke Debian s best practices, and it eventually led to the abandonment of the Kubernetes source package, which had stalled at version 1.20.5. Due to this abandonment, critical bugs emerged and the package was removed from Debian s testing channel, as we can see in the package tracker.

New Debian Kubernetes Team Around this time, I became a Debian Maintainer (DM), with permissions to upload certain packages. I saw an opportunity to both contribute more deeply to Debian and to fix Kubernetes packaging. In early 2024, just before DebConf Busan in South Korea, I founded the Debian Kubernetes Team. The mission of the team was to repackage Kubernetes in a maintainable, security-conscious, and Debian-compliant way. At DebConf, I shared our progress with the broader community and received great feedback and more visibility, along with people interested in contributing to the team. Our first tasks was to migrate existing Kubernetes-related tools such as kubectx, kubernetes-split-yaml and kubetail into a dedicated namespace on Salsa, Debian s GitLab instance. Many of these tools were stored across different teams (like the Go team), and consolidating them helped us organize development and focus our efforts.

De-vendorizing Kubernetes Our main goal was to un-vendorize Kubernetes and bring it up-to-date with upstream releases. This meant:
  • Removing the vendor directory and all embedded third-party code.
  • Trimming the build scope to focus solely on building kubectl, Kubernetes CLI.
  • Using Files-Excluded in debian/copyright to cleanly drop unneeded files during source imports.
  • Rebuilding the dependency tree, ensuring all Go modules were separately packaged in Debian.
We used uscan, a standard Debian packaging tool that fetches upstream tarballs and prepares them accordingly. The Files-Excluded directive in our debian/copyright file instructed uscan to automatically remove unnecessary files during the repackaging process:
$ uscan
Newest version of kubernetes on remote site is 1.32.3, specified download version is 1.32.3
Successfully repacked ../v1.32.3 as ../kubernetes_1.32.3+ds.orig.tar.gz, deleting 30616 files from it.
The results were dramatic. By comparing the original upstream tarball with our repackaged version, we can see that our approach reduced the tarball size by over 75%:
$ du -h upstream-v1.32.3.tar.gz kubernetes_1.32.3+ds.orig.tar.gz
14M	upstream-v1.32.3.tar.gz
3.2M	kubernetes_1.32.3+ds.orig.tar.gz
This significant reduction wasn t just about saving space. By removing over 30,000 files, we simplified the package, making it more maintainable. Each dependency could now be properly tracked, updated, and patched independently, resolving the security concerns that had plagued the previous packaging approach.

Dependency Graph To give you an idea of the complexity involved in packaging Kubernetes for Debian, the image below is a dependency graph generated with debtree, visualizing all the Go modules and other dependencies required to build the kubectl binary. kubectl-depgraph This web of nodes and edges represents every module and its relationship during the compilation process of kubectl. Each box is a Debian package, and the lines connecting them show how deeply intertwined the ecosystem is. What might look like a mess of blue spaghetti is actually a clear demonstration of the vast and interconnected upstream world that tools like kubectl rely on. But more importantly, this graph is a testament to the effort that went into making kubectl build entirely using Debian-packaged dependencies only, no vendoring, no downloading from the internet, no proprietary blobs.

Upstream Version 1.32.3 and Beyond After nearly two years of work, we successfully uploaded version 1.32.3+ds of kubectl to Debian unstable. kubernetes/-/merge_requests/1 The new package also includes:
  • Zsh, Fish, and Bash completions installed automatically
  • Man pages and metadata for improved discoverability
  • Full integration with kind and docker for testing purposes

Integration Testing with Autopkgtest To ensure the reliability of kubectl in real-world scenarios, we developed a new autopkgtest suite that runs integration tests using real Kubernetes clusters created via Kind. Autopkgtest is a Debian tool used to run automated tests on binary packages. These tests are executed after the package is built but before it s accepted into the Debian archive, helping catch regressions and integration issues early in the packaging pipeline. Our test workflow validates kubectl by performing the following steps:
  • Installing Kind and Docker as test dependencies.
  • Spinning up two local Kubernetes clusters.
  • Switching between cluster contexts to ensure multi-cluster support.
  • Deploying and scaling a sample nginx application using kubectl.
  • Cleaning up the entire test environment to avoid side effects.
  • debian/tests/kubectl.sh

Popcon: Measuring Adoption To measure real-world usage, we rely on data from Debian s popularity contest (popcon), which gives insight into how many users have each binary installed. popcon-graph popcon-table Here s what the data tells us:
  • kubectl (new binary): Already installed on 2,124 systems.
  • golang-k8s-kubectl-dev: This is the Go development package (a library), useful for other packages and developers who want to interact with Kubernetes programmatically.
  • kubernetes-client: The legacy package that kubectl is replacing. We expect this number to decrease in future releases as more systems transition to the new package.
Although the popcon data shows activity for kubectl before the official Debian upload date, it s important to note that those numbers represent users who had it installed from upstream source-lists, not from the Debian repositories. This distinction underscores a demand that existed even before the package was available in Debian proper, and it validates the importance of bringing it into the archive.
Also worth mentioning: this number is not the real total number of installations, since users can choose not to participate in the popularity contest. So the actual adoption is likely higher than what popcon reflects.

Community and Documentation The team also maintains a dedicated wiki page which documents:
  • Maintained tools and packages
  • Contribution guidelines
  • Our roadmap for the upcoming Debian releases
https://debian-kubernetes.org

Looking Ahead to Debian 13 (Trixie) The next stable release of Debian will ship with kubectl version 1.32.3, built from a clean, de-vendorized source. This version includes nearly all the latest upstream features, and will be the first time in years that Debian users can rely on an up-to-date, policy-compliant kubectl directly from the archive. By comparing with upstream, our Debian package even delivers more out of the box, including shell completions, which the upstream still requires users to generate manually. In 2025, the Debian Kubernetes team will continue expanding our packaging efforts for the Kubernetes ecosystem. Our roadmap includes:
  • kubelet: The primary node agent that runs on each node. This will enable Debian users to create fully functional Kubernetes nodes without relying on external packages.
  • kubeadm: A tool for creating Kubernetes clusters. With kubeadm in Debian, users will then be able to bootstrap minimum viable clusters directly from the official repositories.
  • helm: The package manager for Kubernetes that helps manage applications through Kubernetes YAML files defined as charts.
  • kompose: A conversion tool that helps users familiar with docker-compose move to Kubernetes by translating Docker Compose files into Kubernetes resources.

Final Thoughts This journey was only possible thanks to the amazing support of the debian-devel-br community and the collective effort of contributors who stepped up to package missing dependencies, fix bugs, and test new versions. Special thanks to:
  • Carlos Henrique Melara (@charles)
  • Guilherme Puida (@puida)
  • Jo o Pedro Nobrega (@jnpf)
  • Lucas Kanashiro (@kanashiro)
  • Matheus Polkorny (@polkorny)
  • Samuel Henrique (@samueloph)
  • Sergio Cipriano (@cipriano)
  • Sergio Durigan Junior (@sergiodj)
I look forward to continuing this work, bringing more Kubernetes tools into Debian and improving the developer experience for everyone.

11 May 2025

Sergio Durigan Junior: Debian Bug Squashing Party Brazil 2025

With the trixie release approaching, I had the idea back in April to organize a bug squashing party with the Debian Brasil community. I believe the outcome was very positive, and we were able to tackle and fix quite a number of release-critical bugs. This is a brief report of what we did.

A remote BSP It s not the first time I organize a BSP: back in 2019, I helped throw another similar party in Toronto. The difference this time is that, because Brazil is a big country and (perhaps most importantly) because I m not currently living there, the BSP had to be done online. I m a fan of social interactions (especially with the Brazilian community), and in my experience we usually can achieve much more when we get together in a physical place, but hey, you gotta do what you gotta do Most (if not all) of the folks interested in participating had busy weekdays, so it was decided that we would meet during the weekends and try to work on a few bugs over Jitsi. Nothing stopped people from working on bugs during the week as well, of course.

A tag to rule them all We used the bsp-2025-04-brazil usertag to mark those bugs that were touched by us. You can see the full list of bugs here, although the current list (as of 2025-05-11) is smaller than the one we had by the end of April. I don t know what happened; maybe it s some glitch with the BTS, or maybe someone removed the usertag by mistake.

Stats In total, we had:
  • 7 participants
  • 37 bugs handled. Of those,
  • 35 bugs fixed
The BSP officially started on 04 April 2025, and ended on 30 April 2025. I was able to attend meetings during two weekends; other people participated more sporadically.

Outcome As I said above, the Debian Brasil community is great and very engaged in the project. Speaking more specifically about the Debian Brasil Devel group, I can say that We have contributors with strong technical skills, and I really love that we have this inclusive, extremely technical culture where debugging and understanding things is really core to pretty much all our discussions. We already meet weekly on Thursdays to talk shop and help newcomers, so having a remote BSP with this group seemed like a logical thing to do. I m really glad to see our results and even happier to hear positive feedback from the community during the last MiniDebConf in Macei . There s some interest in organizing another BSP, this time face-to-face and during the next DebConf. I m all for it, as I love fixing bugs and having a great time with friends. If you re interested in attending, let me know. Thanks, and until next time.

7 September 2024

Sergio Durigan Junior: Chatting in the 21st century

Several people have been asking me to explain and/or write about my solution for chatting nowadays. I realize that the current scenario is much more complex than, say, 10 or 20 years ago. Back then, this post would probably be more about the IRC client I used than about different chatting technologies. I have also spent a non trivial amount of time setting things up the way I want, so I understand that it s about time to write about my setup not only because I think it can be helpful to others, but also because I would like to document things for myself.

The backbone: Matrix I chose to use Matrix as the place where I integrate everything. Despite there being some heavy (and justified) criticism on the protocol itself, it serves me well for what I need right now. Obviously, I don t like the fact that I have to provide Matrix and all of its accompanying bridges a VPS with 4GB of RAM and 3 vCPUs, but I think that that ship has sailed, unfortunately. In an ideal world, I would be using XMPP and dedicating only a fraction of the resources I m using today to have a full chat system. And since I have been running my personal XMPP server for more than a decade now, I did try to find a solution that would allow me to keep using it, but unfortunately the protocol became almost a hobbyist thing, so there s that.

A few disclaimers I self-host everything, including my Matrix server. Much of what I did won t work if you don t self-host Matrix, so keep that in mind. This won t be a post teaching you how to deploy the services. My intention is to describe what I use and for what purpose. Also, as much as I try to use Debian packages for everything I do, I opted to deploy all services using a community-maintained Ansible playbook which is very well written and organized: matrix-docker-ansible-deploy. Last but not least, as I said above, you will likely need a machine with a good amount of RAM, CPU and storage, especially if you deploy Synapse as your Matrix homeserver (which is what I recommend if you plan to use the bridges I ll mention). My current VPS has 4GB of RAM, 3 vCPUs and 80GB of storage (of which I m currently using approximately 55GB).

Problem #1: my Matrix client(s) There are a lot of clients that can talk the Matrix protocol, but most of them are either web clients or GUI programs. I live on the terminal, more specifically inside Emacs, so I settled for the amazing ement.el Emacs mode. It works surprisingly well, but unfortunately doesn t support end-to-end encryption out of the box; for that, you have to hook it up with pantalaimon. Unfortunately, the project seems abandoned and therefore I don t recommend you to use it. I don t use it myself. When I have to reply some E2E encrypted message from another user, I go to my web browser and use my self-hosted Element client. It s a nuisance, but one that I m willing to accept because of security concerns. If you re into web clients and don t want to use Element (because it is heavy), you can try Cinny. It s lightweight and supports a decent set of features. If you re a terminal lover but don t use Emacs, you may want to try gomuks or iamb.

Problem #2: IRC bridging There are basically two types of IRC bridges for Matrix:
  • The regular and most used matrix-appservice-irc. This bridge takes Matrix to IRC (think of IRC users with the [m] suffix appended to their nicknames), and is what the matrix.org and other big homeservers (including matrix.debian.social) use. It s a complex service which allows thousands of Matrix users to connect to IRC networks, but that unfortunately has complex problems and is only worth using if you intend to host a community server.
  • A bouncer-like bridge called Heisenbridge. This is what I use personally. It takes IRC to Matrix, which means that people on IRC will not know that you re using Matrix. This bridge is much simpler, and because it acts like a bouncer it s pretty much impossible for it to cause problems with the IRC network.
Due to the fact that I sometimes like to use other IRC clients, I still run a regular ZNC bouncer, and I use Heisenbridge to connect to my ZNC. This means that I can use, e.g., ERC inside Emacs and my Matrix bridge at the same time. But you don t necessarily need to run another bouncer; you can simply use Heisenbridge and connect directly to the IRC network(s) you want. A word of caution, though: unlike ZNC, Heisenbridge doesn t support per-user configuration when you use it in bouncer mode. This is the reason why you need to self-host it, and why it s not possible to offer the service to other users (they would have access to your IRC network configuration otherwise). It s also worth talking about logs. I find that keeping logs of everything that goes on IRC has saved me a bunch of times, and so I find it really important to continue doing that. Unfortunately, neither ement.el nor Element support logging things out of the box (at least not that I know). This is also one of the reasons why I still keep my ZNC around: I configure it to log everything.

Problem #3: Telegram I don t use Telegram myself, but unfortunately several people from the Debian community do, especially in Brazil. There is a whole Debian community on Telegram, and I wanted to be able to bridge our Debian Matrix channels to their Telegram counterparts. I am currently using mautrix-telegram for that, and it s working great. You need someone with a Telegram account to configure their credentials so that the bridge can connect to it, but afterwards it s really easy to bridge channels together.

Problem #4: GitLab webhooks Something else I wanted to be able to do was to receive notifications regarding new issues, merge requests and other activities from Salsa. For this, I m using maubot, which is awesome and has a huge list of plugins. I m using the gitlab one.

Final thoughts Overall, I m satisfied with the setup I have now. It has certainly taken some time and effort to find the right tool for each problem I needed to solve, and I still feel like there are some rough edges to soften (like the fact that my Emacs client doesn t support E2E encryption out of the box, or the whole logging situation), but otherwise things are working fine and I haven t had any big problems with the deployment. You do have to be much more careful about stuff (for example, when I installed an unrelated service that hijacked my Apache configuration and made Matrix s federation silently stop working), though. If you have more specific questions about any part of my setup, shoot me an email and I ll do my best to help. Happy chatting!

4 September 2024

Samuel Henrique: DebConf24 was fun!: Security, curl, wcurl, Debian's quality

A picture of a badger2040w with Samuel's badge and the curl manpage PCB on the side

tl;dr DebConf24 was fun! A playlist of all of my talks, with subtitles (en, pt-br) and chapters is available on YouTube.

Overview DebConf24 was held in Busan, South Korea, between Sunday July 28th to Sunday August 4th 2024. As usual for DebConfs, I had a great time meeting my friends, but also met new people and got to learn a bit about the interesting things they're working on. I ended up getting too excited during the talk submission stage of the conference and as a result I presented 5 different activities (3 talks, 1 BoF and 1 lightning talk). Since I was too busy with the presentations, I did not have a lot of time to actually hang out with folks, or even to go out in the city, I guess I've learned my lesson for next time. The main purpose of this post is to write about all of the things I presented at the conference. I did want to list some of the interesting talks I've watched, but that I would not be able to be fair as I'm sure I would miss some. You can get the schedule and the recordings of any talks from the conference's website: https://debconf24.debconf.org/schedule/

wcurl Lightning Talk The most fun of my presentations, during the second-to-last day of the conference, I've asked for help from Sergio Durigan Junior <sergiodj> to setup an URL containing a whitespace and redirecting that to wcurl's manpage. I then did a little demo to showcase why me (and a lot others) struggle with downloading things with curl, and how wcurl solves that. https://www.youtube.com/watch?v=eM8M5qa4pPM

Fixing CVEs on Debian: Everything you probably know already I've always felt like DebConf was missing security-related talks, so I decided to do something about it and presented a few of the things I've learned when fixing CVEs for Debian. This is an area where we don't get a lot of new contributors, I'm trying to change that, and this talk can be used to introduce newcomers to it. https://www.youtube.com/watch?v=XzNVVILVyUM

The secret sauce of Debian Debian is not very vocal about all of the nice things it has regarding quality-assurance, testing, or CI, even though it's at the state-of-the-art for a lot of things. This talk is an initial step towards making people aware of the cool things happening behind the scenes. Ideally we should have it well-documented somewhere. https://www.youtube.com/watch?v=x_X2IBnpjic

"I use Debian BTW": fzf, tmux, zoxide and friends One of my earliest good memories of Debian was when it started coming with a colored PS1 by default, I still remember the feeling of relief whenever I jumped into a Debian server and didn't have to deal with a black and white PS1. There's still a lot of room for Debian to ship better defaults, and I think some of them can actually happen. This talk is a bit of a silly one where I'm just making people aware of the existence of a few Golang/Rust CLI tools, and also some dotfiles configurations that should probably be the default. https://www.youtube.com/watch?v=tfto3Seokn4

curl The curl project does such a great job with their security advisories that it will likely never receive the amount of praise it deserves, but I did my best at mentioning it throughout my CVEs talk. Maybe I will write more extensively about this someday, but in case I don't:
There's no other project which always consistently mentions the exact range of commits that are affected by a given CVE. Forget about whether the versions are EOL, curl doesn't have LTS releases, yet they do such a great job at clearly documenting their CVEs that I would take that over having LTS releases anytime (that's for curl at least, I acknowledge some types of projects have a different need for LTS releases). Not only that, but they are also always careful about explaining alternative mitigations such as configuration changes, build flags that defuse the exploitation, or parameters that you should not use.
Just like we tend to do every time we meet, me and the other Debian curl maintainers spent the first 2 or 3 days of the conference talking about how we wanted to eventually meet up to discuss the package. It was going to be informal, maybe during the Cheese and Wine party, but then I've realized we should make it part of the official schedule, which would also give us the recordings for later. And so the "curl maintainers BoF" happened, where we spoke about HTTP3, GnutTLS, wcurl and other things. https://www.youtube.com/watch?v=fL7hSypUTdM

wcurl Right after that BoF, Daniel Stenberg asked if we were interested in having wcurl adopted into curl, which we definitely were, so wcurl is now part of the curl project. Daniel was also kind enough to design a logo for the project, which makes me especially happy because I can stop with my own approach at a logo (which I had to redo every few days): A laptop with a curl and a GoHorse sticker, there's a 'w' handwritten with a marker on the right side of the curl sticker, making it 'wcurl' And here is the new logo: 'wcurl' written with the same font and colors as the curl logo, with the 'w' being green instead of blue, and a download icon at the end Much better, I would say :)

curl Swag DebConf24 was my chance at forwarding some curl swag items to the other curl maintainers, so both Sergio Durigan Junior <sergiodj> and Carlos Henrique Lima Melara <charles> got the curl-up t-shirt and the very cool curl PCB coaster, both gifted by Daniel Stenberg. Unfortunately I didn't have any of that for DebConf attendees, but I did drop loads of curl stickers at the stickers table, they were gone very quickly. A table full of different stickers, curl stickers can be seen over the whole table

For the future I used to think the most humbling experience you could have as someone who presented a talk was to have to watch it yourself, you notice a lot of mistakes and you instantly think about things that should be done differently. It turns out the most humbling thing to do is actually to write subtitles for your talks, I noticed every single mistake, often multiple times. So after spending more than 30 hours writing the subtitles for both English and Brazilian Portuguese for my talks, I feel like it's going to be much easier to avoid committing the same mistakes again. After some time you stop feeling shame about those mistakes and you're just left with feelings of annoyance, and at that point it becomes easier to consciously avoid them. I am collecting a list of things I wish I had done differently on all of those talks, so if I end up presenting any one of them again, it will be an improved version. A picture from the top of a group of conference attendees, there's about 150 people in the picture

4 July 2024

Samuel Henrique: Debian's curl now supports HTTP3

tl;dr Starting with curl 8.0.0-2, you can now use HTTP3.
curl --http3-only https://example.com
Or, if you would like to try it out in a container:
podman run debian:unstable apt install --update -y curl && curl --http3-only https://example.com
(in case you haven't noticed, apt now has the --update option for the upgrade and install commands, although not available on stable yet)

Availability
  • Debian unstable - Since 2024-07-02
  • Debian testing - Since 2024-07-18
  • Debian 12/bookworm backports - Expected by the end of August 2024.
  • Debian 12/bookworm - Due to the mechanisms we have in place to make sure Debian stable is in fact stable, we will never be able to ship this in the regular repository. Users can make use of the backports repositories instead.
  • Debian derivatives - Rolling releases will get it by the time it's on Debian testing (e.g.: Kali Linux). Stable derivatives only in their next major release.

The challenge HTTP3 is fresh new, well... not really, but at least fresh enough that I'm not aware of any other Linux distribution supporting it on curl, the reason is likely two-fold:
  1. OpenSSL is not there yet OpenSSL still doesn't have proper HTTP3 support, and given that OpenSSL is so widely used, almost every curl distributor/packager will build curl with it and thus changing the TLS backend to something else is risky. Unfortunately, proper support for the OpenSSL libcurl is unlikely to come anytime before the end of this year, the OpenSSL performance is not good enough yet as of version 3.3. Daniel Stenberg has written about the state of this multiple times, most recently at HTTP/3 in curl mid 2024, if you're interested, I suggest reading through his other posts as well. Some might have noticed that nginx does support HTTP3 through OpenSSL, although when you look closely, it's not exactly perfect:
    An SSL library that provides QUIC support is recommended to build nginx, such as BoringSSL, LibreSSL, or QuicTLS. Otherwise, the OpenSSL compatibility layer will be used that does not support early data.
    As you can see, they don't recommend using OpenSSL, and when doing so, you don't get complete support.

  2. HTTP3 support for GnuTLS/nghttp3/ngtcp2 is recent The non-experimental support arrived back in October 2023, and so that's when I started seriously planning for this. curl has been working on HTTP3 support for years, and so it did support other TLS backends before that, but out of them, the one most feasible for a distribution to ship would be GnuTLS, which gets HTTP3 support through ngctp2 and nghttp3.

How it was done The Debian curl package has historically shipped at least two variants of libcurl, an OpenSSL and a GnuTLS one. The OpenSSL libcurl can't support HTTP3 for the reasons explained above, but the GnuTLS libcurl can (with ngtcp2 and nghtp3). Debian packages can choose which version of libcurl to link against (without having to modify any upstream source code). Debian's "git" package being a famous example of a package that links against the GnuTLS libcurl. Enabling HTTP3 on curl was done in three steps:
  1. Make sure all required dependencies fulfill the minimum requirements.
  2. Enable HTTP3 for GnuTLS libcurl.
  3. Change the libcurl used by the curl CLI, from OpenSSL to GnuTLS.
curl's HTTP3 support requires a somewhat recent version of nghttp3 and updating that required a transition (due to the SONAME bump), while we've also had months of freeze for transitions due to the time_t transition. After the dependencies were in place, enabling HTTP3 for the GnuTLS libcurl was straightforward. Then, for the last part, we had to switch the TLS backend used by the curl CLI. Doing the swap is also quite easy on the packaging level, but we have to consider the chances of this change breaking our users' environments.

Ensuring there are no breakages The first thing to consider regarding breakages is that this change is not going to be pushed directly to the current Debian stable releases, it will be present in the next stable release (13/trixie) but the current one will stick to the version that's already shipped. Secondly, we have to consider the risk of losing the ability to use certain parameters from the curl CLI which could be limited to the OpenSSL backend. During curl-up 2024, the curl developers pointed out the existence of a page that lists the TLS related options and the backends they work with. Analysing that page, ignoring all of the options that are suffixed with "BLOB" (only pertinent to the library, not the CLI), the only one left which is attention worthy is CURLOPT_ECH.
This experimental feature requires a special build of OpenSSL, as ECH is not yet supported in OpenSSL releases. In contrast ECH is supported by the latest BoringSSL and wolfSSL releases.
As it turns out, Encrypted Client Hello is experimental and it's not supported by the vanilla OpenSSL. This was enough of an investigation for me to go ahead with the change. Noting that even in the worst case scenario (we find a horrible regression), we can rollback without having affected a single stable release. Now that the package is on Debian unstable, the CI tests (autopkgtest) of every package that depends on curl is currently running, the results are compared against the migration-reference (in this case, the curl CLI with OpenSSL, before the change). If everything goes right, curl with HTTP3 support will migrate to Debian testing in around 5 days. If we spot any issues, we'll have to solve them first and it's going to be hard to predict how long it takes, although it's fair to expect less than a month.

Feedback Feel free to join the Matrix room for the Debian curl maintainers:
https://matrix.to/#/#debian-curl-maintainers:matrix.org

Acknowledgements It took us a bit longer than expected to be able to enable HTTP3, nonetheless it's still early enough to be excited about. A lot of people were crucial to make this happen. I should recognize in the first place, obviously, the curl developers and the developers of the supporting libraries: GnuTLS, nghttp3, ngtcp2. Participating in the curl-up 2024 conference helped me get motivated to push this through, besides becoming aware of the right documentation to research for impact. On the Debian side, Sakirnth Nagarasa <sakirnth> was responsible for updating and taking care of the transition for nghttp3 and ngtcp2. Also on the Debian side, I've got loads of help and support from the co-maintainers of the curl package: Sergio Durigan Junior <sergiodj> and Carlos Henrique Lima Melara <charles>.

Changes since publication

2024-07-18
  • Update date of availability for Debian testing and expected date for bookworm backports.
  • We have historically spoken Portuguese in the room but we'll switch to English in case anyone joins.

3 July 2024

Samuel Henrique: Announcing wcurl: a curl wrapper to download files

tl;dr Whenever you need to download files through the terminal and don't feel like using wget:
wcurl example.com/filename.txt
Manpage:
https://manpages.debian.org/unstable/curl/wcurl.1.en.html

Availability (comes installed with the curl package):
  • Debian unstable - Since 2024-07-02
  • Debian testing - Since 2024-07-18
  • Debian 12/bookworm backports - Expected by the end of August 2024.
  • Debian 12/bookworm - Depends on whether Debian's release team will approve it, it could be available in the next point release.
  • Debian derivatives - Rolling releases will get it by the time it's on Debian testing (e.g.: Kali Linux). Stable derivatives only in their next major release.
If you don't want to wait for the package update to arrive, you can always copy the script and place it in your /usr/bin, the code is here:
https://github.com/Debian/wcurl/blob/main/wcurl
https://salsa.debian.org/debian/wcurl/-/blob/main/wcurl?ref_type=heads

Smoother CLI experience Starting with curl version 8.8.0-2, the Debian's curl package now ships a wcurl executable. wcurl is the solution for those who just need to download files without having to remember curl's parameters for things like automatically naming the files. Some people, myself included, would fall back to using wget whenever there was a need to download a file. Sometimes even installing wget just for that usecase. After all, it's easier to remember "apt install wget" rather than "curl -L -O -C - ...". wcurl consists of a simple shell script that provides sane defaults for the curl invocation, for when the use case is to just download files. By default, wcurl will:
  • Encode whitespaces in URLs;
  • Download multiple URLs in parallel if the installed curl's version is >= 7.66.0;
  • Follow redirects;
  • Automatically choose a filename as output;
  • Avoid overwriting files if the installed curl's version is >= 7.83.0 (--no-clobber);
  • Perform retries;
  • Set the downloaded file timestamp to the value provided by the server, if available;
  • Default to the protocol used as https if the URL doesn't contain any;
  • Disable curl's URL globbing parser so and [] characters in URLs are not treated specially.
Example to download a single file:
wcurl example.com/filename.txt
If you ever need to set a custom flag, you can make use of the --curl-options wcurl option, anything set there will be passed to the curl invocation. Just beware that if you need to set any custom flags, it's likely you will be better served by calling curl directly. The --curl-options option is there to allow for some flexibility in unforeseen circumstances.

The need for wcurl I've always felt a bit ashamed of not remembering curl's parameters for downloading a file and automatically naming it, having resorted to wget most of the times this was needed (even installing wget when it wasn't there, just for this). I've spoken to a few other experienced people I know and confirmed what could be obvious to others: a lot of people struggle with this. Recently, the curl project released the results of 2024's curl survey, which also showed this is as a much needed feature, just look at some of the answers:

Q: Which curl command line option do you think needs improvement and how?
-O, I really want wget like functionality where I don't have to specify the name
Downloading a file (like wget) could be improved - with automatic naming of the file
downloading files - wget is much cleaner
I wish the default behaviour when GETting a binary was to drop it on disk. That's the only reason 'wget foo.tgz" is still ingrained in my muscle memory .
Maybe have a way to download without specifying something in -o (the only reason i used wget still)
--remote-time should be default
--remote-name-all could really use a short flag

Q: If you miss support for something, tell us what!
"Write the data to the file named in the URL (or in redirects if I'm feeling daring), and timestamp the file to the last-modified-date". This is the main reason I'm still using wget.
I can finally feel less bad about falling back to wget due to not remembering the parameters I want.

Idealization vs. reality I don't believe curl will ever change its default behavior in such a way that would accommodate this need, as that would have a side-effect of breaking things which expect the current behavior (the blast radius is literally the solar system). This means a new executable needs to be shipped side-by-side with curl, an opportunity to start fresh and work with a more focused use case (to download files). Ideally, this new executable would be maintained by the curl project, make use of libcurl under-the-hood, and be available everywhere. Nobody wants to worry if their systems have the tool or not, it should always be there. Given I'm just a Debian Developer, with not as much free time as I wish, I've decided to write a simple shell script wrapper calling the curl CLI under-the-hood. wcurl will come installed with the curl package from now on, and I will check with the release team about shipping it on the current Debian stable as well. Shipping wcurl in other distros will be up to them (Debian-derivatives should pick it up automatically, though). We've tried to make it easy for anyone to ship this by using the curl license, keeping the script POSIX-compliant, and shipping a manpage. Maybe if there's enough interest across distributions, someone might sign up for implementing this in upstream curl and increase its reach. I would be happy with the curl project reusing the wcurl name when that happens. It's unlikely that wcurl would be shipped by curl upstream as it is, assuming they would prefer a solution that uses libcurl direclty (more similar to curl the CLI, to maintain). In the worst case, wcurl becomes a Debian-specific tool that only a few people are aware of, in the best case, it becomes the new go-to CLI tool for simply downloading files. I would be happy if at least someone other than me finds it useful.

Naming is hard When I started working on it, I was calling the new executable "curld" (stands for "curl download"), but then when discussing this in one of our weekly calls in the Debian Bras lia community, it was mentioned that this could be confused for a daemon. We then settled for the name "wcurl", suggested by Carlos Henrique Lima Melara <charles>. It doesn't really stand for anything, but it's very easy to remember. You know... "it's that wget alternative for when you want to use curl instead" :)

Feedback I'm hosting the code on Github and Debian's GitLab instance, feel free to open an issue to provide feedback.
https://salsa.debian.org/debian/wcurl
https://github.com/Debian/wcurl We also have a Matrix room for the Debian curl maintainers:
https://matrix.to/#/#debian-curl-maintainers:matrix.org

Acknowledgments The idea for wcurl came a few days before the curl-up conference 2024. I've been thinking a lot about developer productivity in the terminal lately, different tools and better defaults. Before curl-up, I was also thinking about packaging improvements for the curl package. I don't remember what exactly happened, but I likely had to download something and felt a bit ashamed of maintaining curl and not remembering the parameters to download files the way I wanted. I first discussed this idea in the conference, where I asked the participants about it and there were no concerns raised, and some people said I should give it a go. Participating in curl-up was a really great experience and I'm thankful for the interactions I've had there. On the Debian side, I've got reviews of the code and manpage by Sergio Durigan Junior <sergiodj>, Guilherme Puida Moreira <puida> and Carlos Henrique Lima Melara <charles>. Sergio ended up rewriting the tool to be POSIX-compliant (my version was written in bash), so he takes all the credit for the portability.

Changes since publication

2024-07-18
  • Update date of availability for Debian testing and expected date for bookworm backports.
  • Mention charles as the person who suggested "wcurl" as a name.
  • Update wcurl's -o/--opts options, it's now just --curl-options.
  • Remove mention of language spoken in the Matrix room, we are using English now.
  • Update list of features of wcurl.

13 June 2024

Sergio Durigan Junior: The Pagure Debian package is now orphan

As promised in the last post, I have now orphaned the Pagure Debian package. Here s the full text I posted on the BTS:
After several years, I finally decided to orphan pagure :-(. I haven t been using it as my personal forge anymore, and unfortunately upstream development slowed down quite a bit after the main author and maintainer stopped contributing regularly to the project. But that is not to say that upstream is dead; they are still working towards preparing the next release. Pagure is a big package with several components and an extensive list of build dependencies (and an even bigger testsuite which I never managed to make fully work on Debian). It is not for the faint of heart, and most of the time is usually spent fixing its (build) dependencies so that it doesn t get removed from testing. If I may, I would like to leave some suggestions for a future maintainer.
  • I never had the time to write dep8 tests, mainly because setting up the software is not trivial. It would be great if the package had more Debian-centric testing.
  • Speaking of the hurdles to setting up Pagure, I believe the package installation could be made a bit more automated using debconf. I don t see a way to fully automate it (look at d/README.Debian), but there is certainly room for improvement.
I also left a brief TODO list inside d/README.source; feel free to tackle any item there! I wish the next maintainer can have as much fun with the package as I did when I first made it for Debian! Thank you,
That s it. It was good while it lasted, but I needed to feel myself unburdened so that I don t have that constant feeling of I should be properly maintaining this package . If you feel like you d like to give it a try at maintaining Pagure, now is the time!

26 February 2024

Sergio Durigan Junior: Planning to orphan Pagure on Debian

I have been thinking more and more about orphaning the Pagure Debian package. I don t have the time to maintain it properly anymore, and I have also lost interest in doing so.

What s Pagure Pagure is a git forge written entirely in Python using pygit2. It was almost entirely developed by one person, Pierre-Yves Chibon. He is (was?) a Red Hat employee and started working on this new git forge almost 10 years ago because the company wanted to develop something in-house for Fedora. The software is amazing and I admire Pierre-Yves quite a lot for what he was able to achieve basically alone. Unfortunately, a few years ago Fedora decided to move to Gitlab and the Pagure development pretty much stalled.

Pagure in Debian Packaging Pagure for Debian was hard, but it was also very fun. I learned quite a bit about many things (packaging and non-packaging related), interacted with the upstream community, decided to dogfood my own work and run my Pagure instance for a while, and tried to get newcomers to help me with the package (without much success, unfortunately). I remember that when I had started to package Pagure, Debian was also moving away from Alioth and discussing options. For a brief moment Pagure was a contender, but in the end the community decided to self-host Gitlab, and that s why we have Salsa now. I feel like I could have tipped the scales in favour of Pagure had I finished packaging it for Debian before the decision was made, but then again, to the best of my knowledge Salsa doesn t use our Gitlab package anyway

Are you interested in maintaining it? If you re interested in maintaining the package, please get in touch with me. I will happily pass the torch to someone else who is still using the software and wants to keep it healthy in Debian. If there is nobody interested, then I will just orphan it.

23 May 2023

Sergio Durigan Junior: Using WireGuard to host services at home

It s been a while since I had this idea to leverage the power of WireGuard to self-host stuff at home. Even though I pay for a proper server somewhere in the world, there are some services that I don t consider critical to put there, or that I consider too critical to host outside my home.

It s only NATural With today s ISP packages for end users, I find it very annoying the amount of trouble they create when you try to host anything at home. Dynamic IPs, NAT/CGNAT, port-blocking, traffic shapping are only a few examples of methods or limitations that prevent users from making local services reachable in a reliable way from outside.

WireGuard comes to help If you already pay for a VPS or a dedicated server somewhere, why not use its existing infrastructure (and public availability) in your favour? That s what I thought when I started this journey. My initial idea was to use a reverse proxy to redirect external requests to the service running at my home. But how could I make sure that these requests reach my dynamic-IP-behind-a-NAT-behind-another-NAT? Well, let s create a tunnel! WireGuard is the perfect tool for that because of many things: it s stateless, very performant, secure, and requires very little configuration.

Setting up on the server On the server side (i.e., VPS or dedicated server), you will create the first endpoint. Something like the following should do:
[Interface]
PrivateKey = PRIVATE_KEY_HERE
Address = 10.0.0.1/32
ListenPort = 51821
[Peer]
PublicKey = PUBLIC_KEY_HERE
AllowedIps = 10.0.0.2/32
PersistentKeepalive = 10
A few interesting points to note:
  • The Peer section contains information about the home service that will be configured below.
  • I m using PersistentKeepalive because I have a dynamic IP at my home. If you have a static IP, you could get rid of PersistentKeepalive and specify an Endpoint here (don t forget to set a ListenPort below, in the Interface section).
  • Now you have an IP where you can forward requests to. If we re talking about HTTP traffic, Apache and nginx are absolutely capable of doing it. If we re talking about other kind of traffic, you might want to look into other utilities, like HAProxy, Traefik and others.

Setting up at your home At your home, you will configure the peer:
[Interface]
PrivateKey = PRIVATE_KEY_HERE
Address = 10.0.0.2/32
[Peer]
PublicKey = PUBLIC_KEY_HERE
AllowedIps = 10.0.0.1/32
Endpoint = YOUR_SERVER:51821
PersistentKeepalive = 10

A few notes about security I would be remiss if I didn t say anything about security, especially because we re talking about hosting services at home. So, here are a few recommendations:
  • Make sure to put your services in a separate local network. Using VLANs is also a good option.
  • Don t run services on your personal (or work!) computer, even if they ll be running inside a VM.
  • Run a firewall on the WireGuard interface and make sure that you only allow traffic over the required ports.
Have fun!

13 May 2023

Sergio Durigan Junior: Ubuntu debuginfod and source code indexing

You might remember that in my last post about the Ubuntu debuginfod service I talked about wanting to extend it and make it index and serve source code from packages. I m excited to announce that this is now a reality since the Ubuntu Lunar (23.04) release. The feature should work for a lot of packages from the archive, but not all of them. Keep reading to better understand why.

The problem While debugging a package in Ubuntu, one of the first steps you need to take is to install its source code. There are some problems with this:
  • apt-get source required dpkg-dev to be installed, which ends up pulling in a lot of other dependencies.
  • GDB needs to be taught how to find the source code for the package being debugged. This can usually be done by using the dir command, but finding the proper path to be is usually not trivial, and you find yourself having to use more complex commands like set substitute-path, for example.
  • You have to make sure that the version of the source package is the same as the version of the binary package(s) you want to debug.
  • If you want to debug the libraries that the package links against, you will face the same problems described above for each library.
So yeah, not a trivial/pleasant task after all.

The solution Debuginfod can index source code as well as debug symbols. It is smart enough to keep a relationship between the source package and the corresponding binary s Build-ID, which is what GDB will use when making a request for a specific source file. This means that, just like what happens for debug symbol files, the user does not need to keep track of the source package version. While indexing source code, debuginfod will also maintain a record of the relative pathname of each source file. No more fiddling with paths inside the debugger to get things working properly. Last, but not least, if there s a need for a library source file and if it s indexed by debuginfod, then it will get downloaded automatically as well.

but not a perfect one In order to make debuginfod happy when indexing source files, I had to patch dpkg and make it always use -fdebug-prefix-map when compiling stuff. This GCC option is used to remap pathnames inside the DWARF, which is needed because in Debian/Ubuntu we build our packages inside chroots and the build directories end up containing a bunch of random cruft (like /build/ayusd-ASDSEA/something/here). So we need to make sure the path prefix (the /build/ayusd-ASDSEA part) is uniform across all packages, and that s where -fdebug-prefix-map helps. This means that the package must honour dpkg-buildflags during its build process, otherwise the magic flag won t be passed and your DWARF will end up with bogus paths. This should not be a big problem, because most of our packages do honour dpkg-buildflags, and those who don t should be fixed anyway.

especially if you re using LTO Ubuntu enables LTO by default, and unfortunately we are affected by an annoying (and complex) bug that results in those bogus pathnames not being properly remapped. The bug doesn t affect all packages, but if you see GDB having trouble finding a source file whose full path starts without /usr/src/..., that is a good indication that you re being affected by this bug. Hopefully we should see some progress in the following weeks.

Your feedback is important to us If you have any comments, or if you found something strange that looks like a bug in the service, please reach out. You can either send an email to my public inbox (see below) or file a bug against the ubuntu-debuginfod project on Launchpad.

9 November 2022

Debian Brasil: Brasileiros(as) Mantenedores(as) e Desenvolvedores(as) Debian a partir de julho de 2015

Desde de setembro de 2015, o time de publicidade do Projeto Debian passou a publicar a cada dois meses listas com os nomes dos(as) novos(as) Desenvolvedores(as) Debian (DD - do ingl s Debian Developer) e Mantenedores(as) Debian (DM - do ingl s Debian Maintainer). Estamos aproveitando estas listas para publicar abaixo os nomes dos(as) brasileiros(as) que se tornaram Desenvolvedores(as) e Mantenedores(as) Debian a partir de julho de 2015. Desenvolvedores(as) Debian / Debian Developers / DDs: Marcos Talau Fabio Augusto De Muzio Tobich Gabriel F. T. Gomes Thiago Andrade Marques M rcio de Souza Oliveira Paulo Henrique de Lima Santana Samuel Henrique S rgio Durigan J nior Daniel Lenharo de Souza Giovani Augusto Ferreira Adriano Rafael Gomes Breno Leit o Lucas Kanashiro Herbert Parentes Fortes Neto Mantenedores(as) Debian / Debian Maintainers / DMs: Guilherme de Paula Xavier Segundo David da Silva Polverari Paulo Roberto Alves de Oliveira Sergio Almeida Cipriano Junior Francisco Vilmar Cardoso Ruviaro William Grzybowski Tiago Ilieve
Observa es:
  1. Esta lista ser atualizada quando o time de publicidade do Debian publicar novas listas com DMs e DDs e tiver brasileiros.
  2. Para ver a lista completa de Mantenedores(as) e Desenvolvedores(as) Debian, inclusive outros(as) brasileiros(as) antes de julho de 2015 acesse: https://nm.debian.org/public/people

Debian Brasil: Brasileiros(as) Mantenedores(as) e Desenvolvedores(as) Debian a partir de julho de 2015

Desde de setembro de 2015, o time de publicidade do Projeto Debian passou a publicar a cada dois meses listas com os nomes dos(as) novos(as) Desenvolvedores(as) Debian (DD - do ingl s Debian Developer) e Mantenedores(as) Debian (DM - do ingl s Debian Maintainer). Estamos aproveitando estas listas para publicar abaixo os nomes dos(as) brasileiros(as) que se tornaram Desenvolvedores(as) e Mantenedores(as) Debian a partir de julho de 2015. Desenvolvedores(as) Debian / Debian Developers / DDs: Marcos Talau Fabio Augusto De Muzio Tobich Gabriel F. T. Gomes Thiago Andrade Marques M rcio de Souza Oliveira Paulo Henrique de Lima Santana Samuel Henrique S rgio Durigan J nior Daniel Lenharo de Souza Giovani Augusto Ferreira Adriano Rafael Gomes Breno Leit o Lucas Kanashiro Herbert Parentes Fortes Neto Mantenedores(as) Debian / Debian Maintainers / DMs: Guilherme de Paula Xavier Segundo David da Silva Polverari Paulo Roberto Alves de Oliveira Sergio Almeida Cipriano Junior Francisco Vilmar Cardoso Ruviaro William Grzybowski Tiago Ilieve
Observa es:
  1. Esta lista ser atualizada quando o time de publicidade do Debian publicar novas listas com DMs e DDs e tiver brasileiros.
  2. Para ver a lista completa de Mantenedores(as) e Desenvolvedores(as) Debian, inclusive outros(as) brasileiros(as) antes de julho de 2015 acesse: https://nm.debian.org/public/people

14 August 2022

Sergio Durigan Junior: Debuginfod is coming to Ubuntu

These past couple of months I have been working to bring debuginfod to Ubuntu. I thought it would be a good idea to make this post and explain a little bit about what the service is and how I'm planning to deploy it. A quick recap: what's debuginfod? Here's a good summary of what debuginfod is:
debuginfod is a new-ish project whose purpose is to serve
ELF/DWARF/source-code information over HTTP.  It is developed under the
elfutils umbrella.  You can find more information about it here:
  https://sourceware.org/elfutils/Debuginfod.html
In a nutshell, by using a debuginfod service you will not need to
install debuginfo (a.k.a. dbgsym) files anymore; the symbols will be
served to GDB (or any other debuginfo consumer that supports debuginfod)
over the network.  Ultimately, this makes the debugging experience much
smoother (I myself never remember the full URL of our debuginfo
repository when I need it).
If you follow the Debian project, you might know that I run their debuginfod service. In fact, the excerpt above was taken from the announcement I made last year, letting the Debian community know that the service was available. First stage With more and more GNU/Linux distributions offering a debuginfod service to their users, I strongly believe that Ubuntu cannot afford to stay out of this "party" anymore. Fortunately, I have a manager who not only agrees with me but also turned the right knobs in order to make this project one of my priorities for this development cycle. The deployment of this service will be made in stages. The first one, whose results are due to be announced in the upcoming weeks, encompasses indexing and serving all of the available debug symbols from the official Ubuntu repository. In other words, the service will serve everything from main, universe and multiverse, from every supported Ubuntu release out there. This initial (a.k.a. "alpha") stage will also allow us to have an estimate of how much the service is used, so that we can better determine the resources allocated to it. More down the road This is just the beginning. In the following cycles, I will be working on a few interesting projects to expand the scope of the service and make it even more useful for the broader Ubuntu community. To give you an idea, here is what is on my plate: As you can see, there's a lot to do. I am happy to be working on this project, and I hope it will be helpful and useful for the Ubuntu community.

14 June 2021

Sergio Durigan Junior: I am not on Freenode anymore

This is a quick public announcement to say that I am not on the Freenode IRC network anymore. My nickname (sergiodj), which was more than a decade old, has just been deleted (along with every other nickname and channel in that network) from their database today, 2021-06-14. For your safety, you should assume that everybody you knew at Freenode is not there either, even if you see their nicknames online. Do not trust without verifying. In fact, I would strongly encourage that you do not join Freenode anymore: their new policies are absolutely questionable and their disregard for their users is blatant. If you would like to chat with me, you can find me at OFTC (preferred) and Libera.

Next.