Search Results: "sylvestre"

29 January 2022

Sylvestre Ledru: An update on rust/coreutils

TLDR: we are making progress on the Rust implementation of the GNU coreutils. Well, it is an understatement to say my previous blog post interested many people. Many articles, blog posts and some podcasts talked about it! As we pushed coreutils 0.0.12 a few days ago and getting closer to the 10 000 stars on github, it is now time to give an update! This has brought a lot of new contributors to this project. Instead of 30 to 60 patches per month, we jumped to 400 to 472 patches every month. Similarly, we saw an increase in the number of contributors (20 to 50 per month from 3 to 8). Two new maintainers (Michael Debertol & Terts Diepraam) stepped in and have been doing a much better job than myself as reviewers now! As a silly metric, according to github, we had 5 561 clones of the repository over the last 2 weeks! The new contributors focused on: Closing the gap with GNU As far as I know, we are only missing stty (change and print terminal line settings) as a program. Thanks to some heroes, basenc, pr, chcon and runcon have been implemented. For example, for the two last programs, Koutheir Attouchi wrote new crates to manage SELinux properly. This crate has been used for some other utilities like cp, ls or id. Leveraging the GNU testsuite to test this implementation Because the GNU testsuite is excellent, we now have a proper CI using it to run the tests. It is pretty long on the Github action CI (almost two hours to run it) but it is an amazing improvement to the way we work. It was a joint work from a bunch of folks (James Robson, Roy Ivy III, etc). To achieve this, we also made it easier to run the GNU testsuite locally with the Rust implementation but also to ignore some tests or adjust some error messages (see build-gnu.sh and run-gnu-test.sh). Following a suggestion of Brian G, a colleague at Mozilla (he did the same for some Firefox major change), we are now collecting the history of fail/pass/error into a separate repository and generating a daily graph showing the evolution of regression. Evolution over time At this date, we have, with GNU/Coreutils 9.0:
Total 611 tests
Pass 214
Skip 84
Fail 298
Error 15
We are now automatically identifying new passing tests and regressions in the CI. For example:
Warning: Congrats! The gnu test tests/chmod/c-option is now passing!
<br />Warning: Congrats! The gnu test tests/chmod/silent is now passing!
<br />Warning: Congrats! The gnu test tests/chmod/umask-x is now passing!
<br />Error: GNU test failed: tests/du/long-from-unreadable. tests/du/long-from-unreadable is passing on 'master'. Maybe you have to rebase?
[...]
<br />Warning: Changes from master: PASS +4 / FAIL +0 / ERROR -4 / SKIP +0
This is also beneficial to GNU as, by implementing some options, Michael Debertol noticed some incorrect behaviors (with sort and cat) or an uninitialized variable (with chmod). Documentations Every day, we are generating the user documentation and of the internal coreutils. User documentation: https://uutils.github.io/coreutils-docs/user/ Example: ls or cp The internal documentation can be seen on: https://uutils.github.io/coreutils-docs/dev/uucore/
For example, the backup style is documented here: https://uutils.github.io/coreutils-docs/dev/uucore/backup_control/index.html More? Besides my work on Debian/Ubuntu, I have also noticed that more and more operating systems are starting to look at this: In parallel, https://github.com/uutils/findutils/, a rust dropped-in replacement for find, is getting more attention lately! Here, the graph showing the evolution of the program using the BFS testsuite (much better than GNU's). Evolution over time - BFS testsuite What is next?
  1. stty needs to be implemented
  2. Improve the GNU compatibility on key programs and reduce the gap
  3. Investigate how to reduce the size of the binaries
  4. Allow Debian and Ubuntu to switch by default without tricky manipulation
How to help? I have been maintaining a list of good first bugs for new comers in the repo! Don't hesitate to contribute, it is much easier than it seems and a terrific way to learn Rust!

9 March 2021

Sylvestre Ledru: Debian running on Rust coreutils

tldr: Rust/coreutils ( https://github.com/uutils/coreutils/ ) is now available in Debian, good enough to boot a Debian with GNOME, install the top 1000 packages, build Firefox, the Linux Kernel and LLVM/Clang. Even if I wrote more than 100 patches to achieve that, it will probably be a bumpy ride for many other use cases.
It is also a terrific project to learn Rust. See the list of good first bugs. Even if I see Rust code every day at Mozilla, I was looking for an actual personal project (i.e. this isn't a Mozilla project) to learn Rust during the various COVID lockdowns. I started contributing to the alternative Coreutils developed in Rust. The project aims at proposing a drop-in replacement of the C-based GNU Coreutils, and I wanted to evaluate if this could be used to run a regular Debian. Similar to what I have done with clang.debian.net a few years ago (rebuilding the Debian archive using clang instead of gcc). I expect that most of the readers know what is the Coreutils. It is a set of programs performing simple operations (copy/move file, change permissions/ownership, etc). Even if some commands are from the 70s, they are at the base of Linux, Unix and macOS. While different implementations can be found, they are trying to remain compatible in terms of arguments, options, etc. This implementation of Coreutils isn t different! If you want to learn more about the history of Unix, I recommend this great Corecursive podcast with Brian Kernighan. While a lot of people contributed to this project, much was left to be done: To start easy, I defined 4 goals for this work:
  1. Package Coreutils in Debian/Ubuntu
  2. Boot a Debian system with a Rust-based coreutils
  3. Install the top 1000 packages in Debian - including GNOME
  4. Build Firefox, the Linux Kernel and LLV/Clang

Packaging of Coreutils in Debian Packaging in Debian isn't a trivial or even simple task. It requires uploading independently all the dependencies in the archive. Rust, with its new ecosystem and small crates, is making this task significantly harder. The package is called rust-coreutils - https://tracker.debian.org/pkg/rust-coreutils For Debian/Ubuntu users, to have an idea of the complexity of packaging such applications, just run
debtree --build-dep rust-coreutils dot -Tsvg > coreutils.svg (should be around 1M). Since it isn't production ready, the rust-coreutils is installable in parallel with coreutils. This package does NOT replace the GNU/coreutils files (yet?), the new files are installed in /usr/lib/cargo/bin/. They can be used with: export PATH=/usr/lib/cargo/bin/:$PATH Or, uglier, overriding the files with the new ones.

Booting Debian with rust-coreutils To achieve this, because I knew I would likely break the image a few times, I created a new project to quickly install a full Debian with PXE and preseed. The project is available here: https://github.com/opencollab/qemu-debian-install-pxe-preseed/ A script to create the full qemu image: build_qemu_debian_image.sh A second script to boot on the newly created image: boot.sh Then, building and installing coreutils on the system (yeah, it is ugly - don t do that at home): apt install rust-coreutils
cd /usr/lib/cargo/bin/
for f in *; do
cp -f $f /usr/bin/
done
 First surprise, unlike the old init.d init system, as systemd is not relying on a series of scripts (it is mostly written in C), replacing the coreutils did not have an impact. Therefore, I didn't experience any issue during the boot process

Implementing missing options A significant number of problems could be easily identified as a lack of support for some options. Here is a list of most of the fixes I had to implement to make this plan work:

Different behavior Most of the programs behaved as expected. Here is a list of differences:
  • install doesn't support using /dev/null as source file
    Setting up libreoffice-common (1:6.1.5-3+deb10u6) ...
    install: error: install: cannot install /dev/null to /etc/apparmor.d/local/usr.lib.libreoffice.program.oosplash : the source path is not an existing regular file     A limitation of rust itself https://github.com/rust-lang/rust/issues/79390

Compile Firefox, Clang and the Linux Kernel Build systems can vary significantly one from the other. To verify their usage of coreutils, I built these three major projects

Firefox As Firefox relies mostly on Python as a build system, it went smoothly. I didn t encounter any issue. The only unrelated issue that I noticed working on it was apt-key was broken because the script relied on a buggy option of mktemp.

Linux Kernel I identified only two issues compared to GNU Coreutils:
  • The chown command on a non-existing symlink target doesn t fail on the GNU version, the Rust one was triggering an error.
    https://github.com/uutils/coreutils/pull/1694
  • Linux kernel
    ln -fsn ../../x86/boot/bzImage ./arch/x86_64/boot/bzImage
    ln: error: Unrecognized option: 'n'

LLVM/Clang The llvm toolchain relies on Cmake. Just like for Firefox, I didn t face any issue.

Comparing with GNU coreutils using its testsuite Recently, James Robson added a new test to run the GNU testsuite on the Rust/coreutils.

# TOTAL: 611
# PASS: 144
# SKIP: 86
# XFAIL: 0
# FAIL: 342
# XPASS: 0
# ERROR: 39
compared to 546 test passing with the GNU version. Even if a bunch of errors are just different outputs, it demonstrates that there is still a long road ahead.

Next steps & contribute First, we will need more motivated contributors to work on this project. Many features remain to be implemented, optimizations to be done (e.g. decreasing the memory usage), etc.
I started to create a list of good first bugs for newcomers: https://github.com/uutils/coreutils/issues?q=is%3Aissue+is%3Aopen+label%3A%22Good+first+bug%22
I will update this list of there is some interest for this project.
Helping improve the support of the GNU coreutils testsuite would be a huge step while being a great way to learn Rust! Then, once it is in a better state, we will be able to make it a reliable alternative in Debian/Ubuntu to the GNU/Coreutils. This might be also interesting for other folks who prefer a BSD license over a GPL.

2 June 2020

Sylvestre Ledru: Debian rebuild with clang 10 + some patches

Because of the lock-down in France and thanks to Lucas, I have been able to make some progress rebuilding Debian with clang instead of gcc.

TLDR Instead of patching clang itself, I used a different approach this time: patching Debian tools or implementing some workaround to mitigate an issue.
The percentage of packages failing drop from 4.5% to 3.6% (1400 packages to 1110 - on a total of 31014). I focused on two classes of issues:

Qmake As I have no intention to merge the patch upstream, I used a very dirty workaround. I overwrote the g++ qmake file by clang's:
https://salsa.debian.org/lucas/collab-qa-tools/-/blob/master/modes/clang10#L44-47 I dropped the number of this failure to 0, making some packages build flawlessly (example: qtcreator, chessx, fwbuilder, etc). However, some packages are still failing later and therefore increasing the number of failures in some other categories like link error. For example, qtads fails because of ordered comparison between pointer and zero or oscar fails on a -Werror,-Wdeprecated-copy error. Breaking the build later also highlighted some new classes of issues which didn't occur with clang < 10.
For example, warnings related to C++ range loop or implicit int float conversion (I fixed a bunch of them in Firefox) .

Symbol differences Historically, symbol management for C++ in Debian has been a pain. Russ Allbery wrote a blog post in 2012 explaining the situation. AFAIK, it hasn't changed much.
Once more, I took the dirty approach: if there new or missing symbols, don't fail the build.
The rational is the following: Packages in the Debian archive are supposed to build without any issue. If there is new or missing symbols, it is probably clang generating a different library but this library is very likely working as expected (and usable by a program compiled with g++ or clang). It is purely a different approach taken by the compiler developer. In order to mitigate this issue, before the build starts, I am modifying dpkg-gensymbols to transform the error into a warning.
So, the typical Debian error some new symbols appeared in the symbols file or some symbols or patterns disappeared in the symbols file will NOT fail the build. Unsurprisingly, all but one package (libktorrent) build. Even if I am pessimistic, I reported a bug on dpkg-dev to evaluate if we could improve dpkg-gensymbol not to fail on these cases.

Next steps The next offender is Imake.tmpl:2243:10: fatal error: ' X11 .rules' file not found with more than an hundred occurrences, reported upstream quite sometime ago. Then, the big issues are going to be much harder to fix as they are real issues/warnings (with -Werror) in the code of the packages. Example: -Wc++11-narrowing & Wreserved-user-defined-literal... The list is long.
I will probably work on that when llvm/clang 11 are in RC phase.

For maintainers & upstream Maintainer of Debian/Ubuntu packages? I am providing a list of failing packages per maintainer: https://clang.debian.net/maintainers.php
For upstream, it is also easy to test with clang. Usually, apt install clang && CC=clang CXX=clang++ <build step> is good enough.

Conclusion With these two changes, I have been able to fix about 290 packages. I think I will be able to get that down a bit more but we will soon reach a plateau as many warnings/issues will have to fix in the C/C++ code itself.

22 March 2020

Sylvestre Ledru: Some clang rebuild results (8.0.1, 9.0.1 & 10rc2)

As part of the LLVM release cycle, I am continuing rebuilding the Debian archive with clang instead of gcc to evaluate potential regressions. Processed results are available on the website: https://clang.debian.net/status.php - Now includes some fancy graphs to show the evolution
Raw logs are published on github: https://github.com/opencollab/clang.debian.net/tree/master/logs Since my last blog post on the subject (August 2017), Clang is more and more present in the tech ecosystem. It is now the compiler used to build Firefox and Chrome upstream binaries on all the supported architectures/operating systems. More architectures are supported, it has a new linker (lld), a new hybrid IR (MLIR), a lot of checkers in clang-tidy, cross-language linking with Rust, etc.

Results
Now, about Debian results, we rebuilt using 8.0.1, 9.0.1 and 10.0rc2. Results are pretty similar to what we had with previous versions: between 4 to 5% of packages are failing when gcc is replaced by clang.
Some clang rebuild results (8.0.1, 9.0.1 &amp; 10rc2)

Even if most of the software are still using gcc as compiler, we can see that clang has a positive effect on code quality. With many different kinds of errors and warnings found clang over the years, we noticed a steady decline of the number of errors. For example, the number of incorrect C/C++ main declarations has been decreasing years after years:
Some clang rebuild results (8.0.1, 9.0.1 &amp; 10rc2)

Errors found
The biggest offender is still the qmake changes which doesn't allow the used workaround (replacing /usr/bin/gcc by /usr/bin/clang) - about 250 errors. Most of these packages would probably compile fine with clang. More on the Qt bug tracker. The workaround proposed in the bug isn't applicable for us as we use the dropped-in replacement of the
The second error is still some differences in symbol generation. Unlike gcc, it seems that clang doesn't generate some symbols (or adds some). As a bunch of Debian packages are checking the list of symbols in the library (for ABI management), the build fails on purpose. For example, with libcec, the symbol _ZN10P8PLATFORM14CConditionImplD1Ev@Base 3.1.0 isn't generated anymore. I am not expecting this to be a big deal: the generated libraries probably works most of the time. More on C++ symbol management in Debian.

Current status
As previously said in a blog post, I don't think there is a strong intensive to go away from gcc for most of the Linux distributions. The big reason for BSD was the license (even if the move to the Apache 2 license wasn't received positively by some of them).
While the LLVM/clang ecosystem clearly won the tooling battle, as a C/C++ compiler, gcc is still an excellent compiler which supports more architecture and more languages.
In term of new warnings and checks, as the clang community moved the efforts in clang-tidy (which requires more complex tooling), out of the box, gcc provides a better experience (as example, see the Firefox meta bug to build with -Werror with the default warnings using gcc 9, gcc 10 and clang trunk for example).

Next steps
I see some potential next steps to decrease the number of failure:
  • Workaround the Qt/Qmake issue
  • Fix the objective-c header include issues (echo "#include <objc/objc.h>" > foo.m && clang -c foo.m is currently failing)
  • Identify why clang generates more/less symbols that gcc in the library and try to fix that
  • Rebuild the archive with clang-7 - Seems that I have some data problem

Many thanks to Lucas Nussbaum for the rebuilds.
Original post blogged on b2evolution.

Sylvestre Ledru: Some clang rebuild results (8.0.1, 9.0.1 & 10rc2)

As part of the LLVM release cycle, I am continuing rebuilding the Debian archive with clang instead of gcc to evaluate potential regressions. Processed results are available on the website: https://clang.debian.net/status.php - Now includes some fancy graphs to show the evolution
Raw logs are published on github: https://github.com/opencollab/clang.debian.net/tree/master/logs Since my last blog post on the subject (August 2017), Clang is more and more present in the tech ecosystem. It is now the compiler used to build Firefox and Chrome upstream binaries on all the supported architectures/operating systems. More architectures are supported, it has a new linker (lld), a new hybrid IR (MLIR), a lot of checkers in clang-tidy, cross-language linking with Rust, etc.

Results
Now, about Debian results, we rebuilt using 8.0.1, 9.0.1 and 10.0rc2. Results are pretty similar to what we had with previous versions: between 4 to 5% of packages are failing when gcc is replaced by clang.
Some clang rebuild results (8.0.1, 9.0.1 &amp; 10rc2)

Even if most of the software are still using gcc as compiler, we can see that clang has a positive effect on code quality. With many different kinds of errors and warnings found clang over the years, we noticed a steady decline of the number of errors. For example, the number of incorrect C/C++ main declarations has been decreasing years after years:
Some clang rebuild results (8.0.1, 9.0.1 &amp; 10rc2)

Errors found
The biggest offender is still the qmake changes which doesn't allow the used workaround (replacing /usr/bin/gcc by /usr/bin/clang) - about 250 errors. Most of these packages would probably compile fine with clang. More on the Qt bug tracker. The workaround proposed in the bug isn't applicable for us as we use the dropped-in replacement of the compiler.
The second error is still some differences in symbol generation. Unlike gcc, it seems that clang doesn't generate some symbols (or adds some). As a bunch of Debian packages are checking the list of symbols in the library (for ABI management), the build fails on purpose. For example, with libcec, the symbol _ZN10P8PLATFORM14CConditionImplD1Ev@Base 3.1.0 isn't generated anymore. I am not expecting this to be a big deal: the generated libraries probably works most of the time. More on C++ symbol management in Debian.
I reported this bug upstream a while back: https://bugs.llvm.org/show_bug.cgi?id=30441

Current status
As previously said in a blog post, I don't think there is a strong intensive to go away from gcc for most of the Linux distributions. The big reason for BSD was the license (even if the move to the Apache 2 license wasn't received positively by some of them).
While the LLVM/clang ecosystem clearly won the tooling battle, as a C/C++ compiler, gcc is still an excellent compiler which supports more architecture and more languages.
In term of new warnings and checks, as the clang community moved the efforts in clang-tidy (which requires more complex tooling), out of the box, gcc provides a better experience (as example, see the Firefox meta bug to build with -Werror with the default warnings using gcc 9, gcc 10 and clang trunk for example).

Next steps
I see some potential next steps to decrease the number of failure:
  • Workaround the Qt/Qmake issue
  • Fix the objective-c header include issues (echo "#include <objc/objc.h>" > foo.m && clang -c foo.m is currently failing)
  • Identify why clang generates more/less symbols that gcc in the library and try to fix that
  • Rebuild the archive with clang-7 - Seems that I have some data problem

Many thanks to Lucas Nussbaum for the rebuilds.

23 August 2017

Sylvestre Ledru: Rebuild of Debian using Clang 3.9, 4.0 and 5.0

tldr: The percentage of failure is decreasing, Clang support is improving but there is a long way to go. The goal of this initiative is to rebuild Debian using Clang as a compiler instead of gcc. I have been doing this analysis for the last 6 years. Recently, we rebuilt the archive of the Debian archive with Clang 3.9.1 (July 6th), 4.0.1 (July 6th) and 5.0 rc2 (August 20th). For various reasons, we didn't perform a rebuild since June 2016 with version 3.8. Therefor, we took the opportunity to do three over the last month. Now, the 3.9 & 4.0 results are impacted by a build failure when building all haskell packages (the -no-pie option in Clang doesn't exist - I introduced it in clang 5.0). Fixing this issue with 5.0 removed more than 860 failures. Also, for the same versions, a Qt compiler detection is considering that Clang is not a C++11 compiler because clang++, by default, defines __cplusplus as 199711L (-std=c++11 has to be added to define a correct __cplusplus). See https://bugreports.qt.io/browse/QTBUG-62535 for more information. Some discussions happened on the upstream mailing list about changing the default C++ dialect.
For example, with 4.0, this is causing 132 errors. With 5.0, probably thanks to a new Qt version, roughly the same number of packages are failing but because gcc just triggers a warning with the "nodiscard" attribute being incorrectly used when clang triggers an error. In parallel, ignoring the haskell build failures, the numbers sightly increased since last year even if the overall percentage decreased (new packages being uploaded in the archive).
VersionBuild failuresIgnoring haskell pkgs
3.81367 / 5.6%
3.92274 / 8.1%1618 / 5.8%
4.02311 / 8.3%1655 / 5.9%
5.01445 / 5.1%
In parallel, new warnings and errors showed up in Clang.
This is causing a new set of build failures (especially with the usage of -Werror). As few examples:
* Starting with 4.0, clang triggers an error ordered comparison between pointer and zero ('char *' and 'int').
* Similarly, with this version, -Wmain introduces a new warning which will trigger a warning when a bool literal is returned from main.
* clang also introduced a new warning called -Waddress-of-packed-member causing 5 new errors.
* With the same version, clang can trigger a new error when auto is used in function return type. Now, as a conclusion, having Debian being built with clang by default is still a long shot.
First, when Clang became usable for a general audience, gcc was lagging in term of warning and error detections. Now, gcc is in a much better position than it was, decreasing the interest to have clang replacing gcc. In parallel, most of the efforts in term of warnings
and mistake detections are currently done under the clang tidy umbrella, making them less intrusive as part of this initiative (but harder to use and to deploy).
As an example, the gcc warning -Wmisleading-indentation has been implemented under a clang-tidy checker.
Second, the very permissive license of clang has been a key factor for some operating systems to switch like the PS4, Mac OS X or FreeBSD. With Debian, the community is generally happy with the GPL.
Third, the performances are similar enough that it is not worth the work, except for some projects with very special needs. Last, despite that it is much easier to contribute to llvm/clang than gcc (not copyright assignment or actual review system for example), this isn't a big differentiator for most of the projects. Of course, I will continue to run and analysis these rebuilds as this is a great source of information for clang upstream developers to improve the compatibility with gcc and understand some impacts. However, until there is a big game changer, I will stop pursuing the goal of having Debian switching to clang instead of gcc. I will stop effort on the debile project (which was aiming to rebuild in the background packages).
Original post blogged on b2evolution.

7 November 2016

Reproducible builds folks: Reproducible Builds: week 80 in Stretch cycle

What happened in the Reproducible Builds effort between Sunday October 30 and Saturday November 5 2016: Upcoming events Reproducible work in other projects Bugs filed Reviews of unreproducible packages 81 package reviews have been added, 14 have been updated and 43 have been removed in this week, adding to our knowledge about identified issues. 3 issue types have been updated: 1 issue type has been removed: 1 issue type has been updated: Weekly QA work During of reproducibility testing, some FTBFS bugs have been detected and reported by: diffoscope development buildinfo.debian.net development tests.reproducible-builds.org Reproducible Debian: Misc. Also with thanks to Profitbricks sponsoring the "hardware" resources, Holger created a 13 core machine with 24GB RAM and 100GB SSD based storage so that Ximin can do further tests and development on GCC and other software on a fast machine. This week's edition was written by Chris Lamb, Ximin Luo, Vagrant Cascadian, Holger Levsen and reviewed by a bunch of Reproducible Builds folks on IRC.

10 March 2016

Mike Hommey: RIP Iceweasel, 13 Nov 2006 10 Mar 2016

This took longer than it should have, but a page is now officially turned. I uploaded Firefox and Firefox ESR to Debian unstable. They will have to go through the Debian NEW queue because they are new source packages, so won t be immediately available, but they should arrive soon enough. People using Iceweasel from Debian unstable will be upgraded to Firefox ESR. Debian stable will receive Firefox ESR after Iceweasel/Firefox ESR38 is end-of-lifed, in about 3 months. Thanks go to Sylvestre Ledru, Mike Connor (the same who filed bug 354622) and Stefano Zacchiroli.

11 September 2014

Sylvestre Ledru: Rebuild of Debian using Clang 3.5.0

Clang 3.5.0 has just been released. A new rebuild has been done highlight the progress to get Debian built with clang. tl;dr: Great progress. We decreased from 9.5% to 5.7% of failures. Full results are available on http://clang.debian.net At time of the rebuild with 3.4.2, we had 2040 packages failing to build with clang. With 3.5.0, this dropped to 1261 packages. Fixes With Arthur Marble and Alexander Ovchinnikov, both GSoC students, we worked on various ways to decrease the number of errors. Upstream fixes First, the most obvious way, we fixed programming bugs/mistakes in upstream sources. Basically, we took categories of failure and fixed issues one after the other. We started with simple bugs like 'Wrong main declaration', 'non-void function should return a value' or 'Void function should not return a value'.

They are trivial to fix. We continued with harder fixes like ' Undefined reference' or 'Variable length array for a non POD (plain old data) element'.

So, besides these one, we worked on:
In total, we reported 295 bugs with patches. 85 of them have been fixed (meaning that the Debian maintainer uploaded a new version with the fix).

In parallel, I think that the switch by FreeBSD and Mac OS X to Clang also helped to fix various issues by upstreams. Hacking in clang As a parallel approach, we started to implement a suggestion from Linus Torvalds and a few others. Instead of trying to fix all upstream, where we can, we tried to update clang to improve the gcc compatibility.

gcc has many flags to disable or enable optimizations. Some of them are legacy, others have no sense in clang, etc. Instead of failing in clang with an error, we create a new category of warnings (showing optimization flag '%0' is not supported) and moved all relevant flags into it. Some examples, r212805, r213365, r214906 or r214907

We also updated clang to silent some useless arguments like -finput_charset=UTF-8 (r212110), clang being UTF-8 compliant.

Finally, we worked on the forwarding of linker flags. Clang and gcc have a very different behavior: when gcc does not know an argument, it is going to forward the argument to the linker. Clang, in this case, is going to reject the argument and fail with an error. In clang, we have to explicitly declare which arguments are going to be transfer to the linker. Of course, the correct way to pass arguments to the linker is to use -Xlinker or -Wl but the Debian rebuild proved that these shortcuts are used. Two of these arguments are now forwarded: New errors Just like in other releases, new warnings are added in clang. With (bad) usage of -Werror by upstream software, this causes new build failures: I also took the opportunity to add some further categorizations in the list of errors. Some examples: Next steps The Debile project being close to ready with Cl ment Schreiner's GSoC, we will now have an automatic and transparent way to rebuild packages using clang. Conclusion As stated, we can see a huge drop in term of number of failures over time:
Hopefully, Clang getting better and better, more and more projects adopting it as the default compiler or as a base for plugin/extension developments, this percentage will continue to decrease.
Having some kind of release goal with clang for Jessie+1 can now be considered as potentially reachable. Want to help? There are several things which can be done to help: Acknowledgments Thanks to David Suarez for the rebuilds of the archive, Arthur Marble and Alexander Ovchinnikov for their GSoC works and Nicolas S velin-Radiguet for the few fixes.
Original post blogged on b2evolution.

11 August 2014

Sylvestre Ledru: clang 3.4, 3.5 and 3.6 are now coinstallable in Debian

Clang is finally co installable on Debian. 3.4, 3.5 and the current trunk (snapshot) can be installed together. So, just like gcc, the different version can be called with clang-3.4, clang-3.5 or clang-3.6. /usr/bin/clang, /usr/bin/clang++, /usr/bin/scan-build and /usr/bin/scan-view are now handled through the llvm-defaults package. llvm-defaults is also now managing clang-check, clang-tblgen, c-index-test, clang-apply-replacements, clang-tidy, pp-trace and clang-query. Changes are also available on llvm.org/apt/.
The next step will be to manage also llvm-defaults on llvm.org/apt to simplify the transition for people using these packages. So, with: 
# /etc/apt/sources.list
deb http://llvm.org/apt/unstable/ llvm-toolchain main
deb http://llvm.org/apt/unstable/ llvm-toolchain-3.4 main
deb http://llvm.org/apt/unstable/ llvm-toolchain-3.5 main

$ apt-get install clang-3.4 clang-3.5 clang-3.6
$ clang-3.4 --version
Debian clang version 3.4.2 (branches/release_34) (based on LLVM 3.4.2)
Target: x86_64-pc-linux-gnu
Thread model: posix
$ clang-3.5 --version
Debian clang version 3.5.0-+rc2-1~exp1 (tags/RELEASE_350/rc2) (based on LLVM 3.5.0)
Target: x86_64-pc-linux-gnu
Thread model: posix
$ clang-3.6 --version
Debian clang version 3.6.0-svn214990-1~exp1 (trunk) (based on LLVM 3.6.0)
Target: x86_64-pc-linux-gnu
Thread model: posix
Original post blogged on b2evolution.

7 August 2014

Sylvestre Ledru: Debian Twitter accounts are back

After some downtime due to the identi.ca changes, the Debian Twitter accounts are now back. New Twitter feed ideas are welcome.
Original post blogged on b2evolution.

18 June 2014

Matthias Klumpp: Tanglu 2 (Bartholomea annulata) status update #1

bartholomea-annulata

Bartholomea annulata (c) Kevin Bryant

It is time for a new Tanglu update, which has been overdue for a long time now! Many things happened in Tanglu development, so here is just a short overview of what was done in the past months. Infrastructure Debile The whole Tanglu distribution is now built with Debile, replacing Jenkins, which was difficult to use for package building purposes (although Jenkins is great for other things). You can see the Tanglu builders in action at buildd.tg.o. The migration to Debile took a lot of time (a lot more than expected), and blocked the Bartholomea development at the beginning, but now it is working smoothly. Many thanks to all people who have been involved with making Debile work for Tanglu, especially Jon Severinsson. And of course many thanks to the Debile developers for helping with the integration, Sylvestre Ledru and of course Paul Tagliamonte. Archive Server Migration Those who read the tanglu-announce mailinglist know this already: We moved the main archive server stuff at archive.tg.o to to a new location, and to a very powerful machine. We also added some additional security measures to it, to prevent attacks. The previous machine is now being used for the bugtracker at bugs.tg.o and for some other things, including an archive mirror and the new Tanglu User Forums. See more about that below :-) Transitions There is huge ongoing work on package transitions. Take a look at our transition tracker and the staging migration log to get a taste of it. Merging with Debian Unstable is also going on right now, and we are working on merging some of the Tanglu changes which are useful for Debian as well (or which just reduce the diff to Tanglu) back to their upstream packages. Installer Work on the Tanglu Live-Installer, although badly needed, has not yet been started (it s a task ready for taking by anyone who likes to do it!) however, some awesome progress has been made in making the Debian-Installer work for Tanglu, which allows us to perform minimal installations of the Tanglu base systems and allows easier support of alternative Tanglu falvours. The work on d-i also uncovered a bug which appeared with the latest version of findutils, which has been reported upstream before Debian could run into it. This awesome progress was possible thanks to the work of Philip Mu kovac and Thomas Funk (in really hard debug sessions).
 Tanglu ForumsTanglu Users We finally have the long-awaited Tanglu user forums ready! As discussed in the last meeting, a popular demand on IRC and our mailing lists was a forum or Stackexchange-like service for users to communicate, since many people can work better with that than with mailinglists. Therefore, the new English TangluUsers forum is now ready at TangluUsers.org. The forum software is in an alpha version though, so we might experience some bugs which haven t been uncovered in the testing period. We will watch how the software performs and then decide if we stick to it or maybe switch to another one. But so far, we are really happy with the Misago Forums, and our usage of it already led to the inclusion of some patches against Misago. It also is actively maintained and has an active community. Misc Thingstanglu logo pure KDE We will ship with at least KDE Applications 4.13, maybe some 4.14 things as well (if we are lucky, since Tanglu will likely be in feature-freeze when this stuff is released). The other KDE parts will remain on their latest version from the 4.x series. For Tanglu 3, we might update KDE SC 4.x to KDE Frameworks 5 and use Plasma 5 though. GNOME Due to the lack manpower on the GNOME flavor, GNOME will ship in the same version available in Debian Sid maybe with some stuff pulled from Experimental, where it makes sense. A GNOME flavor is planned to be available. Common infrastructure We currently run with systemd 208, but a switch to 210 is planned. Tanglu 2 also targets the X.org server in version 1.16. For more changes, stay tuned. The kernel release for Bartholomea is also not yet decided. Artwork Work on the default Tanglu 2 design has started as well any artwork submissions are most welcome! Tanglu joins the OIN The Tanglu project is now a proud member (licensee) of the Open Invention Network (OIN), which build a pool of defensive patents to protect the Linux ecosystem from companies who are trying to use patents against Linux. Although the Tanglu community does not fully support the generally positive stance the OIN has about software patents, the OIN effort is very useful and we agree with it s goal. Therefore, Tanglu joined the OIN as licensee.
And that s the stuff for now! If you have further questions, just join us on #tanglu or #tanglu-devel on Freenode, or write to our newly created forum! You can, as always, also subscribe to our mailinglists to get in touch.

2 June 2014

Rapha&#235;l Hertzog: My Free Software Activities since January 2014

If you follow my blog closely, you noticed that I skipped all my usual monthly summaries in 2014. It s not that I stopped doing free software work, instead I was just too busy to be able to report about what I did. As an excuse, let me tell you that we just moved into a new house which was in construction since may last year. The lack of visible activity on my blog resulted in a steady decrease of the amount of donations received (January: 70.72 , February: 71.75 , March: 51.25 , April: 39.9 , May: 40.33 ). Special thanks to all the people who kept supporting my work even though I stopped reporting about it. So let s fix this. This report will be a bit less detailed since it covers the whole period since the start of the year. Debian France Preparations related to general assemblies. The year started with lots of work related to Debian France. First I took care of setting up limesurvey with Alexandre Delano to handle the vote to pick our new logo:
The new logo of Debian France I also helped Sylvestre Ledru to finalize and close the accounting books for 2013 in preparation for the general assembly that was due later in the month. I wrote the moral report of the president to be presented to the assembly. And last step, I collected vote mandates to ensure that we were going to meet the quorum for the extraordinary assembly that was planned just after the usual yearly assembly. The assemblies took place during a two days mini-debconf in Paris (January 17-18) where I was obviously present even though I gave no talk besides announcing the logo contest winner and thanking people for their participation.
Assembl e g n rale 2014 de Debian France

The Debian France members during the general assembly

It s worth noting that the extraordinary assembly was meant primarily to enshrine in our bylaws the possibility to act as a trusted organization for Debian. This status should be officialized by the Debian project leader (Lucas Nussbaum) in the upcoming weeks since we answered satisfactorily to all questions. Our paypal donation form and the accounting tools behind it are ready. Galette packaging and members map. I managed to hand over the package maintenance of galette to Fran ois-R gis Vuillemin. I sponsored all his uploads and we packaged a new plugin that allows to create a map with all the members who accept to share their location. The idea was to let people meet each other when they don t live far away with the long term goal to have Debian France organized activities not only in Paris but everywhere in France. New contributor game. Last but not least, I organized a game to encourage people to do their first contribution to Debian by offering them a copy of my book if they managed to complete a small Debian project. We got many interesting projects but the result so far seem to be very mixed. Many people did not complete their project (yet) that said for the few that did substantial work, it was rather good and they seem to be interested to continue to contribute. Debian France booth at Solutions Linux in Paris. Like each year, I spent two days in Paris to help man the Debian France booth at Solutions Linux. We had lots of goodies on sale and we made more than 2000 EUR in earnings during the two days. I also used this opportunity to try to convince companies to support the new Debian LTS effort.
Debian France booth at Solutions Linux

Tanguy Ortolo and Fernando Lagrange behind the Debian France booth

The Debian Administrator s Handbook In the last days of 2013, we released the wheezy update of the book. Then I quickly organized everything needed so that the various translation teams can now focus their efforts on the latest release of the book. Later (in February) I announced the availability of the French and Spanish translations. Debian Squeeze LTS When the security team called for help to try to put in place long term support for Squeeze, I replied positively because I m convinced that it s very important if Debian wants to stay an acceptable choice in big deployments and because I knew that some of my customers would be interested Thus I followed all the discussions (on a semi-private list first and then on debian-lts@lists.debian.org) and contributed my own experience. I have also taken up the responsibility to coordinate with the Debian contributors who can be hired to work on Squeeze LTS so that we have a clear common offer for all the companies who have offered financial support towards Squeeze LTS. Expect further news on this front in the upcoming days/weeks. Tryton I have been a long time user of SQL-Ledger to manage the accounting of my company Freexian. But while the license is free software, the project is not. It s the work of a single developer who doesn t really accept help. I have thus been considering to move to something else for a long time but never did anything. This year, after some rough evaluation, I decided to switch to Tryton for my company. It s probably not a wise choice from a business perspective because that migration took me many hours of unpaid labor but from a free software perspective it s definitely better than everything else I saw. I contributed a lot of bug reports and a few patches already (#3596, #3631, #3633, #3665, #3667, #3694, #3695, #3696, #3697) mainly about problems found in the French chart of accounts but also about missing features for my use case. I also accepted to sponsor Matthias Berhle, who is maintaining the official Debian packages of Tryton. He s already a Debian maintainer so it s mainly a matter of reviewing new source packages and granting him the required rights. Misc Debian work Thanks See you next month for a new summary of my activities.

2 comments Liked this article? Click here. My blog is Flattr-enabled.

21 March 2014

Sylvestre Ledru: Rebuild of Debian using Clang 3.4

Using the AWS donation, David Suarez and myself have been able to rebuild the whole archive with Clang 3.4.
The rebuild has been done January 10th but, with my new job, I did not find the time to publish the result.
Releases after releases, the results are getting better and better. Currently, 2193 packages in the archive are failing to build from source.
That is roughly the same number of build failures as with the precedent rebuild with Clang 3.3.
However, this is good news for two reasons:
* the number of new packages in Debian increased (18854 at time of the 3.3 release, 21204 for the 3.4)
* clang 3.4 has more checks and error detections. I also started to update clang to make it closer to gcc. For example, I transformed wrong usage of -O (> 6) error to be treated as regular warning. However, a critical bug has emerged during this release. When using -D_FORTIFY_SOURCE=2 (which is the case of many Debian packages), the binaries produced freeze (infinity loop in the ASM). This has been reported upstream as bug 16821 and concerns about 150 packages (including Firefox, gcc, LLVM, etc). Hopefully, this will be fixed in 3.5 (this is unlikely for 3.4.1). About the new build failures, now, Clang triggers an error (or warning + -Werror) on: * Wrong usage of the default argument (should be done in the definition)
16 occurrences * Usage of C++11 feature without the appropriate argument
7 occurrences * Unused static const declaration
5 occurrences * Recursive template instantiation exceeded
4 occurrences * Defitinion of a builtin function
3 occurrences * Read-only variable is not assignable
2 occurrences By the way, I proposed a Google Summer of Code Project to work faster on a support of Debian built by Clang. As requirements, I asked students to fix some bugs, they already did a great job.
Original post blogged on b2evolution.

22 February 2014

Sylvestre Ledru: Some updates on llvm.org/apt/

I made some changes on http://llvm.org/apt/ for the last 2 months.
Original post blogged on b2evolution.

11 February 2014

Sylvestre Ledru: Hello Planet Mozilla

Hello Planet Mozilla I started as a new employee a few weeks ago. I am working as Release Manager for the next releases of Firefox (both on desktop + android).
As a start, Lukas and I will be working on Firefox 29 (no pressure at all to start with Australis + Firefox account).
I grew up with the first versions of Netscape, followed the Phoenix/Firebird adventures and I have been a supporter of Firefox since then.
I am glad to join such an important project, especially in the perspective of the recent privacy scandals. I think that Mozilla is in the best position to communicate to the public, to denounce certain activities and provides technologies to bypass them. Besides that, let me introduce myself. I have been in free software for almost a decade.
My previous work as working on Scilab (from Inria to Scilab Enterprises) where I stayed for almost 8 years. Over the years, I worked on various development (C, C++, Java, Python, Scilab, etc), user experience, roadmap definition, release management, R and D / customer project management, management, board member, etc. During this period, I learnt tons of things from coding, release and product management, QA, etc. In parallel, I worked part time at IRILL, a free software research center (hosted by Inria). I was involved as community management ie organizing event like GNU Hacker meeting, FOSDEM, Euro LLVM, etc and recording them. I had the chance to meet plenty of people from the FLOSS world (including folks from Mozilla who suggested me to join the adventure). I am also involved in two communities: Debian and LLVM. The first one is Debian. I started to contribute in Debian to get Scilab properly packaged. 7 years
after, I am now maintaining
important packages like LLVM, Clang, libedit, emscripten (!) or BLAS/LAPACK, treasurer of the French representative association and organizing the French mini Debian Conference (Debconf) / Meetup. For the last couple of years, I contributed to LLVM, Clang and LLDB. I participated to the release 3.3 and 3.4. My technical contributions were mainly driven by the packaging, the nightly Debian/Ubuntu packages and also by the rebuild of the Debian archive using Clang (instead of gcc). And when I am not behind my laptop, I am teaching scuba diving with my girlfriend or hiking, mountaineering, ski touring, climbing, etc. Yeh, I am a bit hyperactive.
Original post blogged on b2evolution.

17 January 2014

Sylvestre Ledru: Debian & LLVM events

Being a bit hyperactive, I have been involved in the organization of two events. I am the main organizer with Alexandre Delano of the Mini Debconf 2014 in Paris, January 18 & 19th. The (great) planning is available here:
https://france.debian.net/events/minidebconf2014/
Saturday morning presentations will be general public, the beginning of Saturday afternoon will be used by the Debian France association to vote the new status (1901 law and Debian Trusted Organization).
Sunday will be more focused on Debian itself.
During the week end, I will be talking about the Debile project, the finance of Debian France and be part of the round table on compiler selection for Debian.
The (mandatory) registration should be done on the Wiki or meetup.com In parallel, with Tobias Grosser, we organized the LLVM devroom track at FOSDEM (Bruxelles), February 2nd (Sunday).
The schedule is a mix between core developers, third party software using LLVM / Clang and academic users.
https://fosdem.org/2014/schedule/track/llvm/
I will be talking on how to become a LLVM contributor. Both events should be recorded.
Original post blogged on b2evolution.

15 January 2014

Sylvestre Ledru: Debian France shop

During the month of December, we have been working with folks from En Vente Libre to create a shop with Debian products: http://www.enventelibre.org/debian We have now a shop with plenty of Debian products. The design has been done by Ahmet DEMIR. Most of them are already known by the community (they were sold during FOSDEM or various Debconf) like the Buff, Swiss Army knife or the classical polos. But, t-shirts based on Wheezy theme are brand new:
They are available in three colors:
* Red
* Blue
* Grey For now, the website is only French, an English version might come in the future.
Original post blogged on b2evolution.

Sylvestre Ledru: Some thoughts concerning LLVM & Clang and their evolutions: Release of LLVM & Clang 3.4

We just released LLVM & Clang 3.4 (already available in Debian Jessie and Ubuntu Trusty). The clang new release introduces some more warning and error detections. The point of the blog post is not to detail the changes (LLVM / Clang) but more to take a step back on the LLVM and Clang toolchains, usages and dissemination. For the last couple years, I have been presenting our work on the Debian side to make Clang part of the Debian infrastructure. I add the chance to speak at many events (Debconf 12 & 13, FOSDEM 2013, Linux Plumbers 2013, Distro recipes, etc). I even had Linus Torvalds attending to one of these talks (and proposing new approach). I had many feedback from the Debian/Ubuntu communities but also from other communities.
More and more projects are using Clang as part of their workflows, in term of development, production and QA. For example, some of them are running continuous integration using clang instead of gcc, others are running scan-build (Wireshark or LLVM itself) or Address Sanitizer (asan) (Firefox for example), etc.
More and more actors are moving from gcc to clang for their developments (Sony with their PS4, Chrome under Mac OS X, Apple, the Linux Kernel with the LLVMLinux project, FreeBSD, etc).
I stopped counting the number of times when people answered me "Yeh, I am now using Clang" when I said I am involved upstream. In parallel, more and more projects are using LLVM as a backend or computing engine and, in the meantime, getting more and more attraction (not because they are based on LLVM but just because they are great projects). For example, Mesa (llvmpipe), Julia, Rust, Emscripten or Native Client (NaCL) are excellent examples of this. The two last are excellent proofs that LLVM based technologies have an impact which goes further than geek-tools and will be used on a daily basis by millions of users. I also saw an interesting and growing number of new projects using libclang and libTooling. My feeling is that the lack of plugin capabilities in gcc for a long time and, now, its current complexity limited the innovation in term of compilation capabilities, analysis and fun tools. The features and the quality of this library are enabling new developments and I am sure we will see more and more excellent tools based on this library during 2014.
For example, the LLVM toolchain ships Clang modernize and clang format and some cool projects like include-what-you-use, DXR, OCLint or creduce are emerging. Thanks to this great competition, gcc is also improving (ASAN, JIT, better warnings, etc), remaining still relevant as before. No doubt that 2014 will be a great year for compilers.
Original post blogged on b2evolution.

6 January 2014

Rapha&#235;l Hertzog: My Free Software Activities in December 2013

This is my monthly summary of my free software related activities. If you re among the people who made a donation to support my work (147.56 , thanks everybody!), then you can learn how I spent your money. Otherwise it s just an interesting status update on my various projects. The Debian Administrator s Handbook Cover of the Debian Administrator's Handbook (Wheezy edition)I spent a good chunk of December on the book. First finalizing the English version and getting it out (BTW, just for the launch, there s a 10% discount on the paperback that lasts only until January 9th!). Then working on updating the French translation. Eyrolles will publish a new edition of the French book based on this translation. Expect some further news about this during January! Debian France I contributed to many discussions within Debian France. Starting with a complaint that most events are organized in Paris, I proposed to map the location of Debian France members. We added new fields in the membership management page so that members can add their GPS coordinates and Fr d ric Decou made some experiments with Openstreetmap. Someone else (Kiriarat) volunteered to write the required glue code. A manual map is currently maintained on the website. Sample logo receivedIn the discussions about the setup of the Debian France shop, I suggested to update our logo with a nicer looking one. We got a few suggestions and after further discussions with Alexandre Delano and Sylvestre Ledru, we organized a small contest to entice designers to submit a logo proposal to us (the winner earns a set of Debian goodies). We got 46 proposals (see my favorite on the right)! The board is currently pre-selecting the logos and setting up the final vote for our members. The winner shall be announced at the end of the upcoming mini-debconf in Paris. I also continued the work to finalize new bylaws and new internal rules. They shall be adopted during the next general assembly which will happen during the mini-debconf. Misc Debian Work WordPress maintenance. I mentored Pablo Vasquez to do his first small contribution to the WordPress packaging. I really appreciate this but he s not yet ready to assume maintenance of a big package like WordPress on his own. I got multiple other offers of help and pinged them all while filing #733726 to coordinate the work on the new upstream version. But I got no reply :-( Handing over packages to new maintainers is hard Init system discussion. The technical committee has the hard task of picking the default init system that will replace the traditional System V init (see #727708). I followed this huge discussion closely and contributed a bit where I add something meaningful to say. Final decision is expected sometimes in January. FWIW, I share entirely Russ Allbery s point of view in those discussions. I have been running systemd on some of my computers for a few months already. Fixing lxc in stable. The lxc package in stable has a non-working debian template. I really dislike documenting that things are broken so instead of doing that in the Debian Administrator s Handbook, I opted to do something about it. I prepared a non maintainer upload for stable (see #680469 for the problem and #732358 for the stable update request). Misc stuff. I sponsored a tcpdf upload. I filed an enhancement request on Publican to have it keep processing instructions present in translations. I uploaded new versions of publican-librement and debian-handbook. I filed #732678 against git-buildpackage because it failed to properly call lintian when given the -A dpkg-buildpackage argument. Thanks See you next month for a new summary of my activities.

No comment Liked this article? Click here. My blog is Flattr-enabled.

Next.