Testing Picolibc with the glibc tests
Picolibc has a bunch of built-in tests, but more testing is always
better, right? I decided to see how hard it would be to run some of
the tests provided in the GNU C Library (glibc).
Parallel meson build files
Similar to how Picolibc uses meson build files to avoid modifying the
newlib autotools infrastructure, I decided to take the glibc code and
write meson build rules that would compile the tests against Picolibc
header files and link against Picolibc libraries.
I decided to select a single target for this project so I could focus
on getting things building and not worry too much about making it
portable. I wanted to pick something that had hardware floating point
so that I would have rounding modes and exception support, so I picked
the ARM Cortex M7 with hard float ABI:
$ arm-none-eabi-gcc -mcpu=cortex-m7 -mfloat-abi=hard
It should be fairly easy to extend this to other targets, but for now,
that's what I've got working. There's a cross-cortex-m7.txt file
containing all of the cross compilation details which is used when
running
meson setup.
All of the Picolibc-specific files live in a new picolibc directory so
they are isolated from the main glibc code.
Pre-loading a pile of hacks
Adapt Picolibc to support the Glibc test code required a bunch of
random hacks, from providing
_unlocked versions of the stdio macros
to stubbing out various unsupportable syscalls (like
sleep and
chdir). Instead of modifying the Glibc code, I created a file called
hacks.h which is full of this stuff and used the gcc
-include
parameter to read that into the compiler before starting compilation
on all of the files.
Supporting command line parameters
The glibc tests all support a range of command line parameters, some
of which turned out to be quite useful for this work. Picolibc had
limited semihosting support for accessing the command line, but that
required modifying applications to go fetch the command line using a
special semihosting function.
To make this easier, I added a new crt0 variant for picolibc called
(oddly)
semihost. This extends the existing
hosted variant by
adding a call to the semihosting library to fetch the current command
line and splitting that into words at each space. It doesn't handle
any quoting, but it's sufficient for the needs here.
Avoiding glibc headers
The glibc tests use some glibc-specific extensions to the standard
POSIX C library, so I needed to include those in the test
builds. Headers for those extensions are mixed in with the rest of the
POSIX standard headers, which conflict with the Picolibc versions. To
work around this, I stuck stub #include files in the picolibc directory
which directly include the appropriate headers for the glibc API
extensions. This includes things like
argp.h and
array_length.h. For other headers which weren't actually needed for
picolibc, I created empty files.
Adding more POSIX to Picolibc
At this point, code was compiling but failing to find various standard
POSIX functions which aren't available in Picolibc. That included some
syscalls which could be emulated using semihosting, like
gettimeofday and
getpagesize. It also included some generally
useful additions, like replacing
ecvtbuf and
fcvtbuf with
ecvt_r
and
fcvt_r. The
_r variants provide a target buffer size instead
of assuming that it was large enough as the Picolibc
buf variants
did.
Which tests are working?
So far, I've got some of the tests in malloc, math, misc and
stdio-common running.
There are a lot of tests in the malloc directory which cover glibc API
extensions or require POSIX syscalls not supported by semihosting. I
think I've added all of the tests which should be supported.
For the math tests, I'm testing the standard POSIX math APIs in both
float and double forms, except for the Bessel and Gamma
functions. Picolibc's versions of those are so bad that they violate
some pretty basic assumptions about error bounds built into the glibc
test code. Until Picolibc gets better versions of these functions,
we'll have to skip testing them this way.
In the misc directory, I've only added tests for ecvt, fcvt, gcvt,
dirname and hsearch. I don't think there are any other tests
there which should work.
Finally, for stdio-common, almost all of the tests expect a fully
functioning file system, which semihosting really can't support. As a
result, we're only able to run the printf, scanf and some of the
sprintf tests.
All in all, we're running 78 of the glibc test programs, which is a
small fraction of the total tests, but I think it's the bulk of the
tests which cover APIs that don't depend too much on the underlying
POSIX operating system.
Bugs found and fixed in Picolibc
This exercise has resulted in 17 fixes in Picolibc, which can be
classified as:
- Math functions taking sNaN and not raising FE_INVALID and
returning qNaN. Almost any operation on an sNaN value is supposed
to signal an invalid operand and replace that with a qNaN so that
further processing doesn't raise another exception. This was
fairly easy to fix, just need to use
return x + x; instead of
return x;.
- Math functions failing to set errno. I'd recently restructured the
math library to get rid of the separate IEEE version of the
functions which didn't set errno and missed a couple of cases
that needed to use the errno-setting helper functions.
- Corner cases in string/float conversion, including the need to
perform round-to-even for '%a' conversions in printf and
supporting negative decimal values for fcvt. This particular
exercise led to replacing the ecvtbuf and fcvtbuf APIs with
glibc's ecvt_r and fcvt_r versions as those pass explicit
buffer lengths, making overflow prevention far more reliable.
- A bunch of malloc entry points were not handling failure
correctly; allocation values close to the maximum possible size
caused numerous numeric wrapping errors with the usual
consequences (allocations "succeed", but return a far smaller
buffer than requested). Also, realloc was failing to check the
return value from malloc before copying the old data, which really
isn't a good idea.
- Tinystdio's POSIX support code was returning an error instead of
EOF at end of file.
- Error bounds for the Picolibc math library aren't great; I had to
generate Picolibc-specific ulps files. Most functions are between
0 and 3 ulps, but for some reason, the float version of erfc
(ercf) has errors as large as 64 ulps. That needs investigation.
Tests added to Picolibc
With all of the fixes applied to Picolibc, I added some tests to
verify them without relying on running the glibc tests, that includes
sNaN vs qNaN tests for math functions, testing fopen and mktemp,
checking the printf %a rounding support and adding a ecvt/fcvt tests.
I also discovered that the github-based CI system was compiling but
not testing when using clang with a riscv target, so that got added
in.
Where's the source code?
The Picolibc changes are sitting on the
glibc-testing branch.
I'll merge them once the current CI pass finishes.
The hacked-up Glibc bits are in a glibc mirror at github in the
picolibc project on the
picolibc-testing branch.
It would be interesting to know what additional tests should be usable
in this environment. And, perhaps, finding a way to use this for
picolibc CI testing in the future.
Concluding thoughts
Overall, I'm pretty pleased with these results. The bugs in malloc
are fairly serious and could easily result in trouble, but the
remaining issues are mostly minor and shouldn't have a big impact on
applications using Picolibc.
I'll get the changes merged and start thinking about doing another
Picolibc release.
As mentioned in this post, most of my activity around Debian has come to a complete halt (Send your thanks to da-manager@debian.org!). Since I still have a few computers running Debian, I keep maintaining a few things which I heavily rely on, in particular KDE/Plasma and RSSguard.
For KDE/Plasma, there are the well known OBS repos I maintain, please read the blog posts tagged KDE.
For RSSguard, I just uploaded a package for the recently released RSSguard 4.2.0 to a new OBS project 
Reading 
A new major release of 
