Float/String Conversion in Picolibc: Enter Ry
I
recently wrote about this topic
having concluded that the best route for now was to use the
malloc-free, but imprecise, conversion routines in the tinystdio alternative.
A few days later, Sreepathi Pai pointed me at some very recent work in this area:
This is amazing! Thirty years after the papers referenced in the
previous post, Ulf Adams came up with some really cool ideas and
managed to reduce the math required for 64-bit conversion to 128 bit
integers. This is a huge leap forward; we were doing long
multi-precision computations before, and now it's all short enough to
fit in registers (ok, a
lot of registers, but still).
Getting the Ry Code
The code is available on github:
https://github.com/ulfjack/ryu. Reading
through it, it's very clear that the author focuses on performance
with lots of tuning for common cases. Still, it's quite readable,
especially compared with the newlib multi-precision based code.
Picolibc String/Float conversion interface
Picolibc has some pretty basic needs for the float/string conversion code, it wants four functions:
- __dtoa_engine
int
__dtoa_engine(double x, struct dtoa *dtoa, uint8_t max_digits, uint8_t max_decimals);
This converts the double x to a string of decimal digits and a
decimal exponent stored inside the 'dtoa' struct. It limits the
total number of digits to max_digits and, optionally (when
max_decimals is non-zero), limits the number of fractional digits
to max_decimals - 1. This latter supports 'f' formats. Returns
the number of digits stored, which is <= max_digits. Less if the
number can be accurately represented in fewer digits.
- __ftoa_engine
int
__ftoa_engine(float x, struct ftoa *ftoa, uint8_t max_digits, uint8_t max_decimals);
The same as __dtoa_engine, except for floats.
- __atod_engine
double
__atod_engine(uint64_t m10, int e10);
To avoid needing to handle stdio inside the conversion function,
__atod_engine receives fully parsed values, the base-10
significand (m10) and exponent (e10). The value to convert is
m10 * pow(10, e10).
- __atof_engine
float
__atof_engine(uint32_t m10, int e10);
The same as __atod_engine, except for floats.
With these, it can do printf, scanf, ecvt, fcvt, gcvt, strtod, strtof
and atof.
Porting Ry to Picolibc
The existing Ry float-to-string code always generates the number of
digits necessary for accurate output. I had to hack it up to generate
correctly rounded shorter output when
max_digits or
max_decimals
were smaller. I'm not sure I managed to do that correctly, but at
least it appears to be passing all of the test cases I have. In normal
operation, Ry iteratively removes digits from the answer that aren't
necessary to disambiguate with neighboring values.
What I changed was to keep removing digits using that method until the
answer had few enough digits to fit in the desired length. There's
some tricky rounding code that adjusts the final result and I had to
bypass that if I'd removed extra digits.
That was about the only change necessary to the core algorithm. I also
trimmed the code to only include the general case and not the
performance improvements, then wrapped it with code to provide the
_engine interface.
On the string-to-float side, most of what I needed to do was remove
the string parsing bits at the start of the function and switch from
performance-optimized to space-optimized versions of a couple of
internal routines.
Correctness Results
Because these new functions are now 'exact', I was able to adjust the
picolibc tests to compare all of the bits for string/float conversion
instead of having to permit a bit of slop in the answers. With those
changes, the picolibc test suite passes, which offers some assurance
that things aren't completely broken.
Size Results
Snek uses the 32-bit float versions of the conversion routines, and
for that, the size difference is:
text data bss dec hex filename
59068 44 37968 97080 17b38 snek-qemu-riscv-orig.elf
59430 44 37968 97442 17ca2 snek-qemu-riscv-ryu.elf
362
362 bytes added to gain accurate printf/strtof results seems like a
good trade-off in this case.
Performance
I haven't measured performance at all, but I suspect that it won't be
nearly as problematic on most platforms as the source code makes it
appear. And that's because Ry is entirely integer arithmetic with no
floating point at all. This avoids using the soft fp code for
platforms without hardware float support.
Pointers to the Code
I haven't merged this to picolibc master yet, it's on the ryu branch:
Review, especially of the hack above to return short results, would be
greatly appreciated!
Thanks again to Ulf Adams for creating this code and to Sreepathi Pai
for sending me a note about it!
For about two months I've been working on a new project, debplate,
which currently lives
at benh/debplate
on Salsa. This is a template system for Debian packages, primarily
intended to ease building multiple similar binary packages from a
single source. With some changes, it could also be useful for
making multiple source packages consistent
(issue
#9).
I want debplate to be capable of replacing the kernel team's
existing template system and a lot of its custom scripting, but it
is also meant to a general tool. I believe it's already capable of
supporting source packages with relatively simple needs, and there
are some examples of these in the debplate source. My long-term
goal is that debplate will replace most team-specific and
package-specific template systems, making those source packages
using it less unusual and easier to contribute to.
I gave
a short
talk about debplate at MiniDebConf Online on Sunday.
At work we have a wonderful pyhon tool that we are able to send CoAP messages to and from our products. Perfect for development work. However recently I needed to install a copy of the tools onto my personal laptop because the only work laptops I have access to have completely dead batteries and so are not suitable for taking out into a field to perform RF range tests .
The eleventh release of 
I have been quite absent from Debian stuff lately, but this increased since COVID-19 hits us. In this blog post I'll try to sketch what I have been doing to help fight COVID-19 this last few months.
Discussion is progressing on funding & governance of larger LTS-related projects. Who should decide: contributors, Freexian, sponsors? Do we fund with a percentage or by capping resources allocated on security updates? I voiced concerns over funding these at the expense of smaller, more organic, more recurrent tasks that are less easy to specify but greatly contribute to the overall quality nevertheless.
ELTS - Wheezy
I would say that this was a crazy month, but with everything ever escalating, does that even mean anything anymore?
I lost track of tracking my activities in the second half of the month, and I m still not very good at logging the soft stuff , that is, things like non-technical work but that also takes up a lot of time, but will continue to work on it.
In May 2020, I have worked on the Debian LTS project for 14.5 hours (of 14.5 hours planned).
LTS Work
A Quick Recap from last year:
Kotlin is being packaged under the Google Summer of Code within the Debian organization itself. The major reason behind bringing Kotlin in Debian is to update all the Android packages which are now heavily dependent upon the Kotlin libraries.
The major work to bring Kotlin into Debian is done for the version 1.3.30, by Saif Abdul Cassim (goes by m36 on IRC) as a part of his GSoC'2019.
All his contributions to the team can be found in 
Tired of staying inside in Tokyo.
Here is my monthly update covering what I have been doing in the free software world during May 2020 (
