Julian Andres Klode: Much faster incremental apt updates
APT s performance in applying the Pdiffs files, which are the diff format used for Packages, Sources, and other files in the archive has been slow.
Improving performance for uncompressed files
The reason for this is that our I/O is unbuffered, and we were reading one byte at a time in order to read lines. This changed on December 24, by adding read buffering for reading lines, vastly improving the performance of rred.
But it was still slow, so today I profiled using gperftools the rred method running on a 430MB uncompressed Contents file with a 75 KB large patch. I noticed that our ReadLine() method was calling some method which took a long time (google-pprof told me it was some _nss method, but that was wrong [thank you, addr2line]).
After some further look into the code, I noticed that we set the length of the buffer using the length of the line. And whenever we moved some data out of the buffer, we called 
Performance for patching compressed files
Contents files are usually compressed with gzip, and kept compressed locally because they are about 500 MB uncompressed and only 30MB compressed. I profiled this, and it turns out there is not much we can do about it: The majority of the time is spent inside zlib, mostly combining CRC checksums:
Going forward, I think a reasonable option might be to recompress Contents files using lzo they will be a bit bigger (50MB instead of 30MB), but lzo is about 6 times as fast (compressing a 430MB Contents file took 1 second instead of 6).
Filed under: Debian, Uncategorized
memmove() to move the remaining data to the front of the buffer.
So, I tried to use a fixed buffer size of 4096 (commit). Now memmove() would spend less time moving memory around inside the buffer. This helped a lot, bringing the run time on my example file down from 46 seconds to about 2 seconds.
Later on, I rewrote the code to not use memmove() at all opting for start and end variables instead; and increasing the start variable when reading from the buffer (commit).
This in turn further improved things, bringing it down to about 1.6 seconds. We could now increase the buffer size again, without any negative effect.
Effects on apt-get update
I measured the run-time of apt-get update, excluding appstream and apt-file files, for the update from todays 07:52 to the 13:52 dinstall run. Configured sources are unstable and experimental with amd64 and i386 architectures. appstream and apt-file indexes are disabled for testing, so only Packages and Sources indexes are fetched.
The results are impressive:
- For APT 1.1.6, updating with PDiffs enabled took 41 seconds.
- For APT 1.1.7, updating with PDiffs enabled took 4 seconds.
Performance for patching compressed files
Contents files are usually compressed with gzip, and kept compressed locally because they are about 500 MB uncompressed and only 30MB compressed. I profiled this, and it turns out there is not much we can do about it: The majority of the time is spent inside zlib, mostly combining CRC checksums:
Going forward, I think a reasonable option might be to recompress Contents files using lzo they will be a bit bigger (50MB instead of 30MB), but lzo is about 6 times as fast (compressing a 430MB Contents file took 1 second instead of 6).Filed under: Debian, Uncategorized
Kurt has trouble keeping his mouth shut. This became widely apparent
when he was expelled from Catholic school for telling a visiting
dignitary to go fuck himself.
WB used to preach angrily against casual sex.
Kurt enlisted in the Army, and due to his high IQ he ended up in the
Army Security Agency. After training, he was stationed in Germany
where he performed signals intelligence functions like direction
finding and passing on information to the CIA and NSA.
One day WB shagged a boy with a leather hat.
Contrary to what one might assume, Kurt had awareness of consequences.
He refused officer training because that would have extended his
commitment from 4 to 6 years. He purposefully flunked his French
language proficiency exam. He wanted there to be no chance of him
getting transferred to Vietnam.
The following day, WB found herself confronted and asked to explain
why she had had a change of heart about casual sex. Oh, it wasn't
casual sex, she explained. He came over and said, I've been dying
to make love with you for the past three hours, so it was special,
she clarified. Thus the nature of the past communication failure
became clear to everyone but WB.
Despite all his efforts to avoid Vietnam, Kurt, of course, had trouble
keeping his mouth shut. So in 1967 he pissed off the wrong person and
found himself having to choose between Leavenworth and Saigon. He
chose life of alcoholism over prison.
This week I'll be mostly doing drive-by bug-reporting.
As with last year we start by using the Debian Code Search, to look
for obviously broken patterns such as "
The Software Freedom Conservancy is currently running a 
What happened in the 
Portability toward other systems has been improved: old versions of GNU diff are now supported (Mike McQuaid), suggestion of the appropriate locale is now the more generic 
For the last week I have been stuck in England. For the vast majority of that time, I've had nothing to do except work on Debian and this blog post documents some of the things I worked on.
Obviously spending a whole week on Debian, there's going to be some packaging involved. The following packages got new versions in unstable this last week:
And to finish on a more close note:
After a short hiatus, a new version of the (wonderfully useful, if I dare say so ;-) 
QEMU 2.5 has just been released, with 
A while back, I wrote about 
If that's not dramatic enough for you (trust me, you'll notice it when
it's animated as you flicker through the two different fields), here's
an even more high-contrast example (same ordering):
I guess it's obvious in retrospect what happens; the HF filter picks up
residue from its outer edges, and even if the coefficient is just 0.031
(well, times two; it adds that value from both the previous and next field),
3% the photons of a fully lit pixel (which is what you get when working in
linear light) is actually quite a bit, whereas a 3% gray is only pixel value
8 or so, which is barely visible.
So what am I to make of this? I'm honestly not sure. Maybe it's somehow
related to that these filter values were chosen in 1988, where they were
relatively unlikely to do this in linear light (although if they did it
with analog circuitry, perhaps they could?) and it was tweaked to look good
despite doing the wrong thing. Or maybe I need to change my approach here
entirely.
It always sucks when your fundamental assumptions are challenged, but I
think it shows once again that if you notice something funny in your output,
you really ought to investigate, because you never know how deep the rabbit
hole goes. :-/
In my previous 
Thanks to my employer I had opportunity to spent two days just working on my packages.
