Search Results: "spa"

7 January 2016

Sven Hoexter: Failing with F5: stderr, stdout - who cares

[root@adc:Standby:In Sync] config # tmsh save /sys ucs /var/tmp/foo.ucs
Saving active configuration...
/var/tmp/foo.ucs is saved.
[root@adc:Standby:In Sync] config # tmsh save /sys ucs /var/tmp/foo.ucs > /dev/null
Saving active configuration...
[root@adc:Standby:In Sync] config # tmsh save /sys ucs /var/tmp/foo.ucs 2> /dev/null
/var/tmp/foo.ucs is saved.
[root@adc:Standby:In Sync] config #
Seems F5 is not alone with such glorious ideas. A coworker pointed out that the "ipspace list" command on our old NetApps outputs a space and a backspace in some places.

Enrico Zini: downgrading-network-manager

Downgrading network-manager This morning I woke up. Bad idea. I find in the work mail a compiler error that I cannot reproduce, so I need to log into a machine at work. But #809195. I decided to downgrade network-manager. I recall there was a tool to download packages from snapshots.debian.org, I discussed it recently on IRC, let's sync the IRC logs from my server. Or not (#810212). Never mind, I'll log into the server and grep. Ooh, it's debsnap. However, it doesn't quite do what I hoped (#667712). After some help from #debian-devel (thanks jcristau and LebedevRI), here is how to downgrade network-manager: 
# echo "deb http://snapshot.debian.org/archive/debian/20151125T155830Z/ sid main" >> /etc/apt/sources.list.d/tmp-downgrade-nm.list
# apt -o Acquire::Check-Valid-Until=false update
# apt -o Acquire::Check-Valid-Until=false install network-manager=1.0.6-1
# rm /etc/apt/sources.list.d/tmp-downgrade-nm.list
# service network-manager restart
And as user: 
$ killall nm-applet
$ nm-applet &
The yak is now nice and shaved, I can now go and see what those compiler errors are all about. Actually, no, there was still an unshaved patch on the yak, and now we have a debcya script.

Thorsten Glaser: git find published; test, review, fix it please

I just published the first version of git find on gh/mirabilos/git-find for easy collaboration. The repository deliberately only contains the script and the manual page so it can easily be merged into git.git with complete history later, should they accept it. git find is MirOS licenced. It does require a recent mksh (Update: I did start it in POSIX sh first, but it eventually turned out to require arrays, and I don t know perl(1) and am not going to rewrite it in C) and some common utility extensions to deal with NUL-separated lines (sort -z, grep -z, git ls-tree -z); also, support for '\0' in tr(1) and a comm(1) that does not choke on embedded NULs in lines.

To install or uninstall it, run 

	$ git clone git@github.com:mirabilos/git-find.git
	$ cd git-find
	$ sudo ln -sf $PWD/git-find /usr/lib/git-core/
	$ sudo cp git-find.1 /usr/local/share/man/man1/
	  hack  
	$ sudo rm /usr/lib/git-core/git-find \
	    /usr/local/share/man/man1/git-find.1

then you can call it as git find and look at the documentation with git help find , as is customary. The idea behind this utility is to have a tool like git grep that acts on the list of files known to git (and not e.g. ignored files) to quickly search for, say, all PNG files in the repository (but not the generated ones). git find acts on the index for the HEAD, i.e. whatever commit is currently checked-out (unlike git grep which also knows about git add ed files; fix welcome) and then offers a filter syntax similar to find(1) to follow up: parenthes s, ! for negation, -a and -o for boolean are supported, as well as -name, -regex and -wholename and their case-insensitive variants, although regex uses grep(1) without (or, if the global option -E is given, with) -E, and the pattern matches use mksh(1) s, which ignores the locale and doesn t do [[:alpha:]] character classes yet. On the plus side, the output is guaranteed to be sorted; on the minus side, it is rather wastefully using temporary files (under $TMPDIR of course, so use of tmpfs is recommended). -print0 is the only output option (-print being the default). Another mode forwards the file list to the system find; since it doesn t support DOS-style response files, this only works if the amount of files is smaller than the operating system s limit; this mode supports the full range (except -maxdepth) of the system find(1) filters, e.g. -mmin -1 and -ls, but it occurs filesystem access penalty for the entire tree and doesn t sort the output, but can do -ls or even -exec. The idea here is that it can collaboratively be improved, reviewed, fixed, etc. and then, should they agree, with the entire history, subtree-merged into git.git and shipped to the world. Part of the development was sponsored by tarent solutions GmbH, the rest and the entire manual page were done in my vacation.

6 January 2016

Steinar H. Gunderson: IPv6 non-alternatives: DJB's article, 13 years later

With the world passing 10% IPv6 penetration over the weekend, we see the same old debates coming up again; people claiming IPv6 will never happen (despite several years now of exponential growth!), and that if they had only designed it differently, it would have been all over by now. In particular, people like to point to a 2002 3 article by D. J. Bernstein, complete with rants about how Google would never set up useless IPv6 addresses (and then they did that in 2007 I was involved). It's difficult to understand exactly what the article proposes since it's heavy on calling people idiots and light on actual implementation details (as opposed to when DJB's gotten involved in other fields; e.g. thanks to him we now have elliptical curve crypto that doesn't suck, even if the reference implementation was sort of a pain to build), but I will try to go through it nevertheless and show how I cannot find any way it would work well in practice. One thing first, though: Sorry, guys, the ship has sailed. Whatever genius solution DJB may have thought up that I'm missing, and whatever IPv6's shortcomings (they're certainly there), IPv6 is what we have. By now, you can not expect anything else to arise and take over the momentum; we will either live with IPv6 or die with IPv4. So, let's see what DJB says. As far as I can see, his primary call is for a version of IPv6 where the address space is an extension of the IPv4 space. For sake of discussion, let's call that IPv4+ , although it would share a number of properties with IPv6. In particular, his proposal requires changing the OS and other software on every single end host out there, just as IPv6; he readily admits that and outlines how it's done in rough terms (change all structs, change all configuration files, change all databases, change all OS APIs, etc.). From what I can see, he also readily admits that IPv4 and IPv4+ hosts cannot talk to each other, or more clearly, we cannot start using the extended address space before almost everybody has IPv4+ capable software. (E.g., quote: Once these software upgrades have been done on practically every Internet computer, we'll have reached the magic moment: people can start relying on public IPv6 addresses as replacements for public IPv4 addresses. ) So, exactly how does the IPv4 address space fit into the IPv4+ address space? The article doesn't really say anything about this, but I can imagine only two strategies: Build the IPv4+ space around the IPv4 space (ie., the IPv4 space occupies a little corner of the IPv4+ space, similar to how v4-mapped addresses are used within software but not on the wire today, to let applications do unified treatment of IPv4 addresses as a sort of special IPv6 address), or build it as a hierarchical extension. Let's look at the former first; one IPv4 address gives you one IPv4+ addresses. Somehow this seems to give you all the disadvantages of IPv4 and all the disadvantages of IPv6. The ISP is not supposed to give you any more IPv4+ addresses (or at least DJB doesn't want to contact his ISP about more also saying that the fact that automatic address distribution does not change his argument), so if you have one, you're stuck with one. So you still need NAT. (DJB talks about proxies , but I guess that the way things evolved, this either actually means NAT, or it talks about the practice of application-level proxies such as Squid or SOCKS proxies to reach the Internet, which really isn't commonplace anymore, so I'll assume for the sake of discussion it means NAT.) However, we already do NAT. The IPv4 crunch happened despite ubiquitous NAT everywhere; we're actually pretty empty. So we will need to hand out IPv4+ addresses at the very least to new deployments, and also probably reconfigure every site that wants to expand and is out of IPv4 addresses. ( Site here could mean any organizational unit, such as if your neighborhood gets too many new subscribers for your ISP's local addressing scheme to have enough addresses for you.) A much more difficult problem is that we now need to route these addresses on the wire. Ironically, the least clear part of DJB's plan is step 1, saying we will extend the format of IP packets to allow 16-byte addresses ; how exactly will this happen? For this scheme, I can only assume some sort of IPv4 option that says the stuff in the dstaddr field is just the start and doesn't make sense as an IPv4 address on its own; here are the remaining 12 bytes to complete the IPv4+ address . But now your routers need to understand that format, so you cannot do with only upgrading the end hosts; you also need to upgrade every single router out there, not just the end hosts. (Note that many of these do routing in hardware, so you can't just upgrade the software and call it a day.) And until that's done, you're exactly in the same situation as with IPv4/IPv6 today; it's incompatible. I do believe this option is what DJB talks about. However, I fail to see exactly how it is much better than the IPv6 we got ourselves into; you still need to upgrade all software on the planet and all routers on the planet. The benefit is supposedly that a company or user that doesn't care can just keep doing nothing, but they do need to care, since they need to upgrade 100% of their stuff to understand IPv4+ before we can start even deploying it alongside IPv4 (in contrast with IPv6, where we now have lots of experience in running production networks). The single benefit is that they won't have to renumber until they need to grow, at which point they need to anyway. However, let me also discuss the other possible interpretation, namely that of the IPv4+ address space being an extension of IPv4, ie. if you have 1.2.3.4 in IPv4, you have 1.2.3.4.x.x.x.x or similar in IPv4+. (DJB's article mentions 128-bit addresses and not 64-bit, though; we'll get to that in a moment.) People keep bringing this up, too; it's occasionally been called BangIP (probably jokingly, as in this April Fool's joke) due to the similarity with how explicit mail routing would work before SMTP became commonplace. I'll use that name, even though others have been proposed. The main advantage of BangIP is that you can keep your Internet core routing infrastructure. One way or the other, they will keep seeing IPv4 addresses and IPv4 packets; you need no new peering arrangements etc.. The exact details are unclear, though; I've seen people suggest GRE tunneling, ignoring problems they have through NAT, and I've seen suggestions of IPv4 options for source/destination addresses, also ignoring that someting as innocious as setting the ECN bits has been known to break middleboxes left and right. But let's assume you can pull that off, because your middlebox will almost certainly need to be the point that decapsulates BangIP anyway and converts it to IPv4 on the inside, presumably with a 10.0.0.0/8 address space so that your internal routing can keep using IPv4 without an IPv4+ forklift upgrade. (Note that you now lose the supposed security benefit of NAT, by the way, although you could probably encrypt the address.) Of course, your hosts will need to support IPv4+ still, and you will need some way of communicating that you are on the inside of the BangIP boundary. And you will need to know what the inside is, so that when you communicate on this side, you'll send IPv4 and not IPv4+. (For a home network with no routing, you could probably even just do IPv4+ on the inside, although I can imagine complications.) But like I wrote above, experience has shown us that 32 extra bits isn't enough. One layer of NAT isn't doing it, we need two. You could imagine the inter-block routability of BangIP helping a fair bit here (e.g., a company with too many machines for 10.0.0.0/8 could probably easily get more addresses for more external IPv4 addresses, yielding 10.0.0.0/8 blocks), but ultimately, it is a problem that you chop the Internet off in two distinct halves that work very differently. My ISP will probably want to use BangIP for itself, meaning I'm on the outside of the core; how many of those extra bits will they allocate for me? Any at all? Having multiple levels of bang sounds like pain; effectively we're creating a variable-length address. Does anyone ever want that? From experience, when we're creating protocols with variable-length addresses, people just tend to use the maximum level anyway, so why not design it with 128-bit to begin with? (The original IP protocol proposals actually had variable-length addresses, by the way.) So we can create our 32/96 BangIP , where the first 32 bits are for the existing public Internet, and then every IPv4 address gives you a 2^96 addresses to play with. (In a sense, it reminds me of 6to4, which never worked very well and is now thankfully dead.) However, this makes the inside/outside-core problem even worse. I now need two very different wire protocols coexisting on the Internet; IPv4+ (which looks like regular IPv4 to the core) for the core, and a sort of IPv4+-for-the-outside (similar to IPv6) outside it. If I build a company network, I need to make sure all of my routers are IPv4+-for-the-outside and talk that, while if I build the Internet core, I need to make sure all of my connections are IPv4 since I have no guarantee that I will be routable on the Internet otherwise. Furthermore, I have a fixed prefix that I cannot really get out of, defined by my IPv4 address(es). This is called hierarchical routing , and the IPv6 world gave it up relatively early despite it sounding like a great idea at first, because it makes multihoming a complete pain: If I have an address 1.2.3.4 from ISP A and 5.6.7.8 from ISP B, which one do I use as the first 32 bits of my IPv4+ network if I want it routable on the public Internet? You could argue that the solution for me is to get an IPv4 PI netblock (supposedly a /24, since we're not changing the Internet core), but we're already out of those, which is why we started this thing to begin with. Furthermore, if the IPv4/IPv4+ boundary is above my immediate connection to the Internet (say, ISP A doesn't have an IPv4 address, just IPv4+), I'm pretty hosed; I cannot announce an IPv4 netblock in BGP. The fact that the Internet runs on largely the same protocol everywhere is a very nice thing; in contrast, what is described here really would be a mess! So, well. I honestly don't think it's as easy to just do extension instead of alternative when it comes to the address spaces. We'll just need to deal with the pain and realize that upgrading the equipment and software is the larger part of the job anyway, and we'll need to do that no matter what solution we go with. Congrats on reaching 10%! Now get to work with the remaining 90%.

Norbert Preining: TeX Live security improvements

Today I committed a set of changes to the TeX Live subversion repository that should pave the way for better security handling in the future. Work is underway to use strong cryptographic signatures to verify that packages downloaded and installed into a TeX Live installation have not been tinkered with. texlive2015 While there is still a long way to go and to figure out, the current changes already improve the situation considerably. Status up to now Although we did ship size and checksum information within the TeX Live database, these information were only considered by the installer when re-starting an installation to make sure that the downloaded packages are the ones we should use. Neither the installer nor tlmgr did use the checksum to verify that the downloaded packages is correct, relying mostly on the fact that the packages are xz-compressed and would create rubbish when there is a transfer error. Although none of us believes that there is a serious interest in tinkering with the TeX Live distribution maybe to steal just another boring scientific paper? the door was still open. Changes implemented The changes committed to the repository today, which will be in a testing phase to get rid of bugs, consists of the following: Together these two changes will allow install-tl/tlmgr to verify that a package (.tar.xz) is according to the information in the accompanying texlive.tlpdb. This still leaves the option of tampering with a package and updating the texlive.tlpdb for the fixes checksums/sizes. Upcoming changes For the future we do plan mostly two things: The last step above will give a very high level of security, as it will be not practically possible to alter the information in the texlive.tlpdb, thus no tampering with the checksum information of the containers, and in turn no tampering with the actual containers will be possible. Restrictions Due to the wide range of supported architectures and operating systems, we will not make verification obligatory, but if a gpg binary is found, it will be used to verify the downloaded texlive.tlpdb.checksum. Details have to be hammered out and the actual programming has to be done, but we are on the way. Enjoy.

5 January 2016

Lior Kaplan: PHP 5 Support Timeline

With the new year starting the PHP project is being asked to decide about the PHP 5 support timeline. While Aligning PHP 5.6 support timeline with the release date of PHP 7.0 seems like common sense to keep the support schedule continuous, there s a big question whether to extend it further to an additional one year of security support till the end of 2018. This would make PHP 5.6, the last of the PHP 5 branch, to have 2 years of security support and de facto getting the same life span as PHP 7.0 would (ending support of both in Dec 2018). But beside of support issues, this also affects adoption rate of PHP 7.0, serving as a catalyst due to end of support for the PHP 5 branch. My concerns are that with the additional security support the project would need to deal with the many branches (5.6, 7.0, 7.1, 7.2 and then to be 7.3 branch). I think we should limit what we guarantee (meaning keeping only one year of security support till end of 2017), and encourage project members and the eco-system (e.g. Linux distributions) to maintain further security based on best effort. This is already the case for out of official support releases like the 5.3 and 5.4 branches (examples for backports done by Debian: 5.3 and 5.4). And of course, we also have companies that make their money out of long term support (e.g. RedHat). On the other hand, we should help the eco system in doing such extended support, and hosting backported fixes in the project s git repo instead of having each Linux disto do the same patch work on its own.
Filed under: PHP

4 January 2016

John Goerzen: Hiking a mountain with Ian Murdock

Would you like to hike a mountain? That question caught me by surprise. It was early in 2000, and I had flown to Tucson for a job interview. Ian Murdock was starting a new company, Progeny, and I was being interviewed for their first hire. Well, I thought, hiking will be fun. So we rode a bus or something to the top of the mountain and then hiked down. Our hike was full of well, everything. Ian talked about Tucson and the mountains, about his time as the Debian project leader, about his college days. I asked about the plants and such we were walking past. We talked about the plans for Progeny, my background, how I might fit in. It was part interview, part hike, part two geeks chatting. Ian had no HR telling him you can t go hiking down a mountain with a job candidate, as I m sure HR would have. And I am glad of it, because even 16 years later, that is still by far the best time I ever had at a job interview, despite the fact that it ruined the only pair of shoes I had brought along I had foolishly brought dress shoes for a, well, job interview. I guess it worked, too, because I was hired. Ian wanted to start up the company in Indianapolis, so over the next little while there was the busy work of moving myself and setting up an office. I remember those early days Ian and I went computer shopping at a local shop more than once to get the first workstations and servers for the company. Somehow he had found a deal on some office space in a high-rent office building. I still remember the puzzlement on the faces of accountants and lawyers dressed up in suits riding in the elevators with us in our shorts and sandals, or tie-die, next to them. Progeny s story was to be a complicated one. We set out to rock the world. We didn t. We didn t set out to make lasting friendships, but we often did. We set out to accomplish great things, and we did some of that, too. We experienced a full range of emotions there elation when we got hardware auto-detection working well or when our downloads looked very popular, despair when our funding didn t come through as we had hoped, being lost when our strategy had to change multiple times. And, as is the case everywhere, none of us were perfect. I still remember the excitement after we published our first release on the Internet. Our little server that could got pegged at 100Mb of outbound bandwidth (that was something for a small company in those days.) The moment must have meant something, because I still have the mrtg chart from that day on my computer, 15 years later. Progeny's Bandwidth Chart We made a good Linux distribution, an excellent Debian derivative, but commercial success did not flow from it. In the succeeding months, Ian and the company tried hard to find a strategy that would stick and make our big break. But that never happened. We had several rounds of layoffs when hoped-for funding never materialized. Ian eventually lost control of the company, and despite a few years of Itanium contract work after I left, closed for good. Looking back, Progeny was life compressed. During the good times, we had joy, sense of accomplishment, a sense of purpose at doing something well that was worth doing. I had what was my dream job back then: working on Debian as I loved to do, making the world a better place through Free Software, and getting paid to do it. And during the bad times, different people at Progeny experienced anger, cynicism, apathy, sorrow for the loss of our friends or plans, or simply a feeling to soldier on. All of the emotions, good or bad, were warranted in their own way. Bruce Byfield, one of my co-workers at Progeny, recently wrote a wonderful memoriam of Ian. He wrote, More than anything, he wanted to repeat his accomplishment with Debian, and, naturally he wondered if he could live up to his own expectations of himself. That, I think, was Ian s personal tragedy that he had succeeded early in life, and nothing else he did with his life could quite measure up to his expectations and memories. Ian was not the only one to have some guilt over Progeny. I, for years, wondered if I should have done more for the company, could have saved things by doing something more, or different. But I always came back to the conclusion I had at the time: that there was nothing I could do a terribly sad realization. In the years since, I watched Ubuntu take the mantle of easy-to-install Debian derivative. I saw them reprise some of the ideas we had, and even some of our mistakes. But by that time, Progeny was so thoroughly forgotten that I doubt they even realized they were doing it. I had long looked at our work at Progeny as a failure. Our main goal was never accomplished, our big product never sold many copies, our company eventually shuttered, our rock-the-world plan crumpled and forgotten. And by those traditional measurements, you could say it was a failure. But I have come to learn in the years since that success is a lot more that those things. Success is also about finding meaning and purpose through our work. As a programmer, success is nailing that algorithm that lets the application scale 10x more than before, or solving that difficult problem. As a manager, success is helping team members thrive, watching pieces come together on projects that no one person could ever do themselves. And as a person, success comes from learning from our experiences, and especially our mistakes. As J. Michael Straczynski wrote in a Babylon 5 episode, loosely paraphrased: Maybe this experience will be a good lesson. Too bad it was so painful, but there ain t no other kind. The thing about Progeny is this Ian built a group of people that wanted to change the world for the better. We gave it our all. And there s nothing wrong with that. Progeny did change the world. As us Progeny alumni have scattered around the country, we benefit from the lessons we learned there. And many of us were different , sort of out of place before Progeny, and there we found others that loved C compilers, bootloaders, and GPL licenses just as much as we did. We belonged, not just online but in life, and we went on to pull confidence and skill out of our experience at Progeny and use them in all sorts of ways over the years. And so did Ian. Who could have imagined the founder of Debian and Progeny would one day lead the cause of an old-guard Unix turning Open Source? I run ZFS on my Debian system today, and Ian is partly responsible for that and his time at Progeny is too. So I can remember Ian, and Progeny, as a success. And I leave you with a photo of my best memento from the time there: an original unopened boxed copy of Progeny Linux. IMG_6197_v1

3 January 2016

Carl Chenet: To be alerted if no more tweet is sent from your Twitter account: Twitterwatch

This version 0.1 of Twitterwatch is dedicated to Ian Murdock, Debian project Founder. If you use an automated system like Feed2tweet in order to feed your Twitter account from your website of any RSS feed, it is of the utmost important to know when this system is broken, because you are losing traffic because of this malfunction. At the Journal du hacker, a French-speaking FOSS website where people contribute weblinks like Hacker News or Lobste.rs, every new contribution is sent through the RSS feed to the Twitter account of the Journal du hacker. A lot of our readers only follow the news on Twitter. So if the link breaks, our readers are not fed any more :) To be alerted to this issue, I wrote a small self hosted and documented app. I m proud to introduce Twitterwatch ! It runs with Python 3.4 and the Tweepy library. You can install it from PyPI or from sources. The documentation is available on Readthedocs. You can read about how to install it, configure it and use it. It s the first release but the main features do the job already. In order to use it, you need a twitterwatch.ini file as following: 
[twitter]
consumer_key=ml6jaiBnf3pkU9uIrKNIxAr3o
consumer_secret=4Cmljklzerkhfer4hlj3ljl2hfvc123rezrfsdatpokaelzerp
access_token=213416790-jgJnrJG5gz132nzerl5zerwi1ahmnwkfJFN9nr2j
access_token_secret=WjanlPgqDKlunJ4Knr11k2bnfk3jfnwkFjeriFZERj16Z
[schedule]
; duration between 2 checks in minutes
check_interval=60
[mail]
host=localhost
from=admin@myserver.org
to=foo@mylaptop.org
The most difficult part is now complete. To launch Twitterwatch, just launch the following command: 
$ twitterwatch /path/to/twitterwatch.ini
Whenever no more new tweet is sent from your Twitter account during 60 minutes, you ll receive the following mail (here the user is @journalduhacker): 
Title: Twitter stream of journalduhacker has stopped
Body: Twitter stream of journalduhacker has stopped since 23/12/2015 - 14:49:23
It is still pretty rough, but it does the job. New features will come whenever I need them (or if you require/open bugs/submit push requests ;) ). Write a comment to tell me what you think of it or why you use it!

Lunar: Reproducible builds: week 35 in Stretch cycle

What happened in the reproducible builds effort between December 20th to December 26th: Toolchain fixes Mattia Rizzolo rebased our experimental versions of debhelper (twice!) and dpkg on top of the latest releases. Reiner Herrmann submited a patch for mozilla-devscripts to sort the file list in generated preferences.js files. To be able to lift the restriction that packages must be built in the same path, translation support for the __FILE__ C pre-processor macro would also be required. Joerg Sonnenberger submitted a patch back in 2010 that would still be useful today. Chris Lamb started work on providing a deterministic mode for debootstrap. Packages fixed The following packages have become reproducible due to changes in their build dependencies: bouncycastle, cairo-dock-plug-ins, darktable, gshare, libgpod, pafy, ruby-redis-namespace, ruby-rouge, sparkleshare. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues, but not all of them: Patches submitted which have not made their way to the archive yet: reproducible.debian.net Statistics for package sets are now visible for the armhf architecture. (h01ger) The second build now has a longer timeout (18 hours) than the first build (12 hours). This should prevent wasting resources when a machine is loaded. (h01ger) Builds of Arch Linux packages are now done using a tmpfs. (h01ger) 200 GiB have been added to jenkins.debian.net (thanks to ProfitBricks!) to make room for new jobs. The current count is at 962 and growing! diffoscope development Aside from some minor bugs that have been fixed, a one-line change made huge memory (and time) savings as the output of transformation tool is now streamed line by line instead of loaded entirely in memory at once. disorderfs development Andrew Ayer released disorderfs version 0.4.2-1 on December 22th. It fixes a memory corruption error when processing command line arguments that could cause command line options to be ignored. Documentation update Many small improvements for the documentation on reproducible-builds.org sent by Georg Koppen were merged. Package reviews 666 (!) reviews have been removed, 189 added and 162 updated in the previous week. 151 new fail to build from source reports have been made by Chris West, Chris Lamb, Mattia Rizzolo, and Niko Tyni. New issues identified: unsorted_filelist_in_xul_ext_preferences, nondeterminstic_output_generated_by_moarvm. Misc. Steven Chamberlain drew our attention to one analysis of the Juniper ScreenOS Authentication Backdoor: Whilst this may have been added in source code, it was well-disguised in the disassembly and just 7 instructions long. I thought this was a good example of the current state-of-the-art, and why we'd like our binaries and eventually, installer and VM images reproducible IMHO. Joanna Rutkowska has mentioned possible ways for Qubes to become reproducible on their development mailing-list.

Niels Thykier: Tor enabled MTA

As I posted earlier, I have migrated to use tor on my machine. Though I had a couple of unsolved issues back then. One of them being my Mail Transport Agent (MTA) did not support tor. A regular user might not have a lot of use for a MTA on their laptop. However, it is needed for a lot of Debian development scripts (bts, mass-bug, nmudiff), if they are to file/manipulate bugs for you. I have some requirements for my MTA I also have some non-requirements: Originally, I used postfix, which supported most of these requirements. Except: Per suggestion of Jakub Wilk, I tried msmtp, which turned out do what I wanted. The only feature I will probably miss is having a local queue, which can be rate limited. But all in all, I am quite happy with it so far. :)
Filed under: Debian

Benjamin Mako Hill: Access Without Empowerment (LibrePlanet 2015 Keynote)

At LibrePlanet 2015 (the FSF s annual conference), I gave a talk called Access Without Empowerment as one of the conference keynote addresses. As I did for my 2013 LibrePlanet talk, I ve edited together a version that includes the slides and I ve posted it online in WebM and on YouTube.
Here s the summary written up in the LibrePlanet program:
The free software movement has twin goals: promoting access to software through users freedom to share, and empowering users by giving them control over their technology. For all our movement s success, we have been much more successful at the former. I will use data from free software and from several related movements to explain why promoting empowerment is systematically more difficult than promoting access and I will explore how our movement might address the second challenge in the future.
In related news, registration is open for LibrePlanet 2016 and that it s free for FSF members. If you re not an FSF member, the FSF annual fundraiser is currently going on so now would be a great time to join.

2 January 2016

Daniel Pocock: The great life of Ian Murdock and police brutality in context

Tributes: (You can Follow or Tweet about this blog on Twitter) Over the last week, people have been saying a lot about the wonderful life of Ian Murdock and his contributions to Debian and the world of free software. According to one news site, a San Francisco police officer, Grace Gatpandan, has been doing the opposite, starting a PR spin operation, leaking snippets of information about what may have happened during Ian's final 24 hours. Sadly, these things are now starting to be regurgitated without proper scrutiny by the mainstream press (note the erroneous reference to SFGate with link to SFBay.ca, this is British tabloid media at its best). The report talks about somebody (no suggestion that it was even Ian) "trying to break into a residence". Let's translate that from the spin-doctor-speak back to English: it is the silly season, when many people have a couple of extra drinks and do silly things like losing their keys. "a residence", or just their own home perhaps? Maybe some AirBNB guest arriving late to the irritation of annoyed neighbours? Doesn't the choice of words make the motive sound so much more sinister? Nobody knows the full story and nobody knows if this was Ian, so snippets of information like this are inappropriate, especially when somebody is deceased. Did they really mean to leave people with the impression that one of the greatest visionaries of the Linux world was also a cat burglar? That somebody who spent his life giving selflessly and generously for the benefit of the whole world (his legacy is far greater than Steve Jobs, as Debian comes with no strings attached) spends the Christmas weekend taking things from other people's houses in the dark of the night? The report doesn't mention any evidence of a break-in or any charges for breaking-in. If having a few drinks and losing your keys in December is such a sorry state to be in, many of us could potentially be framed in the same terms at some point in our lives. That is one of the reasons I feel so compelled to write this: somebody else could be going through exactly the same experience at the moment you are reading this. Any of us could end up facing an assault as unpleasant as the tweets imply at some point in the future. At least I can console myself that as a privileged white male, the risk to myself is much lower than for those with mental illness, the homeless, transgender, Muslim or black people but as the tweets suggest, it could be any of us. The story reports that officers didn't actually come across Ian breaking in to anything, they encountered him at a nearby street corner. If he had weapons or drugs or he was known to police that would have almost certainly been emphasized. Is it right to rush in and deprive somebody of their liberties without first giving them an opportunity to identify themselves and possibly confirm if they had a reason to be there? The report goes on, "he was belligerent", "he became violent", "banging his head" all by himself. How often do you see intelligent and successful people like Ian Murdock spontaneously harming themselves in that way? Can you find anything like that in any of the 4,390 Ian Murdock videos on YouTube? How much more frequently do you see reports that somebody "banged their head", all by themselves of course, during some encounter with law enforcement? Do police never make mistakes like other human beings? If any person was genuinely trying to spontaneously inflict a head injury on himself, as the police have suggested, why wouldn't the police leave them in the hospital or other suitable care? Do they really think that when people are displaying signs of self-harm, rounding them up and taking them to jail will be in their best interests? Now, I'm not suggesting this started out with some sort of conspiracy. Police may have been at the end of a long shift (and it is a disgrace that many US police are not paid for their overtime) or just had a rough experience with somebody far more sinister. On the other hand, there may have been a mistake, gaps in police training or an inappropriate use of a procedure that is not always justified, like a strip search, that causes profound suffering for many victims. A select number of US police forces have been shamed around the world for a series of incidents of extreme violence in recent times, including the death of Michael Brown in Ferguson, shooting Walter Scott in the back, death of Freddie Gray in Baltimore and the attempts of Chicago's police to run an on-shore version of Guantanamo Bay. Beyond those highly violent incidents, the world has also seen the abuse of Ahmed Mohamed, the Muslim schoolboy arrested for his interest in electronics and in 2013, the suicide of Aaron Swartz which appears to be a direct consequence of the "Justice" department's obsession with him. What have the police learned from all this bad publicity? Are they changing their methods, or just hiring more spin doctors? If that is their response, then doesn't it leave them with a cruel advantage over those people who were deceased? Isn't it standard practice for some police to simply round up anybody who is a bit lost and write up a charge sheet for resisting arrest or assaulting an officer as insurance against questions about their own excessive use of force? When British police executed Jean Charles de Menezes on a crowded tube train and realized they had just done something incredibly outrageous, their PR office went to great lengths to try and protect their image, even photoshopping images of Menezes to make him look more like some other suspect in a wanted poster. To this day, they continue to refer to Menezes as a victim of the terrorists, could they be any more arrogant? While nobody believes the police woke up that morning thinking "let's kill some random guy on the tube", it is clear they made a mistake and like many people (not just police), they immediately prioritized protecting their reputation over protecting the truth. Nobody else knows exactly what Ian was doing and exactly what the police did to him. We may never know. However, any disparaging or irrelevant comments from the police should be viewed with some caution. The horrors of incarceration It would be hard for any of us to understand everything that an innocent person goes through when detained by the police. The recently released movie about The Stanford Prison Experiment may be an interesting place to start, a German version produced in 2001, Das Experiment, is also very highly respected. The United States has the largest prison population in the world and the second-highest per-capita incarceration rate. Many, including some on death row, are actually innocent, in the wrong place at the wrong time, without the funds to hire an attorney. The system, and the police and prison officers who operate it, treat these people as packages on a conveyor belt, without even the most basic human dignity. Whether their encounter lasts for just a few hours or decades, is it any surprise that something dies inside them when they discover this cruel side of American society? Worldwide, there is an increasing trend to make incarceration as degrading as possible. People may be innocent until proven guilty, but this hasn't stopped police in the UK from locking up and strip-searching over 4,500 children in a five year period, would these children go away feeling any different than if they had an encounter with Jimmy Saville or Rolf Harris? One can only wonder what they do to adults. What all this boils down to is that people shouldn't really be incarcerated unless it is clear the danger they pose to society is greater than the danger they may face in a prison. What can people do for Ian and for justice? Now that these unfortunate smears have appeared, it would be great to try and fill the Internet with stories of the great things Ian has done for the world. Write whatever you feel about Ian's work and your own experience of Debian. While the circumstances of the final tweets from his Twitter account are confusing, the tweets appear to be consistent with many other complaints about US law enforcement. Are there positive things that people can do in their community to help reduce the harm? Sending books to prisoners (the UK tried to ban this) can make a difference. Treat them like humans, even if the system doesn't. Recording incidents of police activities can also make a huge difference, such as the video of the shooting of Walter Scott or the UK police making a brutal unprovoked attack on a newspaper vendor. Don't just walk past a situation and assume everything is under control. People making recordings may find themselves in danger, it is recommended to use software that automatically duplicates each recording, preferably to the cloud, so that if the police ask you to delete such evidence, you can let them watch you delete it and still have a copy. Can anybody think of awards that Ian Murdock should be nominated for, either in free software, computing or engineering in general? Some, like the prestigious Queen Elizabeth Prize for Engineering can't be awarded posthumously but others may be within reach. Come and share your ideas on the debian-project mailing list, there are already some here. Best of all, Ian didn't just build software, he built an organization, Debian. Debian's principles have helped to unite many people from otherwise different backgrounds and carry on those principles even when Ian is no longer among us. Find out more, install it on your computer or even look for ways to participate in the project.

Niels Thykier: dput change-all-of-debian.changes

Lucas Nussbaum recently did a blog post called Debian is still changing . I found it a very welcome continuation of his previous blog post on the same topic. I find the graphs very interesting and was very happy to learn that he included relative graphs this time. Now I can with relatively ease say that 69% of all Debian packages are using a dh-style build (source). We have another 15% using classic debhelper, which means that at least 84% of all packages uses debhelper directly. Assuming all CDBS based packages rely on the debhelper class , we are at 99%! The latter is certainly an assumption, although I suspect it is probably pretty accurate[1]. Now, it is very cute to have world dominance , but that is not my primary interest in these numbers. Instead, we can use these numbers to determine that: Such as automatic dbgsym packages, indexable build-id from dbg(sym) packages via Packages files[2], and replacing maintscripts with ldconfig triggers. All of these changes happen to be changes that could be trivially deployed with very little risk and very high efficiency[3]. Notably, none of them required a compat bump (or a new debhelper tool). Of course, I do not intend to say that every change can (or should) be deployed via debhelper and much less into an existing dh_cmd -tool. Notably, dh_strip is reaching its breaking point for content. And if we were to require a compat bump for your change, we can now at least see the adoption rate via lintian. :) Nevertheless, it is nice to know that (politics aside) there is some agility in the Debian build system! :) [1] I would very much love to see numbers to (dis)prove my assumption about CDBS + debhelper. In fact, an absolute number of packages not using debhelper (indirectly) in Debian would be very intriguing. [2] New fields by default end up the Packages file. See e.g. the Packages.xz file on the debug mirror or your apt-cache via: 
apt-cache show mscgen-dbgsym   grep ^Build-Ids
The latter assumes that you have the debug mirror in your sources list. [3] Efficiency here being features people rarely override/disable.
Filed under: Debhelper, Debian

1 January 2016

Niels Thykier: Debian, please plan for Stretch

In the 4th quarter of 2016, we will freeze Debian Stretch. If you are hoping to do any larger changes for Stretch, please consider starting on them now. This also includes features that need to be in APT/dpkg (etc.) in Stretch, so we can start using them for Buster. Even something as trivial as the automatic dbgsym packages took over 8 months to complete (from the prototype was announced in April). I call it trivial because: NB: There were certainly other parties involved. But these were the most important ones. Mind you, the dbgsym saga is not complete yet. We are still lacking support for migrating dbgsym packages to testing (and, by extension, the next stable release as well). Meanwhile, you can pull the dbgsym packages from snapshot.debian.org. In summary: If you want a larger change to land in Debian Stretch, please start already now. :)
Filed under: Debian, Release-Team

31 December 2015

Chris Lamb: Free software activities in December 2015

Here is my monthly update covering a large part of what I have been doing in the free software world (previously):
Debian My work in the Reproducible Builds project was also covered in more depth in Lunar's weekly reports (#31, #32, #33, #34).
LTS

This month I have been paid to work 12 hours on Debian Long Term Support (LTS). In that time I did the following:
  • Issued DLA 363-1 for libphp-phpmailer fixing a header injection vulnerability.
  • Issued DLA 365-1 for foomatic-filters correcting a shell injection vulnerability.
  • Issued DLA 369-1 for pygments fixing an issue shell injection vulnerability.
  • Issued DLA 374-1 for cacti to fix an SQL injection vulnerability.
  • Did some futher investigation of CVE-2011-5325 in busybox.
Uploads
  • python-django (1.9) New upstream release.
  • redis (3.0.6) New upstream stable release. I additionally backported this package to Debian stable.
  • redis (3.2~rc1) Uploaded upstream's "testing/next" branch to experimental for testing.
  • gunicorn (19.4.1) New upstream release.
Patches contributed

I also filed FTBFS bugs against acpid, android-platform-frameworks-base, antlr3, artemis, beignet, bisonc++, bobcat, bustle, cargo, checkbox-ng, code2html, cplay, datanommer.commands, dcmtkpp, debci, diffutils, diod, django-restricted-resource, docker-libkv, doomsday, dvdauthor, dwww, elasticsearch, elki, flask-script, freeipa, fso-frameworkd, funny-manpages, ggcov, ghc-mod, gmpc-plugins, gparted, gs-collections, guacamole-server, guncat, haskell-concrete-typerep, haskell-geniplate, haskell-nats, haskell-x509-util, hawtbuf, heimdal, htsjdk, inspircd, jboss-xnio, jenkins-winstone, jpeginfo, jruby-openssl, kaffeine, kdbg, ktp-accounts-kcm, kuser, libcommons-cli-java, libcommons-openpgp-java, libconfig-model-lcdproc-perl, libdata-faker-perl, libexplain, libgettext-commons-java, libgtk2-ex-printdialog-perl, libmoops-perl, libnet-frame-perl, libsendmail-milter-perl, libupnp, libuv, libvpx, liwc, m4, maven2, meep-mpich2, nagios-plugin-check-multi, natlog, netpipe, ocserv, ogre-1.8, orthanc-dicomweb, perspectives-extension, php-mail, php-pinba, phpseclib, pkg-haskell-tools, plastimatch, plexus-compiler, plexus-compiler-1.0, python-acme, python-crontab, python-cs, python-csscompressor, python-debian, python-distutils-extra, python-django-compressor, python-django-openstack-auth, python-django-tagging, python-pygit2, python-pyramid, python-pywcs, python-releases, python-shade, python-statsd, python-tasklib, python-tasklib, python-webm, python-websockets, regina-normal, rinetd, roboptim-core, rpm2html, rpm2html, ruby-factory-girl, ruby-fogbugz, ruby-i18n-inflector, ruby-loofah, ruby-protected-attributes, ruby-rack-contrib, ruby-rufus-scheduler, ruby-sanitize, ruby-sidetiq, ruby-sinatra, scsh-0.6, shogun, sleekxmpp, slugimage, spatial4j, sqwebmail-de, trac-announcer, ttt, txaws, umbrello, wine-gecko-2.21, xboxdrv, xfonts-wqy, xserver-xorg-video-openchrome, yorick & yoshimi.

Raphaël Hertzog: My Free Software Activities in December 2015

My monthly report covers a large part of what I have been doing in the free software world. I write it for my donators (thanks to them!) but also for the wider Debian community because it can give ideas to newcomers and it s one of the best ways to find volunteers to work with me on projects that matter to me. Debian LTS This month I have been paid to work 21.25 hours on Debian LTS. During this time I worked on the following things: Distro Tracker I put a big focus on tracker.debian.org work this month. I completed the switch of the mail interface from packages.qa.debian.org to tracker.debian.org and I announced the change on debian-devel-announce. The changes resulted in a few problems that I quickly fixed (like #807073) and some other failures seen only by me and that were generated by weird spam messages (did you know that a subject can t have a newline character but that it can be encoded and folded over multiple lines?). Related to that I fixed some services so that they send their mails to tracker.debian.org directly instead of relying on the old emails (they get forwarded for now but it would be nice to be able to get rid of that forward). I updated (with the help of Lucas Nussbaum) the service that forwards the Launchpad bugs to the tracker, I sent a patch to update the @packages.debian.org aliases (not yet applied), I updated the configuration of all git commit notice scripts in the Alioth collab-maint and python-modules project (many remain to be done). I asked Ubuntu s Merge-O-Matic to use the new emails as well (see LP 1525497). DAK and the Debian BTS still have to be updated, as of yet nobody reacted to my announce last but not least I updated many wiki pages which duplicated the instructions to setup the commit notice sent to the PTS. While on a good track I opted to tackle the long-standing RC bug that was plaguing tracker.debian.org (#789183), so I updated the codebase to rely on Twitter s bootstrap v4 instead of v2. I had to switch to something else for the icons since glyphicons is no longer provided as part of bootstrap and the actual license for the standalone version was not suitable for use. I opted for Github s Octicons. I made numerous little improvements while doing that (closing some bugs in the process) and I believe that the result is more pleasant to use. I also did a lot of bug triage and fixed a few small issues like the incomplete architecture list (#793547), or fixing a page used only by people with javascript disabled that was not working. Or the invalid links for packages still using CVS (ugh, see #561228). Misc packaging Django. After having added DEP-8 tests (as part of my LTS work, see above), I discovered that the current version in unstable did not pass its test suite so I filed the issue upstream (ticket 26016) and added the corresponding patch. And I encouraged others to update python-bcrypt in Debian to a newer version that would have worked with Django 1.9 (see #803096). I also fixed another small issue in Django (see ticket 26017 with my pull request that got accepted). I asked the release managers to consider accepting the latest 1.7.x version in jessie (see #807654) but I have gotten zero answer so far. And I m not the only one waiting an answer. It s a bit of a sad situation we still have a few weeks until the next point release but for once I do it in advance and I would love to have timely feedback. Last but not least, I started the maintaining the current LTS release (1.8.x) in jessie-backports. Tryton. I upgraded to Tryton 3.8 and discovered an issue that I filed in #806781. I sponsored 5 new tryton modules for Matthias Behrle (who is DM) as well as one security upload (for CVE-2015-0861). Debian Handbook. I uploaded a new version to Debian Unstable and requested (to the release managers) the permission to upload a backport of it to jessie so that jessie has a version of the package that documents jessie and not wheezy contrary to my other Django request, this one should be non-controversial but I also have had zero answer so far, see #807515. Misc. I filed #808583 when sbuild stopped working with Perl 5.22. I handled #807860 on publican, I found the corresponding upstream ticket and discovered a work around with the help of upstream (see here). Kali related work I reported a bug to #debian-apt about apt miscalculating download size (ending up with 18 EB!) which resulted in a fix here in version 1.1.4. Installing a meta-package that needed more than 2GB was no longer possible without this fix and we have a kali-linux-all metapackage in that situation that gets regularly installed in a Jenkins test. I added captcha support to Distro Tracker and enabled this feature on pkg.kali.org. I filed #808863 against uhd-host because it was not possible to install the package in a systemd-nspawn s managed chroot where /proc is read-only. And we started using this to test dist-upgrade from one version of Kali to the next Thanks See you next month for a new summary of my activities.

No comment Liked this article? Click here. My blog is Flattr-enabled.

30 December 2015

Bits from Debian: Debian mourns the passing of Ian Murdock

Ian Murdock With a heavy heart Debian mourns the passing of Ian Murdock, stalwart proponent of Free Open Source Software, Father, Son, and the 'ian' in Debian. Ian started the Debian project in August of 1993, releasing the first versions of Debian later that same year. Debian would go on to become the world's Universal Operating System, running on everything from embedded devices to the space station. Ian's sharp focus was on creating a Distribution and community culture that did the right thing, be it ethically, or technically. Releases went out when they were ready, and the project's staunch stance on Software Freedom are the gold standards in the Free and Open Source world. Ian's devotion to the right thing guided his work, both in Debian and in the subsequent years, always working towards the best possible future. Ian's dream has lived on, the Debian community remains incredibly active, with thousands of developers working untold hours to bring the world a reliable and secure operating system. The thoughts of the Debian Community are with Ian's family in this hard time. His family has asked for privacy during this difficult time and we very much wish to respect that. Within our Debian and the larger Linux community condolences may be sent to in-memoriam-ian@debian.org where they will be kept and archived.

Bits from Debian: Debian lamenta el fallecimiento de Ian Murdock

Ian Murdock Con el coraz n compungido, Debian lamenta el fallecimiento de Ian Murdock, firme partidario del software libre y abierto, padre, hijo y el 'ian' en Debian. Ian empez el proyecto en agosto de 1993, publicando la primera versi n de Debian m s tarde en el mismo a o. Debian se convertir a as en el sistema operativo universal para el mundo, funcionando en cualquier dispositivo, desde sistemas embebidos hasta la estaci n espacial. Ian se centr en crear una distribuci n y una cultura de comunidad que hiciera lo correcto tanto en lo tico como en lo t cnico. Publicar cuando est listo y una postura firme sobre la libertad de software son las reglas de oro en el mundo del software libre y de c digo abierto. La devoci n le gui en su trabajo, tanto en Debian como en los a os posteriores, siempre trabajando hacia un futuro lo mejor posible. El sue o de Debian est vivo, la comunidad permanece incre blemente activa con miles de desarrolladores que trabajan incontables horas para ofrecer un sistema operativo seguro y confiable. La comunidad de Debian da el p same a la familia en este momento tan dif cil. Su familia nos ha pedido discreci n durante estos momentos dif ciles y deseamos respetarlo. Desde dentro de Debian y de la amplia comunidad de Linux podemos expresar nuestras muestras de condolencia a in-memoriam-ian@debian.org, donde ser n guardadas y archivadas.

Craig Small: Forking processes and Gtk2

I made a change recently on the gjay program. gjay is a gtk program that basically analyzes your music and makes playlists. There is a gui frontend and a analyzer back-end and they communicate through a pipe. One really useful debugging option gtk has is to make warnings fatal, so when gtk finds one it crashes at that point and you can use gdb to trap it. The flag is g-fatal-warnings. I have been updating gjay and initially it didn t have this option, so I needed to add the gtk options, which is a simple one-liner. But then gjay gave some odd errors about XCB. Often, but not every time gjay started on stderr the following cryptic messages appear: 
[xcb] Unknown sequence number while processing queue
[xcb] Most likely this is a multi-threaded client and XInitThreads has not been called
[xcb] Aborting, sorry about that.
forktest: ../../src/xcb_io.c:274: poll_for_event: Assertion  !xcb_xlib_threads_sequence_lost' failed.
OK, so part of my init sequence was unhappy. The odd thing was it only appeared when I added the gtk options. I narrowed it down with some test code which displayed the error but stripped all the other parts out. 
  1. #include <gtk/gtk.h>
  2.  
  3. int main(int argc, char *argv[])
  4.  
  5.     GOptionContext *context;
  6.     GError *error;
  7.     pid_t pid;
  8.     GtkWidget *win;
  9.  
  10.     context = g_option_context_new("My test");
  11.     g_option_context_add_group (context, gtk_get_option_group (TRUE));
  12.     error = NULL;
  13.     if (!g_option_context_parse(context, &argc, &argv, &error))
  14.         return 1;
  15.     pid = fork();
  16.     if (pid < 0) 
  17.         return 1;
  18.     if (pid == 0)   //child
  19.         GMainLoop *loop;
  20.  
  21.         loop = g_main_new(FALSE);
  22.         g_main_run(loop);
  23.         return 0;
  24.       else   // parent
  25.         if (gtk_init_check(&argc, &argv) == FALSE)
  26.             return 1;
  27.         win = gtk_window_new(GTK_WINDOW_TOPLEVEL);
  28.         gtk_widget_show(win);
  29.         gtk_main();
  30.      
  31.     return 0;
  32.
What we got going on here is a simple gtk program. Line 11 with the gtk_get_option_group() was the line I added. The problem is that the child process is not quite setup and when the parent goes into gtk_main then you get those xcb errors. The options need to be parsed before the fork, because one of the options is to not fork (it just runs the child code directly and uses stdout with no GUI). Gjay is obviously a lot more complex than this, but follows the same pattern. There is the front-end looping through gtk_main and the back-end looping on g_main_run. The backend needs to use glib not gtk as there is a non-gui option which can be used. The solution is actually (but not obviously to me) in the documentation for gtk_get_option_group. The parameter for that function whether to open the default display when parsing the commandline arguments . Changing the TRUE to FALSE on line 11 stops XCB from complaining! The screen still appears fine but not more strange XCB messages.

Julian Andres Klode: APT 1.1.8 to 1.1.10 going faster

Not only do I keep incrementing version numbers faster than ever before, APT also keeps getting faster. But not only that, it also has some bugs fixed and the cache is now checked with a hash when opening. Important fix for 1.1.6 regression Since APT 1.1.6, APT uses the configured xz compression level. Unfortunately, the default was set to 9, which requires 674 MiB of RAM, compared to the 94 MiB required at level 6. This caused the test suite to fail on the Ubuntu autopkgtest servers, but I thought it was just some temporary hickup on their part, and so did not look into it for the 1.1.7, 1.1.8, and 1.1.9 releases. When the Ubuntu servers finally failed with 1.1.9 again (they only started building again on Monday it seems), I noticed something was wrong. Enter git bisect. I created a script that compiles the APT source code and runs a test with ulimit for virtual and resident memory set to 512 (that worked in 1.1.5), and let it ran, and thus found out the reason mentioned above. The solution: APT now defaults to level 6. New Features APT 1.1.8 introduces /usr/lib/apt/apt-helper cat-file which can be used to read files compressed by any compressor understood by APT. It is used in the recent apt-file experimental release, and serves to prepare us for a future in which files on the disk might be compressed with a different compressor (such as LZ4 for Contents files, this will improve rred speed on them by factor 7). David added a feature that enables servers to advertise that they do not want APT to download and use some Architecture: all contents when they include all in their list of architectures. This is to allow archives to drop Architecture: all packages from the architecture-specific content files, to avoid redundant data and (thus) improve the performance of apt-file. Buffered writes APT 1.1.9 introduces buffered writing for rred, reducing the runtime by about 50% on a slowish SSD, and maybe more on HDDs. The 1.1.9 release is a bit buggy and might mess up things when a write syscall is interrupted, this is fixed in 1.1.10. Cache generation improvements APT 1.1.9 and APT 1.1.10 improve the cache generation algorithms in several ways: Switching a lookup table from std::map to std::unordered_map, providing an inline isspace_ascii() function, and inlining the tolower_ascii() function which are tiny functions that are called a lot. APT 1.1.10 also switches the cache s hash function to the DJB hash function and increases the default hash table sizes to the smallest prime larger than 15000, namely 15013. This reduces the average bucket size from 6.5 to 4.5. We might increase this further in the future. Checksum for the cache, but no more syncs Prior to APT 1.1.10 writing the cache was a multi-part process:
  1. Write the the cache to a temporary file with the dirty bit set to true
  2. Call fsync() to sync the cache
  3. Write a new header with the dirty bit set to false
  4. Call fsync() to sync the new header
  5. (Rename the temporary file to the target name)
The last step was obviously not needed, as we could easily live with an intact cache that has its dirty field set to false, as we can just rebuild it. But what matters more is step 2. Synchronizing the entire 40 or 50 MB takes some time. On my HDD system, it consumed 56% of the entire cache generation time, and on my SSD system, it consumed 25% of the time. APT 1.1.10 does not sync the cache at all. It now embeds a hashsum (adler32 for performance reasons) in the cache. This helps ensure that no matter what parts of the cache are written in case of some failure somewhere, we can still detect a failure with reasonable confidence (and even more errors than before). This means that cache generation is now much faster for a lot of people. On the bad side, commands like apt-cache show that previously took maybe 10 ms to execute can now take about 80 ms. Please report back on your performance experience with 1.1.10 release, I m very interested to see if that works reasonably for other people. And if you have any other idea how to solve the issue, I d be interested to hear them (all data needs to be written before the header with dirty=0 is written, but we don t want to sync the data). Future work We seem to have a lot of temporary (?) std::string objects during the cache generation, accounting for about 10% of the run time. I m thinking of introducing a string_view class similar to the one proposed for C++17 and make use of that. I also thought about calling posix_fadvise() before starting to parse files, but the cache generation process does not seem to spend a lot of its time in system calls (even with all caches dropped before the run), so I don t think this will improve things. If anyone has some other suggestions or patches for performance stuff, let me know.
Filed under: Debian, Ubuntu

Next.

Previous.