Search Results: "reg"

31 December 2015

Raphaël Hertzog: My Free Software Activities in December 2015

My monthly report covers a large part of what I have been doing in the free software world. I write it for my donators (thanks to them!) but also for the wider Debian community because it can give ideas to newcomers and it s one of the best ways to find volunteers to work with me on projects that matter to me. Debian LTS This month I have been paid to work 21.25 hours on Debian LTS. During this time I worked on the following things: Distro Tracker I put a big focus on tracker.debian.org work this month. I completed the switch of the mail interface from packages.qa.debian.org to tracker.debian.org and I announced the change on debian-devel-announce. The changes resulted in a few problems that I quickly fixed (like #807073) and some other failures seen only by me and that were generated by weird spam messages (did you know that a subject can t have a newline character but that it can be encoded and folded over multiple lines?). Related to that I fixed some services so that they send their mails to tracker.debian.org directly instead of relying on the old emails (they get forwarded for now but it would be nice to be able to get rid of that forward). I updated (with the help of Lucas Nussbaum) the service that forwards the Launchpad bugs to the tracker, I sent a patch to update the @packages.debian.org aliases (not yet applied), I updated the configuration of all git commit notice scripts in the Alioth collab-maint and python-modules project (many remain to be done). I asked Ubuntu s Merge-O-Matic to use the new emails as well (see LP 1525497). DAK and the Debian BTS still have to be updated, as of yet nobody reacted to my announce last but not least I updated many wiki pages which duplicated the instructions to setup the commit notice sent to the PTS. While on a good track I opted to tackle the long-standing RC bug that was plaguing tracker.debian.org (#789183), so I updated the codebase to rely on Twitter s bootstrap v4 instead of v2. I had to switch to something else for the icons since glyphicons is no longer provided as part of bootstrap and the actual license for the standalone version was not suitable for use. I opted for Github s Octicons. I made numerous little improvements while doing that (closing some bugs in the process) and I believe that the result is more pleasant to use. I also did a lot of bug triage and fixed a few small issues like the incomplete architecture list (#793547), or fixing a page used only by people with javascript disabled that was not working. Or the invalid links for packages still using CVS (ugh, see #561228). Misc packaging Django. After having added DEP-8 tests (as part of my LTS work, see above), I discovered that the current version in unstable did not pass its test suite so I filed the issue upstream (ticket 26016) and added the corresponding patch. And I encouraged others to update python-bcrypt in Debian to a newer version that would have worked with Django 1.9 (see #803096). I also fixed another small issue in Django (see ticket 26017 with my pull request that got accepted). I asked the release managers to consider accepting the latest 1.7.x version in jessie (see #807654) but I have gotten zero answer so far. And I m not the only one waiting an answer. It s a bit of a sad situation we still have a few weeks until the next point release but for once I do it in advance and I would love to have timely feedback. Last but not least, I started the maintaining the current LTS release (1.8.x) in jessie-backports. Tryton. I upgraded to Tryton 3.8 and discovered an issue that I filed in #806781. I sponsored 5 new tryton modules for Matthias Behrle (who is DM) as well as one security upload (for CVE-2015-0861). Debian Handbook. I uploaded a new version to Debian Unstable and requested (to the release managers) the permission to upload a backport of it to jessie so that jessie has a version of the package that documents jessie and not wheezy contrary to my other Django request, this one should be non-controversial but I also have had zero answer so far, see #807515. Misc. I filed #808583 when sbuild stopped working with Perl 5.22. I handled #807860 on publican, I found the corresponding upstream ticket and discovered a work around with the help of upstream (see here). Kali related work I reported a bug to #debian-apt about apt miscalculating download size (ending up with 18 EB!) which resulted in a fix here in version 1.1.4. Installing a meta-package that needed more than 2GB was no longer possible without this fix and we have a kali-linux-all metapackage in that situation that gets regularly installed in a Jenkins test. I added captcha support to Distro Tracker and enabled this feature on pkg.kali.org. I filed #808863 against uhd-host because it was not possible to install the package in a systemd-nspawn s managed chroot where /proc is read-only. And we started using this to test dist-upgrade from one version of Kali to the next Thanks See you next month for a new summary of my activities.

No comment Liked this article? Click here. My blog is Flattr-enabled.

30 December 2015

Bits from Debian: Debian lamenta el dec s de l'Ian Murdock

Ian Murdock Amb el cor compungit, Debian lamenta la p rdua d'Ian Murdock, ferm partidari del programari lliure i de codi obert, pare, fill i l'"ian" a Debian. L'Ian va comen ar el projecte Debian l'agost de 1993, i va publicar les primeres versions m s tard el mateix any. Debian es convertiria en el sistema operatiu universal pel m n, que funciona en tot tipus de dispositius, des de sistemes encastats fins a l'estaci especial. L'Ian es va centrar en crear una distribuci i una cultura de comunitat que fessin el correcte, ticament o t cnica. Publicar quan estigui a punt i la ferma posici sobre la llibertat de programari s n les regles d'or en el m n del programari lliure i de codi obert. La seva devoci per fer el correcte va guiar la seva feina, tant a Debian com els seg ents anys, sempre treball per assolir el millor futur possible. El somni de l'Ian est viu, la comunitat de Debian continua incre blement activa, amb milers de desenvolupadors que treballen innumerables hores per proporcionar un sistema operatiu segur i fiable. Els pensaments de la comunitat de Debian s n amb la fam lia de l'Ian en aquests moments tant durs. La seva fam lia ens ha demanat privacitat en aquests moments dif cils i volem respectar-ho. Des de Debian i des de la comunitat de Linux en general podem enviar mostres de condol a in-memoriam-ian@debian.org; seran guardades i arxivades.

Bits from Debian: Debian lamenta el fallecimiento de Ian Murdock

Ian Murdock Con el coraz n compungido, Debian lamenta el fallecimiento de Ian Murdock, firme partidario del software libre y abierto, padre, hijo y el 'ian' en Debian. Ian empez el proyecto en agosto de 1993, publicando la primera versi n de Debian m s tarde en el mismo a o. Debian se convertir a as en el sistema operativo universal para el mundo, funcionando en cualquier dispositivo, desde sistemas embebidos hasta la estaci n espacial. Ian se centr en crear una distribuci n y una cultura de comunidad que hiciera lo correcto tanto en lo tico como en lo t cnico. Publicar cuando est listo y una postura firme sobre la libertad de software son las reglas de oro en el mundo del software libre y de c digo abierto. La devoci n le gui en su trabajo, tanto en Debian como en los a os posteriores, siempre trabajando hacia un futuro lo mejor posible. El sue o de Debian est vivo, la comunidad permanece incre blemente activa con miles de desarrolladores que trabajan incontables horas para ofrecer un sistema operativo seguro y confiable. La comunidad de Debian da el p same a la familia en este momento tan dif cil. Su familia nos ha pedido discreci n durante estos momentos dif ciles y deseamos respetarlo. Desde dentro de Debian y de la amplia comunidad de Linux podemos expresar nuestras muestras de condolencia a in-memoriam-ian@debian.org, donde ser n guardadas y archivadas.

Julian Andres Klode: APT 1.1.8 to 1.1.10 going faster

Not only do I keep incrementing version numbers faster than ever before, APT also keeps getting faster. But not only that, it also has some bugs fixed and the cache is now checked with a hash when opening. Important fix for 1.1.6 regression Since APT 1.1.6, APT uses the configured xz compression level. Unfortunately, the default was set to 9, which requires 674 MiB of RAM, compared to the 94 MiB required at level 6. This caused the test suite to fail on the Ubuntu autopkgtest servers, but I thought it was just some temporary hickup on their part, and so did not look into it for the 1.1.7, 1.1.8, and 1.1.9 releases. When the Ubuntu servers finally failed with 1.1.9 again (they only started building again on Monday it seems), I noticed something was wrong. Enter git bisect. I created a script that compiles the APT source code and runs a test with ulimit for virtual and resident memory set to 512 (that worked in 1.1.5), and let it ran, and thus found out the reason mentioned above. The solution: APT now defaults to level 6. New Features APT 1.1.8 introduces /usr/lib/apt/apt-helper cat-file which can be used to read files compressed by any compressor understood by APT. It is used in the recent apt-file experimental release, and serves to prepare us for a future in which files on the disk might be compressed with a different compressor (such as LZ4 for Contents files, this will improve rred speed on them by factor 7). David added a feature that enables servers to advertise that they do not want APT to download and use some Architecture: all contents when they include all in their list of architectures. This is to allow archives to drop Architecture: all packages from the architecture-specific content files, to avoid redundant data and (thus) improve the performance of apt-file. Buffered writes APT 1.1.9 introduces buffered writing for rred, reducing the runtime by about 50% on a slowish SSD, and maybe more on HDDs. The 1.1.9 release is a bit buggy and might mess up things when a write syscall is interrupted, this is fixed in 1.1.10. Cache generation improvements APT 1.1.9 and APT 1.1.10 improve the cache generation algorithms in several ways: Switching a lookup table from std::map to std::unordered_map, providing an inline isspace_ascii() function, and inlining the tolower_ascii() function which are tiny functions that are called a lot. APT 1.1.10 also switches the cache s hash function to the DJB hash function and increases the default hash table sizes to the smallest prime larger than 15000, namely 15013. This reduces the average bucket size from 6.5 to 4.5. We might increase this further in the future. Checksum for the cache, but no more syncs Prior to APT 1.1.10 writing the cache was a multi-part process:
  1. Write the the cache to a temporary file with the dirty bit set to true
  2. Call fsync() to sync the cache
  3. Write a new header with the dirty bit set to false
  4. Call fsync() to sync the new header
  5. (Rename the temporary file to the target name)
The last step was obviously not needed, as we could easily live with an intact cache that has its dirty field set to false, as we can just rebuild it. But what matters more is step 2. Synchronizing the entire 40 or 50 MB takes some time. On my HDD system, it consumed 56% of the entire cache generation time, and on my SSD system, it consumed 25% of the time. APT 1.1.10 does not sync the cache at all. It now embeds a hashsum (adler32 for performance reasons) in the cache. This helps ensure that no matter what parts of the cache are written in case of some failure somewhere, we can still detect a failure with reasonable confidence (and even more errors than before). This means that cache generation is now much faster for a lot of people. On the bad side, commands like apt-cache show that previously took maybe 10 ms to execute can now take about 80 ms. Please report back on your performance experience with 1.1.10 release, I m very interested to see if that works reasonably for other people. And if you have any other idea how to solve the issue, I d be interested to hear them (all data needs to be written before the header with dirty=0 is written, but we don t want to sync the data). Future work We seem to have a lot of temporary (?) std::string objects during the cache generation, accounting for about 10% of the run time. I m thinking of introducing a string_view class similar to the one proposed for C++17 and make use of that. I also thought about calling posix_fadvise() before starting to parse files, but the cache generation process does not seem to spend a lot of its time in system calls (even with all caches dropped before the run), so I don t think this will improve things. If anyone has some other suggestions or patches for performance stuff, let me know.
Filed under: Debian, Ubuntu

29 December 2015

David Bremner: Converting PDFs to DJVU

Today I was wondering about converting a pdf made from scan of a book into djvu, hopefully to reduce the size, without too much loss of quality. My initial experiments with pdf2djvu were a bit discouraging, so I invested some time building gsdjvu in order to be able to run djvudigital. Watching the messages from djvudigital I realized that the reason it was achieving so much better compression was that it was using black and white for the foreground layer by default. I also figured out that the default 300dpi looks crappy since my source document is apparently 600dpi. I then went back an compared djvudigital to pdf2djvu a bit more carefully. My not-very-scientific conclusions: Perhaps most compellingly, the output from pdf2djvu has sensible metadata and is searchable in evince. Even with the --words option, the output from djvudigital is not. This is possibly related to the error messages like 
Can't build /Identity.Unicode /CIDDecoding resource. See gs_ciddc.ps .
It could well be my fault, because building gsdjvu involved guessing at corrections for several errors. Some of these issues have to do with building software from 2009 (the instructions suggestion building with ghostscript 8.64) in a modern toolchain; others I'm not sure. There was an upload of gsdjvu in February of 2015, somewhat to my surprise. AT&T has more or less crippled the project by licensing it under the CPL, which means binaries are not distributable, hence motivation to fix all the rough edges is minimal.
Version kilobytes per page position in figure
Original PDF 80.9 top
pdf2djvu --dpi=450 92.0 not shown
pdf2djvu --monochrome --dpi=450 27.5 second from top
pdf2djvu --monochrome --dpi=600 --loss-level=50 21.3 second from bottom
djvudigital --dpi=450 29.4 bottom
djvu-compare.png

Stefano Zacchiroli: Shuttleworth Foundation Flash Grant - 2015 report

1 year of Shuttleworth Foundation Flash Grant As announced last year, starting January 2015 I've benefited from a "Flash Grant" kindly awarded to me by the Shuttleworth Foundation. This post reports publicly about how I've used the money to promote Free Software via my own activism, over the period January-December 2015. I'm lucky to have a full-time academic job that provides me with a salary and basic computer hardware. But Free Software not being the only focus of my job, it gets difficult at times to get travel funding to specific Free Software events. So that is what I've mostly used the grant money for: attend Free Software events that I wouldn't have been able to attend otherwise. On grant money I've attended LibrePlanet 2015 (2015-03-19-boston-libreplanet label in the financial reports below), where I've given the talk Distributions and the Free "Cloud", and FSFE's LLW 2015 (2015-04-15-barcelona-fsfe-legal) workshop. Furthermore I've used the grant to reimburse otherwise not reimbursed out of pocket expenses in a trip to San Francisco (2015-11-06-san-francisco-gsoc+osi) that have been otherwise sponsored by Google (to attend the Summer of Code Mentor Summit) and OSI (to attend a F2F meeting of the Board of Directors). Finally, I've used grant money to offer lunch to invited lecturers in my master-level Free Software class at the university (label 2015-foss-class). Actual financial reports are reported below, in ledger format. It should be noted that, contrary to the usual expected 6-month duration of flash grants, I've used only about half the grant amount over a 12-month period; I do not plan to pocket what remains, but rather keep on using it over the next year, reporting again publicly at the end of the period. Also, I did not breakdown further out of pocket expenses, but they invariably stand for public transport tickets and meals. Balance sheet Overall: 
         1966,11 EUR  Assets:Funds
        -4052,52 EUR  Equity:Opening balances
         2086,41 EUR  Expenses
           15,90 EUR    Bank:Commissions
          424,00 EUR    Conference:Registration
           56,50 EUR    Teaching:Speaker-invitation
         1590,01 EUR    Travel
          249,02 EUR      Lodgement
          562,51 EUR      Out-of-pocket
          778,48 EUR      Plane
--------------------
                   0
Breakdown by purpose: Journal 
2014-12-03 Shuttleworth Foundation flash grant                                    Equity:Opening balances                     -4052,52 EUR    -4052,52 EUR
                                                                                  Assets:Funds                                 4052,52 EUR               0
2014-12-04 bank commissions on incoming transfer                                  Expenses:Bank:Commissions                      15,90 EUR       15,90 EUR
                                                                                  Assets:Funds                                  -15,90 EUR               0
2014-12-24 plane tickets Paris-Boston round trip to attend LibrePlanet 2015       Expenses:Travel:Plane                         627,84 EUR      627,84 EUR
                                                                                  Assets:Funds                                 -627,84 EUR               0
2015-01-02 LibrePlanet 2015 registration + travel fund contribution               Expenses:Conference:Registration              424,00 EUR      424,00 EUR
                                                                                  Assets:Funds                                 -424,00 EUR               0
2015-03-02 plane tickets Paris-Barcelona round trip to attend LLW 2015            Expenses:Travel:Plane                         150,64 EUR      150,64 EUR
                                                                                  Assets:Funds                                 -150,64 EUR               0
2015-03-19 lunch with invited speaker for lecture about FOSS release management   Expenses:Teaching:Speaker-invitation           28,00 EUR       28,00 EUR
                                                                                  Assets:Funds                                  -28,00 EUR               0
2015-03-25 lunch with invited speaker for lecture about FOSS business models      Expenses:Teaching:Speaker-invitation           28,50 EUR       28,50 EUR
                                                                                  Assets:Funds                                  -28,50 EUR               0
2015-04-03 LibrePlanet 2015 out of pocket expenses                                Expenses:Travel:Out-of-pocket                 213,38 EUR      213,38 EUR
                                                                                  Assets:Funds                                 -213,38 EUR               0
2015-04-15 LLW 2015 out of pocket expenses                                        Expenses:Travel:Out-of-pocket                  80,00 EUR       80,00 EUR
                                                                                  Assets:Funds                                  -80,00 EUR               0
2015-05-06 hotel in Barcelona for LLW 2015 (3 nights)                             Expenses:Travel:Lodgement                     249,02 EUR      249,02 EUR
                                                                                  Assets:Funds                                 -249,02 EUR               0
2015-11-29 OSI F2F Fall 2015 out of pocket expenses                               Expenses:Travel:Out-of-pocket                 269,13 EUR      269,13 EUR
                                                                                  Assets:Funds                                 -269,13 EUR               0

28 December 2015

Dirk Eddelbuettel: BH 1.60.0-1

A new release of BH is now on CRAN. BH provides a large part of the Boost C++ libraries as a set of template headers for use by R, possibly with Rcpp as well as other packages. This release both upgrades the version of Boost to the recently-released upstream version Boost 1.60.0 and also adds Boost Phoenix. A brief summary of changes from the NEWS file is below.
Changes in version 1.60.0-1 (2015-12-24)
  • Upgraded to Boost 1.60 installed directly from upstream source
  • Added Boost phoenix as discussed in GH ticket #19
Courtesy of CRANberries, there is also a diffstat report for the most recent release. Comments and suggestions are welcome via the mailing list or the issue tracker at the GitHubGitHub repo.

Edited on 2015-12-30: Typo in titled corrected, it is BH 1.60.0-1

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Alessio Treglia: Pallinux: Olly Olly Oxen Free!

Pallinux: Artwork by Fabio "Pixel" Colinelli

Pallinux: Artwork by Fabio Pixel Colinelli

In a world far away, in the dark Land of Digitos only populated by machines and computers, the evil Mister Woo was ruling over all. Over time, this terrible dictator was becoming a horrendous fire-eyed giant, walking the whole day by vibrating the heavy steps into his Kingdom, leaving behind him a trail of smoke and terror. Mr. Woo always wore a long, shabby and dirty top hat that had once been white, so old and ragged that he could not even keep it up straight on his head. Throughout the Land of Digitos, the inhabitants computers were scattered, each <Read more >

27 December 2015

Ben Hutchings: What's new in initramfs-tools

I spent much of my Debian time in the last few weeks triaging and fixing bugs in initramfs-tools. All of the important bugs should be fixed, except for two where I needed more information from the submitter. Due to the scope of these changes and potential for regressions, I've made a release candidate, version 0.121~rc2, rather than a full release. This is now available in experimental, and I would appreciate real-world testing feedback in the next few weeks. (If you're wondering what happened to rc1, that had 'unstable' as the distribution, which I didn't notice until after pushing the tag.) From the release announcement, the major changes are: Thanks to Andy Whitcroft, Boris Egorov, Laurent Bigonville, Roger Leigh, Roger Shimizu and Salvatore Bonaccorso for their patches which I applied in this version.

Gregor Herrmann: RC bugs 2015/51-52

the upload of perl 5.22 to unstable some days ago provided ample opportunity to fix some new RC bugs. here's the list of my work:

Gregor Herrmann: RC bugs 2015/51-52

the upload of perl 5.22 to unstable some days ago provided ample opportunity to fix some new RC bugs. here's the list of my work:

Steve Kemp: Some things are universal?

I don't often do retrospectives, but this year has been an unusual one for me, as I moved to Finland almost six months ago. The topic has come up in conversation a lot over the past few months, so when people ask me what I think I can give some simple answers without too much thought. Here's a brief summary. There are some obvious changes:
The Traffic
The traffic drives on the right-hand side of the roads, which took a bit of getting used to, but isn't a huge surprise as I've travelled in Europe in the past. There aren't so many countries that drive on the left after all so most people probably wouldn't even notice this as odd. When it comes to traffic one thing nice about Helsinki is that most junctions are "zebra crossings". Sure they don't have flashing lights, but they have shaded areas, and pedestrians have right of way. As for transport the city of Helsinki has local trains, trams, buses and taxis. The trams and buses all use the same card for payment so transport is integrated very well. I buy a time-based card, spending about 50 for a month of unlimited travel. If you prefer you may add euros to your card and pay for distinct journeys - but that works out more expensive if you travel twice, or more, a day.
The Money
Finland uses the Euro these days, having switched from the Finnish markka in 2002. Enough said. Costs are largely in line with what I'd expect: Cigarettes are cheap, beer is expensive. Some things are very expensive, some things are very cheap. Largely the expensive things are those that are imported. It is a very small country after all.
The Language
Finnish is .. complex. But I've not struggled too much. Mostly I can buy what I want without difficulty. There are weird exceptions though for example I went out to buy soup one day and had to return carrying only shame and disappointment: I can't read the language on the tins and what I thought was soup turned out to be a can of chopped tomatoes. Food is good though, and available easily (!!). The only significant surprise when it comes to shopping is that loose goods must be weighed yourself. You pick up a bunch of bananas, take it to the scales, press the button that has a picture of a banana on it, and it prints out a label you attach to them - at the till the cashier will scan the label and charge you. If you forget, or don't know how to do it they'll tut and complain. In daily life I use two phrases frequently and they are sufficient for communcation:
  • "minua haluan ... kahvi kakku olut"
    • "I want ... coffee cake beer".
  • "kiitos"
    • "Thanks"
Usually people speak to me in English, which is a little annoying as it means I'm not learning as much as I could. But that said over the past few months I've had proper conversations entirely in Finnish with shop-keepers, and similar. So I'm getting better.
The Culture
Finnish people are friendly, but terse. That's the reputation. The Finnish people are alcoholics, and have high rates of suicide. Also the reputation. Finally we know that the Finnish people consume more coffee than the rest of the world. All those things are true, but they're not enough by far to describe the people. Obviously they're all different, and we have a lot of people from other parts of the world here too - Russians, Asians, Somalians. So culture is complex .. but markedly different than in the UK. I could write more about this, but I think for the moment I'll just draw a line under culture and say that I'm enjoying the interactions with people here, and while many things are slightly "off", it's not bad. Just different. Also saunas are fun. I've never had any qualms about being naked with strangers, so I don't really understand why Americans, and others, find this so difficult/surprising. But yeah, saunas are great. Things that Finland is known for internationally: The invention of the molotov cocktail, rally-driving, hockey, world's strongest man, Moomins, Tom of Finland, Salmiakki.
The Weather
Not too hot. Not too cold. But that's largely because I'm one of those "hot" people who doesn't really get cold even at the best of times. My ideal temperatures are about 13 C. My wife prefers 15 C, or more. We don't fight any more. Mostly. Winter is apparently full of snow, but this year has been poor. We had the first snowfall yesterday, here in Helsinki, and we woke this morning a blanket of snow about two inches high. It looks pretty. The biggest thing about weather in Finland is the constant darkness in winter, and the constant sun in Summer. In Summer there were like 22 hours of sunlight a day which made sleeping hard when we moved into our flat - with no curtains. In winter it feels like there is 20 minutes of sunlight a day. It's not that bad here in the south, although I think it is something like five hours and less in the north. I've never had any real issues with depression, or similar: People have good days and bad days, I'd generally be "OK" or "great". In the darkness? I've been grumpy at colleagues, I've made bad choices, I've lapsed attention. I'm not sure I can blame it on the weather, or my reaction to the weather, but I know I've not been as "happy" as I "should". It requires effort to be enthusiastic in a way I've never experienced before. Thankfully once I (slowly) realized this I took action and I think I'm good now. Unlike the UK the buildings here are relatively modern. I think that's the biggest contributing factor to how houses are "warm". Houses have all been built in the last 50-100 years, so you have proper insulation. Even though it might be very very cold outdoors indoors you can be naked without heating. Try that in the UK and you might freeze in some of the older leakier houses! You do have to laugh, though, when people point out "the oldest pub" in the city though. Where I come from if I pub isn't 500+ years old you wouldn't give it a seconds thought - places like The Golden Fleece, etc.
I could write more. I probably should. But it has been an interesting year, and although there are things I miss about the UK, and Edinburgh specifically, I have no regrets. I'm glad I came. What triggered this post? I said "Some things are universal" to my wife, when I saw a child riding a bicycle they'd obviously just received for Christmas. Her reaction "No Finnish person would buy a bicycle at Christmas - they'd expect too much snow!". So perhaps it was another immigrant family. Christmas bicycles universal, or not, it doesn't really matter.

23 December 2015

Petter Reinholdtsen: OpenALPR, find car license plates in video streams - nice free software

When I was a kid, we used to collect "car numbers", as we used to call the car license plate numbers in those days. I would write the numbers down in my little book and compare notes with the other kids to see how many region codes we had seen and if we had seen some exotic or special region codes and numbers. It was a fun game to pass time, as we kids have plenty of it. A few days I came across the OpenALPR project, a free software project to automatically discover and report license plates in images and video streams, and provide the "car numbers" in a machine readable format. I've been looking for such system for a while now, because I believe it is a bad idea that the automatic number plate recognition tool only is available in the hands of the powerful, and want it to be available also for the powerless to even the score when it comes to surveillance and sousveillance. I discovered the developer wanted to get the tool into Debian, and as I too wanted it to be in Debian, I volunteered to help him get it into shape to get the package uploaded into the Debian archive. Today we finally managed to get the package into shape and uploaded it into Debian, where it currently waits in the NEW queue for review by the Debian ftpmasters. I guess you are wondering why on earth such tool would be useful for the common folks, ie those not running a large government surveillance system? Well, I plan to put it in a computer on my bike and in my car, tracking the cars nearby and allowing me to be notified when number plates on my watch list are discovered. Another use case was suggested by a friend of mine, who wanted to set it up at his home to open the car port automatically when it discovered the plate on his car. When I mentioned it perhaps was a bit foolhardy to allow anyone capable of placing his license plate number of a piece of cardboard to open his car port, men replied that it was always unlocked anyway. I guess for such use case it make sense. I am sure there are other use cases too, for those with imagination and a vision. If you want to build your own version of the Debian package, check out the upstream git source and symlink ./distros/debian to ./debian/ before running "debuild" to build the source. Or wait a bit until the package show up in unstable.

22 December 2015

Ian Wienand: Platform bootstrap for OpenStack Infrastructure

This is a terse guide on bootstrapping virtual-machine images for OpenStack infrastructure, with the goal of adding continuous-integration support for new platforms. It might also be handy if you are trying to replicate the upstream CI environment. It covers deployment to Rackspace for testing; Rackspace is one of the major providers of capacity for the OpenStack Infrastructure project, so it makes a good place to start when building up your support story. Firstly, get an Ubuntu Trusty environment to build the image in (other environments, like CentOS or Fedora, probably work -- but take this step to minimise differences to what the automated machinery of what upstream does). You want a fair bit of memory, and plenty of disk-space. The tool used for building virtual-machine images is diskimage-builder. In short, it takes a series of elements, which are really just scripts run in different phases of the build process. I'll describe building a Fedora image, since that's been my focus lately. We will use a fedora-minimal element -- this means the system is bootstrapped from nothing inside a chroot environment, before eventually being turned into a virtual-machine image (contrast this to the fedora element, which bases itself off the qcow2 images provided by the official Fedora cloud project). Thus you'll need a few things installed on the Ubuntu host to deal with bootstrapping a Fedora chroot 
apt-get install yum yum-utils python-lzma
You will hit stuff like that python-lzma dependency on this road-less-travelled -- technically it is a bug that yum packages on Ubuntu don't depend on it; without it you will get strange yum errors about unsupported compression. At this point, you can bootstrap your diskimage-builder environment. You probably want diskimage-builder from git, and then build up a virtualenv for your support bits and pieces. 
git clone git://git.openstack.org/openstack/diskimage-builder
virtualenv dib_env
. dib_env/bin/activate
pip install dib-utils
dib-utils is a small part of diskimage-builder that is split-out; don't think too much about it. While diskimage-builder is responsible for the creation of the basic image, there are a number of elements provided by the OpenStack project-config repository that bootstrap the OpenStack environment. This does a lot of stuff, including caching all git trees (so CI jobs aren't cloning everything constantly) and running puppet setup. 
git clone git://git.openstack.org/openstack-infra/project-config
There's one more trick required for building the VHD images that Rackspace requires; make sure you install the patched vhd-util as described in the script help. At this point, you can probably build an image. Here's something approximating what you will want to do 
break=after-error \
TMP_DIR=~/tmp \
ELEMENTS_PATH=~/project-config/nodepool/elements \
DIB_DEV_USER_PASSWORD="password"  DIB_DEV_USER_PWDLESS_SUDO=1 \
DISTRO=23 \
./bin/disk-image-create -x --no-tmpfs -t vhd \
   fedora-minimal vm devuser simple-init \
   openstack-repos puppet nodepool-base node-devstack
To break some of this down This goes and does its thing; it will take about 20 minutes. Depending on how far your platform diverges from the existing support, it will require a lot of work to get everything working so you can get an image out the other side. To see a rough example of what should be happening, see the logs of the official image builds that happen for a variety of platforms. At some point, you should get a file image.vhd which is now ready to be deployed. The only reasonable way to do this is with shade. You can quickly grab this into the virtualenv we created before 
pip install shade
Now you'll need to setup a clouds.yaml file to give yourself the permissions to upload the image. It should look something like 
clouds:
  rax:
    profile: rackspace
    auth:
       username: your_rax_username
       password: your_rax_password
       project_id: your_rax_accountnum
    regions:
    - IAD
You should know your user-name and password (whatever you log into the website with), and when you login to Rackspace your project_id value is listed in the drop-down box labeled with your username as Account #. shade has no UI as such, so a simple script will do the upload. 
import shade
shade.simple_logging(debug=True)
cloud = shade.openstack_cloud(cloud='rax')
image = cloud.create_image('image-name', filename='image.vhd', wait=True)
Now wait -- this will also take a while. Even after upload it takes a fair while to process (you will see the shade debug output looping around some glance calls seeing if it is ready). If everything works, the script should return and you should be able to see the new image in the output of nova list-images. At this point, you're ready to try booting! One caveat is that the Rackspace web interface does not seem to give you the option to boot with a configuration drive available to the host, essential for simple-init to bring up the network. So boot via the API with something like 
nova boot --flavor=2 --image=image-uuid --config-drive 1 test-image
This should build and boot your image! This will allow you to open a whole new door of debugging to get your image booting correctly. You can now iterate by rebuilding/uploading/booting as required. Note these are pretty big images, and uploaded in broken-up swift files, so I find swift delete --all helpful to reset between builds (obviously, I have nothing else in swift that I want to keep). The Rackspace java-based console UI is rather annoying; it cuts itself off every time the host reboots. This makes it quite difficult to catch the early bootup, or modify grub options via the boot-loader, etc. You might need to fiddle timeout options, etc in the image build. If you've managed to get your image booting and listening on the network, you're a good-deal of the way towards having your platform supported in upstream OpenStack CI. At this point, you likely want to peruse the nodepool configuration and get an official build happening here. Once that is up, you can start the process of adding jobs that use your platform to test! Don't worry, there's plenty more that can go wrong from here -- but you're on the way! OpenStack infra is a very dynamic environment, which many changes in progress; so in general, #openstack-infra on freenode is going to be a great place to start looking for help.

21 December 2015

Alessio Treglia: Creativity, Innovation and the Included Middle logic

The pressure of the post-modernism is establishing its bases on our general lack of ability to overcome a number of dualisms that have become ingrained in the modern way of thinking[1]. This is mainly due to the strong influence of past centuries scientific Reductionism , which postulated that any system to be understood had to be reduced to its minimum component elements.

However, a so defined system is a closed system, which does not interact with the surrounding environment and it can exist (not always) only in a reality-isolated laboratory. The logic of Complexity , instead, takes into account the open systems and all the interconnections and influences of the system itself with the world around it, in every physical, social, psychological and symbolic aspect

<Read more >

20 December 2015

Iain R. Learmonth: A Week of Debian

For the last week I have been stuck in England. For the vast majority of that time, I've had nothing to do except work on Debian and this blog post documents some of the things I worked on. Obviously spending a whole week on Debian, there's going to be some packaging involved. The following packages got new versions in unstable this last week: Packaging updates were one of the simpler tasks tackled this week though. I spent a lot of time this week on Debian Live along with others in the #debian-live IRC channel. Over the last week we achieved a number of things, possibly the most important being that all the generic live support packages (i.e. live-boot, live-config and live-tools) have now been converted into native packages, have their VCS repositories hosted on Alioth and have seen a good number of patches merged from the BTS and from the old patch system. All future patches will be managed via the BTS for Debian Live, as with other Debian projects. We've also put effort into getting documentation online again now that the live.debian.net server has been turned off and we now have the live-build documentation and the live-wrapper documentation hosted on the project's Alioth webspace. The live-wrapper user documentation was mostly written this week so may not be amazing. On the smaller task list for Debian Live, the KGB bots are now present in the channel and reporting on git commits, a number of the more popular URLs from live.debian.net are now being redirected to their new locations (although we've had to recover these URLs from web.archive.org, the list of redirections is certainly not complete) and we've made updates to the wiki pages about how to contribute to the project. Unfortunately, no one has stepped forwards as a new lead for live-build and so this package has been orphaned. This does not necessarily mean that the package will be removed from Debian any time soon, just that it does not currently have a maintainer. If you're interested in taking over maintainence of live-build, see #808048. For testing the core live support packages, new live images for stretch have been built using live-wrapper and are available here. There are known issues with the syslinux configuration and these are not isohybrid images. My other major project this week has been my efforts to form a Debian Metadata team. The Debian Metadata team would produce the frameworks and run the services that make data about Debian available in a number of formats to make the data as accessible as possible and to encourage its use both within Debian and in external projects. Currently this includes two experimental services: rdf.debian.net and map.debian.net. We're don't plan to duplicate any of the work done by UDD, but make the data aggregated in UDD more accessible to users and developers. This means publishing that data in JSON/JSONP, KML, RDF, iCalendar and any other format that makes sense. There may be instances where it makes sense to augment the published data with live data, for example mirror availability should probably be live and not just a recent snapshot. If you're interested in participating in the Debian Metadata team, you can register your interested on bug #808049. This is not a complete summary of all my activities over the last week, but for those interested, it should give you an idea of what I've been up to. Finally, for those of you that have been waiting for my write-up on the airgapped GnuPG master key, I decided in the end that my blog was not the right place for this. You can find the guide in two parts: key generation and key export to the YubiKey. I've tried to keep these guides as generic as possible while still being as useful as possible. While writing these up on the wiki, I've also created pages for OpenPGP and GnuPG and I've almost entirely rewritten the DebianKeyring wiki page. I discovered this awesome guide to OpenPGP concepts which I would recommend to anyone that is new to OpenPGP.

Petter Reinholdtsen: Using appstream with isenkram to install hardware related packages in Debian

Around three years ago, I created the isenkram system to get a more practical solution in Debian for handing hardware related packages. A GUI system in the isenkram package will present a pop-up dialog when some hardware dongle supported by relevant packages in Debian is inserted into the machine. The same lookup mechanism to detect packages is available as command line tools in the isenkram-cli package. In addition to mapping hardware, it will also map kernel firmware files to packages and make it easy to install needed firmware packages automatically. The key for this system to work is a good way to map hardware to packages, in other words, allow packages to announce what hardware they will work with. I started by providing data files in the isenkram source, and adding code to download the latest version of these data files at run time, to ensure every user had the most up to date mapping available. I also added support for storing the mapping in the Packages file in the apt repositories, but did not push this approach because while I was trying to figure out how to best store hardware/package mappings, the appstream system was announced. I got in touch and suggested to add the hardware mapping into that data set to be able to use appstream as a data source, and this was accepted at least for the Debian version of appstream. A few days ago using appstream in Debian for this became possible, and today I uploaded a new version 0.20 of isenkram adding support for appstream as a data source for mapping hardware to packages. The only package so far using appstream to announce its hardware support is my pymissile package. I got help from Matthias Klumpp with figuring out how do add the required metadata in pymissile. I added a file debian/pymissile.metainfo.xml with this content: 
<?xml version="1.0" encoding="UTF-8"?>
<component>
  <id>pymissile</id>
  <metadata_license>MIT</metadata_license>
  <name>pymissile</name>
  <summary>Control original Striker USB Missile Launcher</summary>
  <description>
    <p>
      Pymissile provides a curses interface to control an original
      Marks and Spencer / Striker USB Missile Launcher, as well as a
      motion control script to allow a webcamera to control the
      launcher.
    </p>
  </description>
  <provides>
    <modalias>usb:v1130p0202d*</modalias>
  </provides>
</component>
The key for isenkram is the component/provides/modalias value, which is a glob style match rule for hardware specific strings (modalias strings) provided by the Linux kernel. In this case, it will map to all USB devices with vendor code 1130 and product code 0202. Note, it is important that the license of all the metadata files are compatible to have permissions to aggregate them into archive wide appstream files. Matthias suggested to use MIT or BSD licenses for these files. A challenge is figuring out a good id for the data, as it is supposed to be globally unique and shared across distributions (in other words, best to coordinate with upstream what to use). But it can be changed later or, so we went with the package name as upstream for this project is dormant. To get the metadata file installed in the correct location for the mirror update scripts to pick it up and include its content the appstream data source, the file must be installed in the binary package under /usr/share/appdata/. I did this by adding the following line to debian/pymissile.install: 
debian/pymissile.metainfo.xml usr/share/appdata
With that in place, the command line tool isenkram-lookup will list all packages useful on the current computer automatically, and the GUI pop-up handler will propose to install the package not already installed if a hardware dongle is inserted into the machine in question. Details of the modalias field in appstream is available from the DEP-11 proposal. To locate the modalias values of all hardware present in a machine, try running this command on the command line: 
cat $(find /sys/devices/ grep modalias)
To learn more about the isenkram system, please check out my blog posts tagged isenkram.

19 December 2015

Dirk Eddelbuettel: RcppTOML 0.0.5

After a short hiatus, a new version of the (wonderfully useful, if I dare say so ;-) RcppTOML package is now on CRAN. RcppTOML lets R read the (absolutely awesome) TOML configuration file format--which is simply fabulous as it emphasizes strong readability for humans while at the same time supporting strong typing as well as immediate and clear error reports. At work, we're all fans now. The new version on CRAN mostly updates to an updated version of the underlying cpptoml library by Chase. Nothing much changed on the R side. Courtesy of CRANberries, there is a diffstat report for this release More information and examples are on the RcppTOML page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

18 December 2015

Joachim Breitner: A multitude of early Christmas presents

Today was a nice day with a surprising number of early Christmas presents:

Martin Pitt: What s new in autopkgtest: LXD, MaaS, apt pinning, and more

The last two major autopkgtest releases (3.18 from November, and 3.19 fresh from yesterday) bring some new features that are worth spreading. New LXD virtualization backend 3.19 debuts the new adt-virt-lxd virtualization backend. In case you missed it, LXD is an API/CLI layer on top of LXC which introduces proper image management, seamlessly use images and containers on remote locations, intelligently caching them locally, automatically configure performant storage backends like zfs or btrfs, and just generally feels really clean and much simpler to use than the classic LXC. Setting it up is not complicated at all. Install the lxd package (possibly from the backports PPA if you are on 14.04 LTS), and add your user to the lxd group. Then you can add the standard LXD image server with 
  lxc remote add lco https://images.linuxcontainers.org:8443
and use the image to run e. g. the libpng test from the archive: 
  adt-run libpng --- lxd lco:ubuntu/trusty/i386
  adt-run libpng --- lxd lco:debian/sid/amd64
The adt-virt-lxd.1 manpage explains this in more detail, also how to use this to run tests in a container on a remote host (how cool is that!), and how to build local images with the usual autopkgtest customizations/optimizations using adt-build-lxd. I have btrfs running on my laptop, and LXD/autopkgtest automatically use that, so the performance really rocks. Kudos to St phane, Serge, Tycho, and the other LXD authors! The motivation for writing this was to make it possible to move our armhf testing into the cloud (which for $REASONS requires remote containers), but I now have a feeling that soon this will completely replace the existing adt-virt-lxc virt backend, as its much nicer to use. It is covered by the same regression tests as the LXC runner, and from the perspective of package tests that you run in it it should behave very similar to LXC. The one problem I m aware of is that autopkgtest-reboot-prepare is broken, but hardly anything is using that yet. This is a bit complicated to fix, but I expect it will be in the next few weeks. MaaS setup script While most tests are not particularly sensitive about which kind of hardware/platform they run on, low-level software like the Linux kernel, GL libraries, X.org drivers, or Mir very much are. There is a plan for extending our automatic tests to real hardware for these packages, and being able to run autopkgtests on real iron is one important piece of that puzzle. MaaS (Metal as a Service) provides just that it manages a set of machines and provides an API for installing, talking to, and releasing them. The new maas autopkgtest ssh setup script (for the adt-virt-ssh backend) brings together autopkgtest and real hardware. Once you have a MaaS setup, get your API key from the web UI, then you can run a test like this: 
  adt-run libpng --- ssh -s maas -- \
     --acquire "arch=amd64 tags=touchscreen" -r wily \
     http://my.maas.server/MAAS 123DEADBEEF:APIkey
The required arguments are the MaaS URL and the API key. Without any further options you will get any available machine installed with the default release. But usually you want to select a particular one by architecture and/or tags, and install a particular distro release, which you can do with the -r/--release and --acquire options. Note that this is not wired into Ubuntu s production CI environment, but it will be. Selectively using packages from -proposed Up until a few weeks ago, autopkgtest runs in the CI environment were always seeing/using the entirety of -proposed. This often led to lockups where an application foo and one of its dependencies libbar got a new version in -proposed at the same time, and on test regressions it was not clear at all whose fault it was. This often led to perfectly good packages being stuck in -proposed for a long time, and a lot of manual investigation about root causes. . These days we are using a more fine-grained approach: A test run is now specific for a trigger , that is, the new package in -proposed (e. g. a new version of libbar) that caused the test (e. g. for foo ) to run. autopkgtest sets up apt pinning so that only the binary packages for the trigger come from -proposed, the rest from -release. This provides much better isolation between the mush of often hundreds of packages that get synced or uploaded every day. This new behaviour is controlled by an extension of the --apt-pocket option. So you can say 
  adt-run --apt-pocket=proposed=src:foo,libbar1,libbar-data ...
and then only the binaries from the foo source, libbar1, and libbar-data will come from -proposed, everything else from -release. Caveat:Unfortunately apt s pinning is rather limited. As soon as any of the explicitly listed packages depends on a package or version that is only available in -proposed, apt falls over and refuses the installation instead of taking the required dependencies from -proposed as well. In that case, adt-run falls back to the previous behaviour of using no pinning at all. (This unfortunately got worse with apt 1.1, bug report to be done). But it s still helpful in many cases that don t involve library transitions or other package sets that need to land in lockstep. Unified testbed setup script There is a number of changes that need to be made to testbeds so that tests can run with maximum performance (like running dpkg through eatmydata, disabling apt translations, or automatically using the host s apt-cacher-ng), reliable apt sources, and in a minimal environment (to detect missing dependencies and avoid interference from unrelated services these days the standard cloud images have a lot of unnecessary fat). There is also a choice whether to apply these only once (every day) to an autopkgtest specific base image, or on the fly to the current ephemeral testbed for every test run (via --setup-commands). Over time this led to quite a lot of code duplication between adt-setup-vm, adt-build-lxc, the new adt-build-lxd, cloud-vm-setup, and create-nova-image-new-release. I now cleaned this up, and there is now just a single setup-commands/setup-testbed script which works for all kinds of testbeds (LXC, LXD, QEMU images, cloud instances) and both for preparing an image with adt-buildvm-ubuntu-cloud, adt-build-lx[cd] or nova, and with preparing just the current ephemeral testbed via --setup-commands. While this is mostly an internal refactorization, it does impact users who previously used the adt-setup-vm script for e. g. building Debian images with vmdebootstrap. This script is now gone, and the generic setup-testbed entirely replaces it. Misc Aside from the above, every new version has a handful of bug fixes and minor improvements, see the git log for details. As always, if you are interested in helping out or contributing a new feature, don t hesitate to contact me or file a bug report.

Next.

Previous.