I am working in software development field for more then 5 years. During that period I came across of other developers which vast majority can be described as terrible people. That is what sets probably difference between hacker mind and average software engineer. People tend to think if they have position of software developer they are god-like and should be treated like that. Hackers know who they are and tend to be Socrates-wise "I know that I know nothing". As student of mechanical engineering (majoring in mechatronics) I didn't have much of programming so naturally I had some personal difficulties to grasp software field but once I got in I stopped almost totally doing my student duties as it didn't cross path with my love towards hacking. And now after that many years in software field, in which I mostly worked as freelancer for other people and companies (where it wasn't unusual that I got hired personally by a guy who couldn't or didn't know how to finish his work, I hand the code which he would present as his own work, but I don't mind as I needed the money to get on) and my work ended as proprietary - I am just fed up with all this nonsense. Someone would think what is problem there - well for me it is, as being part of Debian and wider Free software ecosystem I know proprietary software is not fostering better society in all together so it made me a bit unhappy about it. The thing that actually is really annoying me are asshole so-called software engineers and the most - bad behavior towards users. Users should be treated as supporters of your software and if proprietary company treats their users as shit then it's obvious that they don't care for the users but rather for the money and only money. Proprietary isn't here the only bad thing, open source community has it share of terrible responses - which comes naturally if you take the sheer amount of people in open source. I for instance was called on Debian IRC a peasant and even I sheep in my early days - but Debian as all communities has its bad seeds and sometimes even non-Debianities trolls hop on our channels. I didn't let myself put down because I personally got the chance to meet many of them and they change the flow of my life at that point. I am unhappy at the current state of things worldwide and personally but I have now made a decision that I am shifting towards being technical customer support engineer or for me better named as Happiness Hero. Why, one would ask because as software engineer you would make more? Well, firstly I don't care that much about money (or to be even more precise, as capitalism has made a sick money-grabbing society, you can never pay me enough as I am more worth then entire capitalistic system so in the end I don't care that much how I am paid) and secondly happiness hero can earn more then software engineer which latest #talkpay hashtag on Twitter showed. I have made IMHO many good pieces of code that will never be contributed as mine, but I achieved no happiness and rather achieved few months ago a rather bad burnout. My shift towards happiness hero should look like this - I am pretty well prepared technically for that position, I have experience with helping users and in general with people and I tend to be pretty good at that (but that we would need to ask them) and with part that I enjoy helping people to debug their technical problem so they can spend more time on others things (read family, creativity, hobbies) is something I believe will put me well into position to achieve happiness so I can create for myself time to finish damn faculty (where TBH professors showed that they can be one of the worst enemies of their students - and I am highly doubting in this generic way of schooling for centuries) and if I get some decent paycheck with this capitalistic system I will have a chance to sponsor a poor family that has child talented in artistic field - or to say SHARE something material for cultural and emotional gain. Take that capitalism! And the best part, I will start to believe more in myself so I have already few offers for that kind of position but am waiting for few more and I am screening companies to find one that fits me best - shares values with mine and has a great interaction with their users. One of the companies that makes me want to screen more companies is Buffer. When you read this and this you will see their way of life and perks are really something every company should do - then you will understand why I want to give my self a chance which I believe I finally deserved. And, no, I didn't apply for Buffer as I don't use their service nor did have time to read the two books needed for applying (which I think is really a great thing to foster people into getting more knowledge). I can only wish them best luck in future. I am currently doing one small software development thing (freelance) for next two weeks and after that I am hoping to get enough good offer to start with happiness job and retreat from software development for some good amount of time. That doesn't mean I will cease my Free software activities - on contrary I think I will have more time and even more will to work on Free software development in my spare time. So if anyone or any company that works with open source community has a position for happiness hero, get in touch with me and we will have a good talk. End point - I want to grow and become who am I, and not who I should be according to others.

This is the first blog post of a series I will be writing about my experiences contributing to Debian for Google Summer of Code 2015.

A bit about myself I m a Spanish student doing a master s in Computer Science in Barcelona. I graduated on Electrical Engineering (we call it Telecommunications here). I ve always been interested in computing and programming and I have worked on several projects on my own using C, python and go. My main interests in the field are programming languages, security, cryptography, distributed and decentralized systems and embedded systems. I m an advocate of free software, I try to use it as much as I m able to in my devices and also try to convince my friends and family of its benefits. I have been using GNU/Linux for nearly ten years as my main operating system and I have tried several *BSD s recently. One of my latest personal projects is a gameboy emulator written in C (still work in progress) which already plays many games (without sound though) . You can find other minor projects in my github page (I try to publish all the code I write online, under free software licence) After so many years of using free software and benefiting from it, I thought it was about time to contribute back! That s why I gave GSoC a try and applied to work on the Reproducible Builds project for Debian :) And I got accepted!

Reproducible Builds The idea behind this project is that currently many packages aren t built in a reproducible manner; that is, they contain timestamps, building machine name, unique IDs, and results from other processes that happen differently between machines, like file ordering in compressed files. The project aims to patch all the Debian packages / the building scripts in order to generate the same binary (bit by bit) independently of the machine, timezone, etc where it is built. This way, a cryptographic hash of the built package can be distributed and many people can rebuild the package to verify that the binary in the repositories indeed corresponds to the right source code by means of comparing the hash.

Motivation One of the main advantages of the free software is that source code is available for peer review. This makes it easier for users to trust their software, as they can check the source to verify that the program is not doing anything bad. Even if the user doesn t do that, they can trust the wider community with that task. But many distributions serve packages in binary form, so how do we know that the binary comes from the publicly available source code? The current solution is that the developers who build the packages sign them cryptographically; but this lands all the trust to the developer and the machines used for building. I became interested in this topic with a very nice talk given at 31c3 by Mike Perry from Tor and Seth Schoen from the EFF. They focused on reproducible builds applied to the tor browser bundle, showing a small demo of how a building machine could be compromised to add hidden functionalities when compiling code (so that the developer could be signing a compromised package without their knowledge).

Benefits There are two main groups who benefit with reproducible builds:

For users The user can be more secure when installing packages in binary form since they don t need to trust a specific developer or building machine. Even if they don t rebuild the package by themselves to verify it, there would be others doing so, who will easily alert the community when the binary doesn t match the source code.

For developers The developer no longer has the responsibility of using his identity to sign the package for wide distribution, nor is that much responsible of the damage to users if their machine is compromised to alter the building process, since the community will easily detect it and alert them. This later point is specially useful with secure and privacy aware software. The reason is that there are many powerful organizations around the world with interest on having backdoors in widely used software, be it to spy on users or to target specific groups of people. Considering the amount of money these organizations have for such purposes, it s not hard to imagine that they could try to blackmail developers into adding such backdoors on the built packages. Or they could try to compromise the building machine. With reproducible builds the developer is safer, as such attack is no longer useful.

Reproducible Builds in Debian The project kicked-off at Debian at mid 2013 , leaded by Lunar and soon followed by many other developers (h01ger, deki, mapreri, ). Right now about 80% of the packages in the unstable branch of Debian can be built reproducibly. The project is very active, with many developers sending patches every week. A machine running Jenkins (which was set up at the end of 2012 for other purposes) is being used since late 2014 to continuously build packages in different settings to check if they are built reproducibly or not. In order to analyze why packages fail to build reproducibly, a tool called debbindiff has been developed, which is able to output in text or html form a smart diff of two builds. Another tool called strip-nondeterminism has been developed to remove non-determinism from files during the building process. For this GSoC I plan on helping improving these tools (mainly debbindiff), write many patches to achieve reproducibility in more packages and write documentation about it. Some of the packages fail to build reproducibly due to specifics of their building processes, whereas others fail due to the usage of toolchains that add non-determinism. I ll focus more on the later ones in order to improve the state more packages. akira will also be working on this project for this GSoC. Finally, I just want to add that I m looking forward to contribute to Debian, meet the community and learn more about the internals of this awesome distribution!

I have spent today making my first ever batch of Kimchi. I have been documenting it in photos as I go, but thought I'd write up what I did so that if anyone else fancies having a go too, we can compare results. For a start, this recipe is nowhere near "traditional" because I don't have access to certain ingredients such as glutinous rice flour. I'm sure if I searched in many of the asian supermarkets around the city centre I could find it, but I'm lazy so I didn't even try. I am not writing this up as a traditional recipe because I'm kinda making it up as I go along, with hints from various sources including the great and wonderful Maangchi whose YouTube channel I follow. Observant readers or followers of Maangchi will recognise the recipe as being close to her Easy Kimchi recipe, however since I'm useless, it won't be exact. If this batch turns out okay then I'll write it up as a proper recipe for you to follow. I started off with three Chinese Leaf cabbages which seemed to be about 1.5kg or so once I'd stripped the less nice outer leaves, cored and chopped them. I then soaked and drained the cabbage in cold water... ...before sprinkling a total of one third of a cup of salt over the cabbage and mixing it to distribute the salt. Then I returned to the cabbage every 30 minutes to re-mix it a total of three times. After the cabbage had been salted for perhaps 1h45m or so, I rinsed it out. Maangchi recommends washing the cabbage three times so that's what I did before setting it out to drain in a colander. Maangchi then calls for the creation of a porridge made from sweet rice flour which it turns out is very glutinous. Since I lack the ability to get that flour easily I substituted cornflour which I hope will be okay and then continued as before. One cup of water, one third of a cup of cornflour was heated until it started to bubble and then one sixth of a cup of sugar was added. Stirred throughout, once it went translucent I turned the heat off and proceeded. One half of a red onion, a good thumb (once peeled) of ginger, half a bulb of garlic and one third of a cup of fish sauce went into a mini-zizzer. I then diagonal-chopped about five spring onions, and one leek, before cutting a fair sized carrot into inch long pieces before halving and then thinly slicing it. Maangchi calls for julienned carrots but I am not that patient. Into the cooled porridge I put two thirds of a cup of korean hot pepper flakes (I have the coarse, but a mix of coarse and fine would possibly be better), the zizzed onion/garlic/ginger/fish mix and the vegetables... ...before mixing that thoroughly with a spatula. Next came the messy bit (I put latex gloves on, and thoroughly washed my gloved hands for this). Into my largest mixing bowl I put a handful of the drained cabbage into the bowl and a handful of the pepper mix. Thoroughly mixing this before adding another handful of cabbage and pepper mix I repeated until all the cabbage and hot pepper mixed vegetables are well combined. I really got your arms into it, squishing it around, separating the leek somewhat, etc. As a final task, I scooped the kimchi into a clicklok type box, pressing it down firmly to try and remove all the air bubbles, before sealing it in for a jolly good fermenting effort. I will have to remove a little tonight for our dinner (beef strips marinated in onion, ginger and garlic, with kimchi on rice) but the rest will then sit to ferment for a bit. Expect a part-2 with the report from tonight's dinner and a part-3 with the results after fermentation.

---

As an aside, I got my hot pepper flakes from Sous Chef who, it turns out, also stock glutinous rice flour -- I may have to get some more from them in the future. (#notsponsored)

So, following the Jessie release, and after a quick approval by the release team for the 4.12 transition, we've uploaded Xfce 4.12 to sid and have asked the RT to schedule the relevant binNMUs for the libxfce4util and xfce4-panel reverse dependencies. It went apparently well (besides some hickups here and there, lilke some lag on sparc, and some build-failulres on hurd). So Xfce 4.12 is now in sid, and should migrate to Stretch in the following weeks, provided nothing release critical is found.

This evening I've been mostly playing with removing duplicate content. I've had this idea for the past few days about object-storage, and obviously in that context if you can handle duplicate content cleanly that's a big win. The naive implementation of object-storage involves splitting uploaded files into chunks, storing them separately, and writing database-entries such that you can reassemble the appropriate chunks when the object is retrieved. If you store chunks on-disk, by the hash of their contents, then things are nice and simple. The end result is that you might upload the file /etc/passwd, split that into four-byte chunks, and then hash each chunk using SHA256. This leaves you with some database-entries, and a bunch of files on-disk:

/tmp/hashed/ef267892ee080862c96a8d2d05de62f48e20f0875f27379e7d58c73ea4455bf1
/tmp/hashed/a378977155fb42bb006496321cbe31f74cbda803c3f6ca590f30e76d1afad921
..
/tmp/hashed/3805b0245bc8375be7125ae228eef711552ac082ffb9bf8756e2964a2393a9de

In my toy-code I wrote out the data in 4-byte chunks, which is grossly ineffeciant. But the value of using such small pieces is that there is liable to be a lot of collisions, and that means we save-space. It is a trade-off. So the main thing I was experimenting with was the size of the chunks. If you make them too small you lose I/O due to the overhead of writing out so many small files, but you gain because collisions are common. The rough testing I did involved using chunks of 16, 32, 128, 255, 512, 1024, 2048, and 4096 bytes. As sizes went up the overhead shrank, but also so did the collisions. Unless you could handle the case of users uploading a lot of files like /bin/ls which are going to collide 100% of the time with prior uploads using larger chunks just didn't win as much as I thought they would. I wrote a toy server using Sinatra & Ruby, which handles the splitting/hashing/and stored block-IDs in SQLite. It's not so novel given that it took only an hour or so to write. The downside of my approach is also immediately apparent. All the data must live on a single machine - so that reassmbly works in the simple fashion. That's possible, even with lots of content if you use GlusterFS, or similar, but it's probably not a great approach in general. If you have large capacity storage avilable locally then this might would well enough for storing backups, etc, but .. yeah.

Debsources (http://sources.debian.net) is still under active development. We recently had a Gnome Outreachy intern, Jingjie Jiang, and we're about to work with 2 GSoC students, Cl ment Schreiner and Orestis Ioannou. I will present here the GitHub mirror we've set up, in order to allow external pull requests to be submitted, and to use the continous integration service provided by Travis-CI. GitHub and Travis-CI Debsources' source code is hosted on Debian's git servers, and from there is mirrored to GitHub. Every time a commit is pushed (to master or other branches) or a pull request is open, the test suite will be automatically run on Travis-CI, and the result (tests pass or don't) is displayed on GitHub. This allows us to quickly filter external contributions (when they are submitted on GitHub), and be sure everything works with our setup, before reviewing work. Travis-CI runs the tests on OpenVZ containers. The complete infrastructure was a bit challenging to setup, but as we now have a Docker recipe to quicly begin to hack on Debsources, most of the work could be done using the Dockerfile instructions. In average, a run on Travis-CI (which includes git cloning the code and test data, setup the server, and run the tests suite) takes 7 minutes, which is an ok amount of time to wait for before submitting a pull request, in my opinion. Bugs discovered in the process Setting up this continuous integration infrastructure made me discover a few bugs. Python magic does black magic Debsources runs fine on Debian (not surprisingly), but I got tricked by black magic when I tried to run it on Ubuntu (which is the OS run in Travis-CI's containers). We use the magic library to guess the type of files we're dealing with, for instance when we need to decide between rendering a file (for text files) or downloading it (for binary files). Here comes the tricky part: the Python bindings for libmagic are not the same in Debian and Pypi. Debsources uses Debian package python-magic, which is not in Ubuntu 12.04. Moreover, there's no Python egg for it on Pypi, which has however another package (called magic) which provides a different API. I solved this with a dirty hack, using the fact python-magic lies in a single file: It simply downloads the library, saves it in a temporary folder and includes it in the Python path. Let's see for how long it works before everything breaks! Size of a directory One test in the suite was ensuring the information returned by ls -l on a directory and stored in the DB was the right information. Inode metadata was tested, such as name, permissions, type, or size. Interestingly enough, the size of a directory was tested, and expected to be 4096 bytes. The size of a directory actually depends on the filesystem in use, and on the number of files this directory contains. We often see 4096 because it's the size of a not-too-big directory on ext4. Travis-CI doesn't use ext4: Simfs is a container filesystem for OpenVZ, on which directories have different sizes than on ext4: Directory sizes are not even powers of 2. Hence I changed the test to not check directory sizes. Hopefully this will help to make Debsources work on more filesystems! An empty file is hiding Last but not least, because this bug is still open in the wild. A file, which appears to be empty, is not taken into account by Debsources' updater. This file is sources/non-free/m/make-doc-non-dfsg/4.0-2/.pc/applied-patches. It is present in the filesystem in the container, is not the only empty file over there, but still doesn't appear in the database, and make fail the test which counts files. The test has been commented out (booooooh), so that we still can use Travis-CI's platform for our GSoC students, before it's fixed. Conclusion Making Debsources run automatically on a different platform as the one we usually use permitted us to spot bugs, write dirty hacks, and expand the filesystems it's supposed to run on. Now, let's hope the continuous integration will help our GSoC students, and let's wish them good luck!

Today at 18:00 UTC (this evening for me) there will be a Debian Publicity Team IRC meeting (open meeting, everybody invited), and I m very happy because it will be the first meeting that I know of, since I joined the team (years!). Being part of the Publicity team There are many tasks handled by Publicity, and when I joined, I supposed that I was going to be part of a team with many members and well structured. And it was true but not as I imagined. Publicity is a great team, in the sense that it accepts contributions from many people, and the few core members do an amazing work: on one side, get things done; on the other side, integrate all those occasional contributions from the wider community. But there are fewer core contributors than what one would expect by the output of the team. I would say we are maximum 10 people (out of 353 voters, 1033 Debian Developers, and 1197 contributors in the Debian Community in 2015). And as far as I know, everybody is member of some other teams too (I m a translator, others in website team, sysadmins, packaging teams and now we have a member sharing Publicity membership with DPL-ship!). Organisation around the tasks Publicity regular tasks (announcements, the newsletter Debian Project News , posting in social networks and in bits.debian.org, and other ) are all well defined and documented, in order to allow anybody jump in and help, and this is great, because it ensures a way for contributions to arrive the wide audience from the very first day: you pick something, you follow the instructions, and you re done. I love this approach, because I tend to prefer to follow instructions than to create something, and my Debian time is made of small chunks at random days/times. But sometimes I feel that we all work alone , in something like a cold, robotic do-ocracy, and I also wonder how many people don t contribute or become regular contributors just because they don t understand the procedures, or they don t like them, or other reasons IRC meeting IRC is something that I use only for contributing to free software, not in other parts of my life. I like IRC, it s productive and fun, but I m not always there, and I don t save logs when I am idle, and I usually prefer email for communication. However, I try to be more present in the Debian IRC channels of the teams where I contribute, because I ve learned that it plays a big role in feeling at home in Debian . Currently you can find me in #debconf-team #debconf15-germany #debian-i18n #debian-l10n-spanish #debian-publicity #debian-women and #debian-www . I ve attended some meetings in IRC (MediaGoblin monthly meetings, and DebConf15 meetings) and I ve learned about MeetBot and more or less how to chair an IRC meeting. Today it will be my first time chairing, it s hard to emulate so great chairs as Marga or Chris Webber, bu I hope I do it decently, and we all have a nice time knowing each other and sharing ideas for the Publicity team. Want to attend? All the details (when, where, agenda ) in the wiki page of the meeting. See you in a few hours!
Filed under: My experiences and opinion Tagged: Communities, Contributing to libre software, Debian, English, Free Software, IRC, libre software, Project Management

Debian Jessie has been released on April 25th, 2015. This has opened the Stretch development cycle. Reactions to the idea of making Debian build reproducibly have been pretty enthusiastic. As the pace is now likely to be even faster, let's see if we can keep everyone up-to-date on the developments. Before the release of Jessie The story goes back a long way but a formal announcement to the project has only been sent in February 2015. Since then, too much work has happened to make a complete report, but to give some highlights:

• New variations are now tested: umask, kernel version, domain name, and timezone. We might only be missing CPU type and current date now.
• Many improvements to the test system on jenkins.debian.net and the pages showing the results.
• Now not only packages from unstable are tested but also those in testing and experimental.
• When rescheduling packages for testing, the build products can be kept and the IRC channel gets a notification when its over.
• binutils version 2.25-6 is now built with the --enable-deterministic-archives flag. Making ar, strip and others create deterministic static libraries.
• Number of identified issues has grown from about 80 to 123 today.

Lunar did a pretty improvised lightning talk during the Mini-DebConf in Lyon. This past week It seems changes were pilling behind the curtains given the amount of activity that happened in just one week. Toolchain fixes

• Niels Thykier uploaded debhelper/9.20150501 which includes fixes to dh_makeshlibs (#774100), dh_icons (#774102), dh_usrlocal (#775020). Patches written by Lunar.
• Helmut Grohne uploaded doxygen/1.8.9.1-3 which will not generate timestamps in HTML by default. Kudos to akira for bringing the issue upstream.
• Kenneth J. Pronovici uploaded epydoc/3.0.1+dfsg-6 adding a --no-include-build-time option. Patch by Jelmer Vernooij.
• David Pr vot uploaded php-apigen/2.8.1+dfsg-2 which now has reproducible output.
• C dric Boutillier uploaded ruby-prawn/2.0.1+dfsg-1 which now produce a deterministic output when using gradients. Patch by Lunar.
• Jelmer Vernooij uploaded samba/2:4.1.17+dfsg-4 which contains a patch by Matthieu Patou making the output of pidl (from libparse-pidl-perl) reproducible.
• Dmitry Shachnev uploaded sphinx/1.3.1-1 in experimental which should produce deterministic output. The original patch from Chris Lamb has inspired the upstream fix.
• gregor herrmann uploaded libextutils-depends-perl/0.404-1 which makes ExtUtils::Depends output deterministic. Original patch by Reiner Herrmann.
• Niko Tyni uploaded perl/5.20.2-4 which makes the output of Pod::Man reproducible. Nice team work visible on #780259.

We also rebased the experimental version of debhelper twice to merge the latest set of changes. Lunar submitted a patch to add a -creation-date to genisoimage. Reiner Herrmann opened #783938 to request making -notimestamp the default behavior for javadoc. Juan Picca submitted a patch to add a --use-date flag to texi2html. Packages fixed The following packages became reproducible due to changes of their build dependencies: apport, batctl, cil, commons-math3, devscripts, disruptor, ehcache, ftphs, gtk2hs-buildtools, haskell-abstract-deque, haskell-abstract-par, haskell-acid-state, haskell-adjunctions, haskell-aeson, haskell-aeson-pretty, haskell-alut, haskell-ansi-terminal, haskell-async, haskell-attoparsec, haskell-augeas, haskell-auto-update, haskell-binary-conduit, haskell-hscurses, jsch, ledgersmb, libapache2-mod-auth-mellon, libarchive-tar-wrapper-perl, libbusiness-onlinepayment-payflowpro-perl, libcapture-tiny-perl, libchi-perl, libcommons-codec-java, libconfig-model-itself-perl, libconfig-model-tester-perl, libcpan-perl-releases-perl, libcrypt-unixcrypt-perl, libdatetime-timezone-perl, libdbd-firebird-perl, libdbix-class-resultset-recursiveupdate-perl, libdbix-profile-perl, libdevel-cover-perl, libdevel-ptkdb-perl, libfile-tail-perl, libfinance-quote-perl, libformat-human-bytes-perl, libgtk2-perl, libhibernate-validator-java, libimage-exiftool-perl, libjson-perl, liblinux-prctl-perl, liblog-any-perl, libmail-imapclient-perl, libmocked-perl, libmodule-build-xsutil-perl, libmodule-extractuse-perl, libmodule-signature-perl, libmoosex-simpleconfig-perl, libmoox-handlesvia-perl, libnet-frame-layer-ipv6-perl, libnet-openssh-perl, libnumber-format-perl, libobject-id-perl, libpackage-pkg-perl, libpdf-fdf-simple-perl, libpod-webserver-perl, libpoe-component-pubsub-perl, libregexp-grammars-perl, libreply-perl, libscalar-defer-perl, libsereal-encoder-perl, libspreadsheet-read-perl, libspring-java, libsql-abstract-more-perl, libsvn-class-perl, libtemplate-plugin-gravatar-perl, libterm-progressbar-perl, libterm-shellui-perl, libtest-dir-perl, libtest-log4perl-perl, libtext-context-eitherside-perl, libtime-warp-perl, libtree-simple-perl, libwww-shorten-simple-perl, libwx-perl-processstream-perl, libxml-filter-xslt-perl, libxml-writer-string-perl, libyaml-tiny-perl, mupen64plus-core, nmap, openssl, pkg-perl-tools, quodlibet, r-cran-rjags, r-cran-rjson, r-cran-sn, r-cran-statmod, ruby-nokogiri, sezpoz, skksearch, slurm-llnl, stellarium. The following packages became reproducible after getting fixed:

• berkeley-abc/1.01+20141105hg5b5af75+dfsg-3 uploaded by Ruben Undheim, original patch by Johann Klammer.
• binutils/2.25-7 by Matthias Klose.
• bitmap-mule/8.5+0.20030825.0433-16 uploaded by Tatsuya Kinoshita, original patch by Chris Lamb.
• blobby/1.0-2 by Felix Geyer.
• codelite/7.0+dfsg-2 by James Cowgill.
• conkeror/1.0~~pre-1+git150409-1 by Axel Beckert.
• convmv/1.15-1 by Christian Perrier.
• cpl/6.6~b-1 by Ole Streicher.
• deheader/1.1-2 by Reiner Herrmann.
• dict-foldoc/20150318-1 uploaded by Iustin Pop, original patch by Chris Lamb.
• ding/1.8-1 by Roland Rosenfeld.
• doc-rfc/20150425-1 by Iustin Pop.
• doxygen/1.8.9.1-3 by Helmut Grohne.
• eureka/1.07-1 by Fabian Greffrath.
• eximdoc4/4.85-2 uploaded by Andreas Metzler, original patch by Chris Lamb.
• flashproxy/1.7-3 by Ximin Luo.
• heimdal/1.6~rc2+dfsg-10 by Jelmer Vernooij.
• init-system-helpers/1.23 by Martin Pitt original patch by Lunar.
• ldb/2:1.1.20-2 by Jelmer Vernooij.
• libalien-wxwidgets-perl/0.67+dfsg-1 uploaded by gregor herrmann, original patch by Chris Lamb.
• libcairo-perl/1.105-1 by intrigeri.
• libclass-methodmaker-perl/2.24-1 uploaded by gregor herrmann, original patch by Chris Lamb.
• libcommon-sense-perl/3.73-3 uploaded by gregor herrmann, original patch by Chris Lamb.
• libelixirfm-perl/1.1.976-4 uploaded by gregor herrmann, original patch by Chris Lamb.
• libgnome2-perl/1.045-3 by intrigeri.
• libjs-jcrop/0.9.12+dfsg-2 uploaded by David Pr vot, original patch by Chris Lamb.
• liblas/1.8.0-2 by Bas Couwenberg.
• libopengl-perl/0.6704+dfsg-1 uploaded by gregor herrmann, original patch by Chris Lamb.
• libparse-recdescent-perl/1.967009+dfsg-2 uploaded by gregor herrmann, original patch by Reiner Herrmann.
• librasterlite/1.1g-5 by Bas Couwenberg.
• libterm-size-perl-perl/0.029-2 uploaded by gregor herrmann, original patch by Chris Lamb.
• libvdpau/1.1-1 by Andreas Beckmann.
• libxml-sax-expatxs-perl/1.33-2 uploaded by gregor herrmann, original patch by Chris Lamb.
• lynx-cur/2.8.9dev4-2 by Axel Beckert.
• mapcache/1.2.1-3 by Bas Couwenberg.
• mdbtools/0.7.1-4 by Jean-Michel Nirgal Vourg re.
• mopidy uploaded by Stein Magnus Jodal, fixed by upstream.
• objenesis/2.1-1 by Markus Koschany.
• opendkim/2.10.1-2 uploaded by Scott Kitterman, original patch by Reiner Herrmann.
• pktools/2.6.3-1 by Bas Couwenberg.
• posh/0.12.4 uploaded by Clint Adams, original patch by Chris Lamb.
• prboom-plus/2:2.5.1.4~svn4425+dfsg1-1 by Fabian Greffrath.
• qlandkartegt/1.8.1+ds-1 by Bas Couwenberg.
• readosm/1.0.0d-1~exp2 by Bas Couwenberg.
• robocode/1.9.2.4-1 by Markus Koschany.
• ruby-prawn/2.0.1+dfsg-1 uploaded by C dric Boutillier, original patch by Lunar.
• sane-backends/1.0.25+git20150425-1 by J rg Frings-F rst.
• scoop/0.7.1-3 by Daniel Stender.
• springlobby/0.218-1 by Markus Koschany.
• subvertpy/0.9.2-1 by Jelmer Vernooij.
• t-prot/3.4-2 by Axel Beckert.
• talloc/2.1.2-3 by Jelmer Vernooij.
• tdb/1.3.4-2 by Jelmer Vernooij, patch now applied by upstream.
• tk-html3/3.0~fossil20110109-5 by Ole Streicher.
• tox/1.9.2-2 uploaded by Barry Warsaw original patch by Reiner Herrmann.
• trove3/3.0.3-2 by Erich Schubert.
• txt2man/1.5.6-2 uploaded by Joao Eriberto Mota Filho, initial patch by Jonathan Wiltshire.
• units/2.11-2 by Stephen Kitt.
• win32-loader/0.7.10 upload by Didier Raboud, original patch by Lunar.
• zec/0.12-3 uploaded by Clint Adams, original patch by Chris Lamb.
• zomg/0.8-1 uploaded by Clint Adams, original patch by Chris Lamb.

Some uploads fixed some reproducibility issues but not all of them:

• ada-reference-manual/1:2012.2-4 by Nicolas Boulenguez.
• adacontrol/1.16r11-3 by Nicolas Boulenguez.
• aspectj/1.8.5-1 by Emmanuel Bourg.
• binutils-m68hc1x/1:2.18-4 uploaded by Riku Voipio, original patch by Chris Lamb.
• debianutils/4.5) uploaded by Clint Adams, original patch by Lunar.
• ioquake3/1.36+u20150412+dfsg1-1 by Simon McVittie.
• libsdl1.2/1.2.15-11 uploaded by Manuel A. Fernandez Montecelo, original patch by Chris Lamb; currently FTBFS.
• libsdl2/2.0.2+dfsg1-7 by Manuel A. Fernandez Montecelo, original patch by Chris Lamb; currently FTBFS.
• nedit/1:5.6a-2 by Paul Gevers.
• openarena-085-data/0.8.5split-4 by Simon McVittie.
• openarena-data/0.8.5split-4 by Simon McVittie.
• openarena/0.8.8-12 by Simon McVittie.
• openchange/1:2.2-6 by Jelmer Vernooij.
• puredata/0.46.6-1 by IOhannes m zm lnig.
• pyexiv2/0.3.2-8 by Michal iha ; currently FTBFS.
• quakespasm/0.90.0-2 by Stephen Kitt.
• sed/4.2.2-5 by Clint Adams, patch by Lunar.
• v4l2loopback/0.8.0-5 uploaded by IOhannes m zm lnig, original patch by Chris Lamb.
• vim/2:7.4.712-1 uploaded by James McCoy, original patch by Reiner Herrmann.
• zookeeper/3.4.6-4 by Emmanuel Bourg.

Patches submitted which did not make their way to the archive yet:

• #783882 on jodconverter by Reiner Herrmann: tell javadoc to stop writing timestamps.
• #783885 on bind9 by Reiner Herrmann: pass -DNO_VERSION_DATE to build system.
• #783688 on serverstats by Chris Lamb: fix permissions.
• #783453 on cdbackup by Chris Lamb: remove build date from version string in binary.
• #783478 on texi2html by Juan Picca: pass --use-date to texi2html.
• #783933 on imagemagick by Reiner Herrmann: remove timestamps from PNG and use deterministic package build date.
• #783515 on memtest86+ by Lunar: fix embedded mtimes and pass -creation-date to genisoimage.
• #783558 on websvn by Chris Lamb: fix permissions.

Improvements to reproducible.debian.net Mattia Rizzolo has been working on compressing logs using gzip to save disk space. The web server would uncompress them on-the-fly for clients which does not accept gzip content. Mattia Rizzolo worked on a new page listing various breakage: missing or bad debbindiff output, missing build logs, unavailable build dependencies. Holger Levsen added a new execution environment to run debbindiff using dependencies from testing. This is required for packages built with GHC as the compiler only understands interfaces built by the same version. debbindiff development Version 17 has been uploaded to unstable. It now supports comparing ISO9660 images, dictzip files and should compare identical files much faster. Documentation update Various small updates and fixes to the pages about PDF produced by LaTeX, DVI produced by LaTeX, static libraries, Javadoc, PE binaries, and Epydoc. Package reviews Known issues have been tagged when known to be deterministic as some might unfortunately not show up on every single build. For example, two new issues have been identified by building with one timezone in April and one in May. RD and help2man add current month and year to the documentation they are producing. 1162 packages have been removed and 774 have been added in the past week. Most of them are the work of proper automated investigation done by Chris West. Summer of code Finally, we learned that both akira and Dhole were accepted for this Google Summer of Code. Let's welcome them! They have until May 25th before coding officialy begins. Now is the good time to help them feel more comfortable by sharing all these little bits of knowledge on how Debian works.

Gnome Shell, besides providing the main user interface for GNOME 3, is a Javascript shell with bindings to many native interfaces that allow e.g. Window manipulation, graphics rendering and animations, compositing, etc. It also allows developers to write extensions changing Gnome Shell s behavior. Less known is that it is possible to replace the entire Javascript code base that Gnome Shell uses. It can be useful to hack on Gnome Shell itself (no need to fiddle with system files, or, since 3.12, no need to rebuild libgnome-shell.so), but it can also be used to implement a completely new User Interface in Javascript. I m starting to experiment with the latter, because I want to try building a window manager that fits my needs, while keeping away the boring details of EWMH, xinerama, and other X11 things. And because it s fun. But baby steps, first: let s bootstrap a Hello world with Gnome shell.

• Create a directory that will hold your code.
• In that directory, create a ui subdirectory.
• In that ui directory, create a environment.js file, with the following contents:
  

  const Shell = imports.gi.Shell;
  function init()  
    window.global = Shell.Global.get();
   
  

• In the same directory, create a main.js file, with the following contents:
  

  const St = imports.gi.St;
  function start()  
    let text = new St.Label(  text: "Hello, world!"  );
    global.stage.add_actor(text);
    global.stage.show();
   
  

• Run Gnome Shell with your code:
  

  $ GNOME_SHELL_JS=/path/to/parent/of/ui gnome-shell
  

  You may want to run this in a separate X server (I use Xephyr)

I tested this with Gnome Shell 3.14. Trying various older versions, I got different results for reasons I don t know. 3.4 doesn t display anything unless, paradoxically, global.stage.show() is removed, and 3.8 doesn t display anything no matter what. I guess the next step is to go through some Clutter tutorials and transpose them to Javascript. Update: On the other hand, a lot of the window managing is still done by mutter under the hood, which doesn t leave a lot of space for something really different.

In my previous post, I covered how to extend an initramfs to serve as a basis for tests and other purposes. In that example, the initramfs was simply copied off a device after installation. It occurred to me that python-guestfs would be a useful tool to make this step easier. So I ve written the vmextract helper which is currently in the vmdebootstrap upstream git and will make it into the next release of vmdebootstrap for Debian (0.7). Once vmdebootstrap has built an image, various files can be generated or modified during the install operations and some of these files can be useful when testing the image or packages which would be included in an updated image, before release. One example is the initrd built by the process of installing a Debian kernel. Rather than having to mount the image and copy the files manually, the vmextract helper can do it for you, without needing root privileges or any special setup.

$ /usr/share/vmdebootstrap/vmextract.py --verbose --boot \
  --image bbb/bbb-debian-armmp.img \
  --path /boot/initrd.img-3.14-2-armmp \
  --path /lib/arm-linux-gnueabihf/libresolv.so.2

This uses python-guestfs (which will become a Recommended package for vmdebootstrap) to prepare a read-only version of the image in this case with the /boot partition also mounted and copies files out into the current working directory. Note the repeating use of --path to build a list of files to be copied out. To copy out an entire directory (and all sub-directories) as a single tarball, use:

$ /usr/share/vmdebootstrap/vmextract.py --verbose --boot \
  --image bbb/bbb-debian-armmp.img \
  --directory /boot/ \
  --filename bbb-armmp-boot.tgz

If --filename is not specified, the default filename is vmextract.tgz. (vmextract uses gzip compression, just because.) The other point to note is the use of the --boot option to mount the /boot partition as well as the root partition of the image as this example uses the beaglebone-black support which has a boot partition. It s just a little helper which I hope will prove useful if only to avoid both the need for sudo and the need for loopback mount operations with the inherent confusion of calculating offsets. Thanks to the developers of python-guestfs for making this workable in barely 100 lines of python. It uses the same cliapp support as vmdebootstrap, so can be used silently in scripts by omitting the --verbose option. I haven t taken this on to the next step of unpacking the initrd and extending it, but that would just a bit of shell scripting using the files extracted by vmextract.py. Next! Next on the list will be extensions to vmdebootstrap to build live images of Debian, essentially adding Debian Installer to a vmdebootstrap image, which could actually be another python-guestfs helper (mounting read-write this time) to avoid adding lots more code to vmdebootstrap (which has grown to nearly 600 lines of python). That way, we can publish the bare VM images as well as a live conversion and reducing the number of times debootstrap needs to be called.

E-Mail is a pest, a big time killer wasting your and my time each and every day. Of course it is also a valuable tool, one that no one can renounce. So how can it be of more use than trouble? So far I ve followed a no-delete policy when it comes to my mails, since space was not a problem at all. But it developed into a big nasty pile of mails, that brought regular distraction, each time I looked at my inbox. So I decided to adopt the Inbox zero concept. Step 1: Get the pile down My e-mails piled up since years, so I had around 10000 mails in my inbox, with some hundred being unread. I needed to get this pile down and started with the most recent mails, trying to identify clusters of mails, filtering for them and then following these steps:

• prevent: A lot of mails I get are: newsletters and mailinglist posts (e.g. Debian lists and some open source products). For each of them, I decided if I really want them to go to my inbox. If not: unsubscribe.
• file or delete: Do I need it for reference or should it go to trash? I trashed basically every newsletter and mail(s) for which copies exist (e.g. mailinglist posts) and archived everything where I was unsure. It doesn t matter, really. Important is, that the inbox get s down to zero, because that s where you spend your daily time. Your archive folders can be as full as good as your search function is

Since it wasn t possible to decide on a course for every mail (that would be a bit like hoovering in the dessert), I did this only for the first 1000 of mails or so. All mails older than a month were marked read and moved to archive immediately after. Another approach would be to move all files to a folder called DMZ and go to step 2. Step 2: Prepare for implanting some habits Most mails are the opposite of good old hackish perl code: read only. They are easy to act on, when they come around: just archive or delete them. But the rest will be what steals your time. Some mails require action, either immediately or in a while, some wait for a schedule, e.g. flight informations or reservation mails and stuff. Whatever the reason is, you want to keep them around, because they still have a purpose. There are various filing systems for those mails, most of them GTD variants. As a gmail user I found this variant, with multiple inboxes in a special gmail view, interesting and now give it a try. One word about the archive folders. I can highly recommend to reduce the number of folders you archive to as much as possible. Step 3: Get into habit Now to the hard part. Get into habit with acting on your inbox. Do it regularly, maybe every hour or so and be prepared to do quick decisions. Act on any mail immediately, which means either file/delete it, reply to it (if this is what takes less time) or mark it according to your filing system as prepared in step 2. And if no mails arrived, then it s a good moment to review your marked mails if any on them can be further processed. Now let s see weither my inbox will still be zeroed in a month from now.

Common Lisp users are very happy to use Quicklisp when it comes to downloading and maintaining dependencies between their own code and the librairies it is using. Sometimes I am pointed that when compared to other programming languages Common Lisp is lacking a lot in the batteries included area. After having had to package about 50 common lisp librairies for debian I can tell you that I politely disagree with that. And this post is about the tool and process I use to maintain all those librairies. Quicklisp is good at ensuring a proper distribution of all those libs it supports and actually tests that they all compile and load together, so I've been using it as my upstream for debian packaging purposes. Using Quicklisp here makes my life much simpler as I can grovel through its metadata and automate most of the maintenance of my cl related packages. It's all automated in the ql-to-deb software which, unsurprisingly, has been written in Common Lisp itself. It's a kind of a Quicklisp client that will fetch Quicklisp current list of releases with version numbers and compare to the list of managed packages for debian in order to then build new version automatically. The current workflow I'm using begins with using ql-to-deb is to check for the work to be done today:

$ /vagrant/build/bin/ql-to-deb check
Fetching "http://beta.quicklisp.org/dist/quicklisp.txt"
Fetching "http://beta.quicklisp.org/dist/quicklisp/2015-04-07/releases.txt"
update: cl+ssl cl-csv cl-db3 drakma esrap graph hunchentoot local-time lparallel nibbles qmynd trivial-backtrace
upload: hunchentoot

After careful manual review of the automatic decision, let's just update all what check decided would have to be:

$ /vagrant/build/bin/ql-to-deb update
Fetching "http://beta.quicklisp.org/dist/quicklisp.txt"
Fetching "http://beta.quicklisp.org/dist/quicklisp/2015-04-07/releases.txt"
Updating package cl-plus-ssl from 20140826 to 20150302.
     see logs in "//tmp/ql-to-deb/logs//cl-plus-ssl.log"
Fetching "http://beta.quicklisp.org/archive/cl+ssl/2015-03-02/cl+ssl-20150302-git.tgz"
Checksum test passed.
     File: "/tmp/ql-to-deb/archives/cl+ssl-20150302-git.tgz"
      md5: 61d9d164d37ab5c91048827dfccd6835
Building package cl-plus-ssl
Updating package cl-csv from 20140826 to 20150302.
     see logs in "//tmp/ql-to-deb/logs//cl-csv.log"
Fetching "http://beta.quicklisp.org/archive/cl-csv/2015-03-02/cl-csv-20150302-git.tgz"
Checksum test passed.
     File: "/tmp/ql-to-deb/archives/cl-csv-20150302-git.tgz"
      md5: 32f6484a899fdc5b690f01c244cd9f55
Building package cl-csv
Updating package cl-db3 from 20131111 to 20150302.
     see logs in "//tmp/ql-to-deb/logs//cl-db3.log"
Fetching "http://beta.quicklisp.org/archive/cl-db3/2015-03-02/cl-db3-20150302-git.tgz"
Checksum test passed.
     File: "/tmp/ql-to-deb/archives/cl-db3-20150302-git.tgz"
      md5: 578896a3f60f474742f240b703f8c5f5
Building package cl-db3
Updating package cl-drakma from 1.3.11 to 1.3.13.
     see logs in "//tmp/ql-to-deb/logs//cl-drakma.log"
Fetching "http://beta.quicklisp.org/archive/drakma/2015-04-07/drakma-1.3.13.tgz"
Checksum test passed.
     File: "/tmp/ql-to-deb/archives/drakma-1.3.13.tgz"
      md5: 3b548bce10728c7a058f19444c8477c3
Building package cl-drakma
Updating package cl-esrap from 20150113 to 20150302.
     see logs in "//tmp/ql-to-deb/logs//cl-esrap.log"
Fetching "http://beta.quicklisp.org/archive/esrap/2015-03-02/esrap-20150302-git.tgz"
Checksum test passed.
     File: "/tmp/ql-to-deb/archives/esrap-20150302-git.tgz"
      md5: 8b198d26c27afcd1e9ce320820b0e569
Building package cl-esrap
Updating package cl-graph from 20141106 to 20150407.
     see logs in "//tmp/ql-to-deb/logs//cl-graph.log"
Fetching "http://beta.quicklisp.org/archive/graph/2015-04-07/graph-20150407-git.tgz"
Checksum test passed.
     File: "/tmp/ql-to-deb/archives/graph-20150407-git.tgz"
      md5: 3894ef9262c0912378aa3b6e8861de79
Building package cl-graph
Updating package hunchentoot from 1.2.29 to 1.2.31.
     see logs in "//tmp/ql-to-deb/logs//hunchentoot.log"
Fetching "http://beta.quicklisp.org/archive/hunchentoot/2015-04-07/hunchentoot-1.2.31.tgz"
Checksum test passed.
     File: "/tmp/ql-to-deb/archives/hunchentoot-1.2.31.tgz"
      md5: 973eccfef87e81f1922424cb19884d63
Building package hunchentoot
Updating package cl-local-time from 20150113 to 20150407.
     see logs in "//tmp/ql-to-deb/logs//cl-local-time.log"
Fetching "http://beta.quicklisp.org/archive/local-time/2015-04-07/local-time-20150407-git.tgz"
Checksum test passed.
     File: "/tmp/ql-to-deb/archives/local-time-20150407-git.tgz"
      md5: 7be4a31d692f5862014426a53eb1e48e
Building package cl-local-time
Updating package cl-lparallel from 20141106 to 20150302.
     see logs in "//tmp/ql-to-deb/logs//cl-lparallel.log"
Fetching "http://beta.quicklisp.org/archive/lparallel/2015-03-02/lparallel-20150302-git.tgz"
Checksum test passed.
     File: "/tmp/ql-to-deb/archives/lparallel-20150302-git.tgz"
      md5: dbda879d0e3abb02a09b326e14fa665d
Building package cl-lparallel
Updating package cl-nibbles from 20141106 to 20150407.
     see logs in "//tmp/ql-to-deb/logs//cl-nibbles.log"
Fetching "http://beta.quicklisp.org/archive/nibbles/2015-04-07/nibbles-20150407-git.tgz"
Checksum test passed.
     File: "/tmp/ql-to-deb/archives/nibbles-20150407-git.tgz"
      md5: 2ffb26241a1b3f49d48d28e7a61b1ab1
Building package cl-nibbles
Updating package cl-qmynd from 20141217 to 20150302.
     see logs in "//tmp/ql-to-deb/logs//cl-qmynd.log"
Fetching "http://beta.quicklisp.org/archive/qmynd/2015-03-02/qmynd-20150302-git.tgz"
Checksum test passed.
     File: "/tmp/ql-to-deb/archives/qmynd-20150302-git.tgz"
      md5: b1cc35f90b0daeb9ba507fd4e1518882
Building package cl-qmynd
Updating package cl-trivial-backtrace from 20120909 to 20150407.
     see logs in "//tmp/ql-to-deb/logs//cl-trivial-backtrace.log"
Fetching "http://beta.quicklisp.org/archive/trivial-backtrace/2015-04-07/trivial-backtrace-20150407-git.tgz"
Checksum test passed.
     File: "/tmp/ql-to-deb/archives/trivial-backtrace-20150407-git.tgz"
      md5: 762b0acf757dc8a2a6812d2f0f2614d9
Building package cl-trivial-backtrace

Quite simple. To be totally honnest, I first had a problem with the parser generator library esrap wherein the README documentation changed to be a README.org file, and I had to tell my debian packaging about that. See the 0ef669579cf7c07280eae7fe6f61f1bd664d337e commit to ql-to-deb for details. What about trying to install those packages locally? That's usually a very good test. Sometimes some dependencies are missing at the dpkg command line, so another apt-get install -f is needed:

$ /vagrant/build/bin/ql-to-deb install
sudo dpkg -i /tmp/ql-to-deb/cl-plus-ssl_20150302-1_all.deb /tmp/ql-to-deb/cl-csv_20150302-1_all.deb /tmp/ql-to-deb/cl-csv-clsql_20150302-1_all.deb /tmp/ql-to-deb/cl-csv-data-table_20150302-1_all.deb /tmp/ql-to-deb/cl-db3_20150302-1_all.deb /tmp/ql-to-deb/cl-drakma_1.3.13-1_all.deb /tmp/ql-to-deb/cl-esrap_20150302-1_all.deb /tmp/ql-to-deb/cl-graph_20150407-1_all.deb /tmp/ql-to-deb/cl-hunchentoot_1.2.31-1_all.deb /tmp/ql-to-deb/cl-local-time_20150407-1_all.deb /tmp/ql-to-deb/cl-lparallel_20150302-1_all.deb /tmp/ql-to-deb/cl-nibbles_20150407-1_all.deb /tmp/ql-to-deb/cl-qmynd_20150302-1_all.deb /tmp/ql-to-deb/cl-trivial-backtrace_20150407-1_all.deb
(Reading database ... 79689 files and directories currently installed.)
Preparing to unpack .../cl-plus-ssl_20150302-1_all.deb ...
Unpacking cl-plus-ssl (20150302-1) over (20140826-1) ...
Selecting previously unselected package cl-csv.
Preparing to unpack .../cl-csv_20150302-1_all.deb ...
Unpacking cl-csv (20150302-1) ...
Selecting previously unselected package cl-csv-clsql.
Preparing to unpack .../cl-csv-clsql_20150302-1_all.deb ...
Unpacking cl-csv-clsql (20150302-1) ...
Selecting previously unselected package cl-csv-data-table.
Preparing to unpack .../cl-csv-data-table_20150302-1_all.deb ...
Unpacking cl-csv-data-table (20150302-1) ...
Selecting previously unselected package cl-db3.
Preparing to unpack .../cl-db3_20150302-1_all.deb ...
Unpacking cl-db3 (20150302-1) ...
Preparing to unpack .../cl-drakma_1.3.13-1_all.deb ...
Unpacking cl-drakma (1.3.13-1) over (1.3.11-1) ...
Preparing to unpack .../cl-esrap_20150302-1_all.deb ...
Unpacking cl-esrap (20150302-1) over (20150113-1) ...
Preparing to unpack .../cl-graph_20150407-1_all.deb ...
Unpacking cl-graph (20150407-1) over (20141106-1) ...
Preparing to unpack .../cl-hunchentoot_1.2.31-1_all.deb ...
Unpacking cl-hunchentoot (1.2.31-1) over (1.2.29-1) ...
Preparing to unpack .../cl-local-time_20150407-1_all.deb ...
Unpacking cl-local-time (20150407-1) over (20150113-1) ...
Preparing to unpack .../cl-lparallel_20150302-1_all.deb ...
Unpacking cl-lparallel (20150302-1) over (20141106-1) ...
Preparing to unpack .../cl-nibbles_20150407-1_all.deb ...
Unpacking cl-nibbles (20150407-1) over (20141106-1) ...
Preparing to unpack .../cl-qmynd_20150302-1_all.deb ...
Unpacking cl-qmynd (20150302-1) over (20141217-1) ...
Preparing to unpack .../cl-trivial-backtrace_20150407-1_all.deb ...
Unpacking cl-trivial-backtrace (20150407-1) over (20120909-2) ...
Setting up cl-plus-ssl (20150302-1) ...
dpkg: dependency problems prevent configuration of cl-csv:
 cl-csv depends on cl-interpol; however:
  Package cl-interpol is not installed.
dpkg: error processing package cl-csv (--install):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of cl-csv-clsql:
 cl-csv-clsql depends on cl-csv; however:
  Package cl-csv is not configured yet.
dpkg: error processing package cl-csv-clsql (--install):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of cl-csv-data-table:
 cl-csv-data-table depends on cl-csv; however:
  Package cl-csv is not configured yet.
dpkg: error processing package cl-csv-data-table (--install):
 dependency problems - leaving unconfigured
Setting up cl-db3 (20150302-1) ...
Setting up cl-drakma (1.3.13-1) ...
Setting up cl-esrap (20150302-1) ...
Setting up cl-graph (20150407-1) ...
Setting up cl-local-time (20150407-1) ...
Setting up cl-lparallel (20150302-1) ...
Setting up cl-nibbles (20150407-1) ...
Setting up cl-qmynd (20150302-1) ...
Setting up cl-trivial-backtrace (20150407-1) ...
Setting up cl-hunchentoot (1.2.31-1) ...
Errors were encountered while processing:
 cl-csv
 cl-csv-clsql
 cl-csv-data-table

Let's make sure that our sid users will be happy with the update here:

$ sudo apt-get install -f
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Correcting dependencies... Done
The following packages were automatically installed and are no longer required:
  g++-4.7 git git-man html2text libaugeas-ruby1.8 libbind9-80
  libclass-isa-perl libcurl3-gnutls libdns88 libdrm-nouveau1a
  libegl1-mesa-drivers libffi5 libgraphite3 libgssglue1 libisc84 libisccc80
  libisccfg82 liblcms1 liblwres80 libmpc2 libopenjpeg2 libopenvg1-mesa
  libpoppler19 librtmp0 libswitch-perl libtiff4 libwayland-egl1-mesa luatex
  openssh-blacklist openssh-blacklist-extra python-chardet python-debian
  python-magic python-pkg-resources python-six ttf-dejavu-core ttf-marvosym
Use 'apt-get autoremove' to remove them.
The following extra packages will be installed:
  cl-interpol
The following NEW packages will be installed:
  cl-interpol
0 upgraded, 1 newly installed, 0 to remove and 51 not upgraded.
3 not fully installed or removed.
Need to get 20.7 kB of archives.
After this operation, 135 kB of additional disk space will be used.
Do you want to continue? [Y/n] 
Get:1 http://ftp.fr.debian.org/debian/ sid/main cl-interpol all 0.2.1-2 [20.7 kB]
Fetched 20.7 kB in 0s (84.5 kB/s)
debconf: unable to initialize frontend: Dialog
debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.)
debconf: falling back to frontend: Readline
Selecting previously unselected package cl-interpol.
(Reading database ... 79725 files and directories currently installed.)
Preparing to unpack .../cl-interpol_0.2.1-2_all.deb ...
Unpacking cl-interpol (0.2.1-2) ...
Setting up cl-interpol (0.2.1-2) ...
Setting up cl-csv (20150302-1) ...
Setting up cl-csv-clsql (20150302-1) ...
Setting up cl-csv-data-table (20150302-1) ...

All looks fine, time to sign those packages. There's a trick here, where you want to be sure you're using a GnuPG setup that allows you to enter your passphrase only once, see ql-to-deb vm setup for details, and the usual documentations about all that if you're interested into the details.

$ /vagrant/build/bin/ql-to-deb sign
 signfile /tmp/ql-to-deb/cl-plus-ssl_20150302-1.dsc 60B1CB4E
 signfile /tmp/ql-to-deb/cl-plus-ssl_20150302-1_amd64.changes 60B1CB4E
Successfully signed dsc and changes files
 signfile /tmp/ql-to-deb/cl-csv_20150302-1.dsc 60B1CB4E
 signfile /tmp/ql-to-deb/cl-csv_20150302-1_amd64.changes 60B1CB4E
Successfully signed dsc and changes files
 signfile /tmp/ql-to-deb/cl-db3_20150302-1.dsc 60B1CB4E
 signfile /tmp/ql-to-deb/cl-db3_20150302-1_amd64.changes 60B1CB4E
Successfully signed dsc and changes files
 signfile /tmp/ql-to-deb/cl-drakma_1.3.13-1.dsc 60B1CB4E
 signfile /tmp/ql-to-deb/cl-drakma_1.3.13-1_amd64.changes 60B1CB4E
Successfully signed dsc and changes files
 signfile /tmp/ql-to-deb/cl-esrap_20150302-1.dsc 60B1CB4E
 signfile /tmp/ql-to-deb/cl-esrap_20150302-1_amd64.changes 60B1CB4E
Successfully signed dsc and changes files
 signfile /tmp/ql-to-deb/cl-graph_20150407-1.dsc 60B1CB4E
 signfile /tmp/ql-to-deb/cl-graph_20150407-1_amd64.changes 60B1CB4E
Successfully signed dsc and changes files
 signfile /tmp/ql-to-deb/hunchentoot_1.2.31-1.dsc 60B1CB4E
 signfile /tmp/ql-to-deb/hunchentoot_1.2.31-1_amd64.changes 60B1CB4E
Successfully signed dsc and changes files
 signfile /tmp/ql-to-deb/cl-local-time_20150407-1.dsc 60B1CB4E
 signfile /tmp/ql-to-deb/cl-local-time_20150407-1_amd64.changes 60B1CB4E
Successfully signed dsc and changes files
 signfile /tmp/ql-to-deb/cl-lparallel_20150302-1.dsc 60B1CB4E
 signfile /tmp/ql-to-deb/cl-lparallel_20150302-1_amd64.changes 60B1CB4E
Successfully signed dsc and changes files
 signfile /tmp/ql-to-deb/cl-nibbles_20150407-1.dsc 60B1CB4E
 signfile /tmp/ql-to-deb/cl-nibbles_20150407-1_amd64.changes 60B1CB4E
Successfully signed dsc and changes files
 signfile /tmp/ql-to-deb/cl-qmynd_20150302-1.dsc 60B1CB4E
 signfile /tmp/ql-to-deb/cl-qmynd_20150302-1_amd64.changes 60B1CB4E
Successfully signed dsc and changes files
 signfile /tmp/ql-to-deb/cl-trivial-backtrace_20150407-1.dsc 60B1CB4E
 signfile /tmp/ql-to-deb/cl-trivial-backtrace_20150407-1_amd64.changes 60B1CB4E
Successfully signed dsc and changes files

Ok, with all tested and signed, it's time we upload our packages on debian servers for our dear debian users to be able to use newer and better versions of their beloved Common Lisp librairies:

$ /vagrant/build/bin/ql-to-deb upload
Trying to upload package to ftp-master (ftp.upload.debian.org)
Checking signature on .changes
gpg: Signature made Sat 02 May 2015 05:06:48 PM MSK using RSA key ID 60B1CB4E
gpg: Good signature from "Dimitri Fontaine <dim@tapoueh.org>"
Good signature on /tmp/ql-to-deb/cl-plus-ssl_20150302-1_amd64.changes.
Checking signature on .dsc
gpg: Signature made Sat 02 May 2015 05:06:46 PM MSK using RSA key ID 60B1CB4E
gpg: Good signature from "Dimitri Fontaine <dim@tapoueh.org>"
Good signature on /tmp/ql-to-deb/cl-plus-ssl_20150302-1.dsc.
Uploading to ftp-master (via ftp to ftp.upload.debian.org):
  Uploading cl-plus-ssl_20150302-1.dsc: done.
  Uploading cl-plus-ssl_20150302.orig.tar.gz: done.
  Uploading cl-plus-ssl_20150302-1.debian.tar.xz: done.
  Uploading cl-plus-ssl_20150302-1_all.deb: done.
  Uploading cl-plus-ssl_20150302-1_amd64.changes: done.
Successfully uploaded packages.

Of course the same text or abouts is then repeated for all the other packages. Enjoy using Common Lisp in debian! Oh and remember, the only reason I've written ql-to-deb and signed myself up to maintain those upteens Common Lisp librairies as debian package is to be able to properly package pgloader in debian, as you can see at https://packages.debian.org/sid/pgloader and in particular in the Other Packages Related to pgloader section of the debian source package for pgloader at https://packages.debian.org/source/sid/pgloader. That level of effort is done to ensure that we respect the Debian Social Contract wherein debian ensures its users that it's possible to rebuild anything from sources as found in the debian repositories.

It was quite an experience to partake in the Jessie release and also a rather long Saturday . This post is mostly a time line of how I spent my release day with doing the actual release. I have glossed over some details the post is long enough without these. :) We started out at 8 (UTC) with a final dinstall run, which took nearly 2 hours. It was going to take longer, but we decided to skip the synchronisation to coccia.debian.org (the server hosting the DD-accessible mirror of release.debian.org). The release itself started with the FTP masters renaming the aliases of Squeeze, Wheezy and Jessie to oldoldstable, oldstable and stable respectively. While they worked, the release team reviewed and double checked their work. After an hour (~11), the FTP masters reported that the stable releases were ready for the final review and the SRMs signed the relevant Release files. Then the FTP masters pushed the stable releases to our CD build server, where Steve McIntyre started building the installation images. While Steve started with the CDs, the FTP masters and the release team continued with creating a suite for Stretch. On the FTP/release side, we finished shortly before 12:30. At this point, our last ETA from Steve suggested that the installation media would take another 11 and a half hours to complete. We could have opened for mirror synchronisation then, but we decided to wait for the installation media. At 12:30, there was a long intermission for the release team in the release process. That was an excellent time to improve some of our tools, but that is for another post. :) We slowly started to resume around 22:20, where we tried to figure out when to open for the mirror synchronisation to time it with the installation media. We agreed to start the mirror sync at 23:00 despite the installation media not being completely done then. They followed half an hour later, when Steve reported that the last CD was complete. At this point, all that was left was to update the website and send out the press announcement. Sadly, we were hit by some (minor) issues then. First, I had underestimated the work involved in updating the website. Secondly, we had no one online at the time to trigger an out of band rebuild of the website. Steve and I spent an hour and a half solving website issues (like arm64 and ppc64el not being listed as a part of the release). Unsurprisingly, I decided to expand our the release checklist to be slightly more verbose on this particular topic. My Saturday had passed its 16th hour, when I thought we had fixed all the website issues (of course, I would be wrong) and we would now just be waiting for the an automatic rebuild. I was tempted to just punt it and go to bed, when Paul Wise rejoined us at about 01:25. He quickly got up to speed and offered to take care of the rest. An offer I thankfully accepted and I checked out 15 minutes later at 01:40 UTC. That more or less covers the Jessie release day from my PoV. After a bit of reflection inside the release team, we have found several points where we can improve the process. This part certainly deserves its own post as well, which will also give us some time to flesh out some of the ideas a bit more. :)
Filed under: Debian, Release-Team

Many years ago, a friend of mine calculated how much it would cost to store the sound of all phone calls in Norway, and came up with the cost of around 20 million NOK (2.4 mill EUR) for all the calls in a year. I got curious and wondered what the same calculation would look like today. To do so one need an idea of how much data storage is needed for each minute of sound, how many minutes all the calls in Norway sums up to, and the cost of data storage. The 2005 numbers are from digi.no, the 2012 numbers are from a NKOM report, and I got the 2013 numbers after asking NKOM via email. I was told the numbers for 2014 will be presented May 20th, and decided not to wait for those, as I doubt they will be very different from the numbers from 2013. The amount of data storage per minute sound depend on the wanted quality, and for phone calls it is generally believed that 8 Kbit/s is enough. See for example a summary on voice quality from Cisco for some alternatives. 8 Kbit/s is 60 Kbytes/min, and this can be multiplied with the number of call minutes to get the storage requirements. Storage prices varies a lot, depending on speed, backup strategies, availability requirements etc. But a simple way to calculate can be to use the price of a TiB-disk (around 1000 NOK / 120 EUR) and double it to take space, power and redundancy into account. It could be much higher with high speed and good redundancy requirements. But back to the question, What would it cost to store all phone calls in Norway? Not much. Here is a small table showing the estimated cost, which is within the budget constraint of most medium and large organisations:

Year	Call minutes	Size	Price in NOK / EUR
2005	24 000 000 000	1.3 PiB	3 mill / 358 000
2012	18 000 000 000	1.0 PiB	2.2 mill / 262 000
2013	17 000 000 000	950 TiB	2.1 mill / 250 000

This is the cost of buying the storage. Maintenance need to be taken into account too, but calculating that is left as an exercise for the reader. But it is obvious to me from those numbers that recording the sound of all phone calls in Norway is not going to be stopped because it is too expensive. I wonder if someone already is collecting the data?

The Machine Architecture Here's a brief introduction to some of the hardware concepts within The Machine. As the team at HP that I'm working with are busy working on Linux kernel changes motivated by the hardware, I'm hoping that providing this kind of documentation will help Linux kernel developers outside of HP evaluate that work, and work by others in related areas. Joining HP and learning about The Machine In January, I joined HP to work on Linux for The Machine. I'd watched Martin Fink's video and read other articles on the new hardware coming out of HP labs. I had hints of what they were up to, and the possibilities seemed exciting enough to entice me to go back to HP. When I arrived at HP, one of the first things I got to read was the external reference specification for The Machine. 170 pages detailing a more significant shift in computer architecture than I had been given any hints of, both in my interviews at HP and from what I could see in the press. Since then, I've been eager to tell people about what we're doing, and I'm happy to say that we're finally ready to start the conversation with this brief description. A Short Outline of Storage within The Machine The basic unit of The Machine is a collection of hardware grouped in a Load Store Domain. A Load Store Domain consists of:

• Multiple independent Compute Nodes
  • Independent operating systems
  • Local memory
  • Load/store access to shared, in-memory storage
• Shared byte-addressable persistent memory:
  • Non-volatile wrt operating system life cycle
  • Global address space
  • Hardware access control
  • Accessed with standard CPU load/store instructions

Here's a diagram of how the various bits of the hardware are hooked together: And, a brief description of the elements within the picture:

• Compute Node. A set of processing cores, caches and various ancillary peripherals.
• Local Store. Memory directly connected to the processing cores.
• Firewall. Hardware access control between the compute node and shared memory.
• Shared Byte-addressable Persistent Memory. This is the storage within The Machine. It is accessed directly via normal CPU load/store instructions in units as small as one byte.

I've intentionally drawn the shared memory in a large box to emphasize the notion that this machine is more "memory-centric" and less "processor-centric". The shared byte-addressable persistent memory forms the sole persistent storage within The Machine. More to Come I'll continue to publish information about The Machine and our related Linux work as we work on the hardware and software.

Yesterday I permanently deleted 15 years of email. It wasn't because I didn't have enough hard disk space to store it. It's because I decided, after 15 years, that the benefits of keeping all this email did not outweigh the risks. Although I have never had my email subpoenaed, I have had many legal interactions due to my involvement with May First/People Link, some of which were about finding the real identities of May First/People Link members. I'd rather not risk compromising anyone or needlessly exposing my networks. Now I have an Inbox, Sent Box, Trash Box and Detected Spam Box. The Inbox I empty manually and the other boxes are automatically purged on a scheduled basis. In this age of surveillance it's sad to see data evaluated based on risk of exposure.

Some time ago I have embarked on a jurney to install NetBSD on one of my two NSLU2-s. I have ran into all sorts of hurdles and problems which I finally managed to overcome, except one:

The NSLU I am using has a standard 20 pin ARM JTAG connector attached to it (as per this page http://www.nslu2-linux.org/wiki/Info/PinoutOfJTAGPort, only TDI, TDO, TMS, TCK, Vref and GND signals), but, although the chip is identified, I am unable to halt the CPU:
My main goal is to make sure I can flash the device via JTAG, in case I break it, but it would be ideal if I could use the JTAG to single step through the code.

I have found that other people have managed to flash the device via JTAG without the other signals, and some have even changed the bootloader (and had JTAG confirmed as backup solution), so I am stuck.

So if anyone can give some insights into ixp42x / Xscale / NSLU2 specific JTAG issues or hints regarding this issue on OpenOCD or other such tool, I would be really grateful.


Note: I have made a hacked second stage Apex bootloader to laod the NetBSD image via TFTP, but the default RedBoot sequence 'boot; exec 0x01d00000' should be 'boot; go 0x01d00000' for NetBSD to work, so I am considering changing the RedBoot partition to alter that command. The gory details can be summed as my Apex is calling RedBoot functions to be network enabled (because Intel's NPE current code is not working on Apex) and I have tested this to work with go, but not with exec.

A small project I worked on during the last few weeks has now come together in new package RcppTOML which arrived on CRAN yesterday. It provides R with a reader for TOML files. TOML stands for Tom's Obvious Markup Language. And before you roll your eyes, glance at the TOML site. It really is different, and has a number of rather wonderful features:

• free-format indentation as you please
• comments anywhere, even on the same line
• actual types such as string, integer, float, bool and datetime (!!) which are all native
• vectors, of course, of the above
• arbitrary nesting of tables

Here is a simple illustration where we parse the TOML example file derived from what is part of the main TOML README

R> p <- parseTOML(system.file("toml", "example.toml", package="RcppTOML"))
R> summary(p)
toml object with top-level slots:
   clients, database, owner, servers, title 
read from /usr/local/lib/R/site-library/RcppTOML/toml/example.toml 
R> p
List of 5
 $ clients :List of 2
  ..$ data :List of 2
  .. ..$ : chr [1:2] "gamma" "delta"
  .. ..$ : int [1:2] 1 2
  ..$ hosts: chr [1:2] "alpha" "omega"
 $ database:List of 4
  ..$ connection_max: int 5000
  ..$ enabled       : logi TRUE
  ..$ ports         : int [1:3] 8001 8001 8002
  ..$ server        : chr "192.168.1.1"
 $ owner   :List of 4
  ..$ bio         : chr "GitHub Cofounder & CEO\\nLikes tater tots and beer."
  ..$ dob         : POSIXct[1:1], format: "1979-05-27 07:32:00"
  ..$ name        : chr "Tom Preston-Werner"
  ..$ organization: chr "GitHub"
 $ servers :List of 2
  ..$ alpha:List of 2
  .. ..$ dc: chr "eqdc10"
  .. ..$ ip: chr "10.0.0.1"
  ..$ beta :List of 2
  .. ..$ dc: chr "eqdc10"
  .. ..$ ip: chr "10.0.0.2"
 $ title   : chr "TOML Example"
NULL
R> 

See much more at the TOML site. I converted one first project at work to this and it really rocks. Point to a file, get a list back and index all components by their names. We also added really simple S3 classes to the default print() method uses str() for a more compact presentation of what (in R) is of course nested list types. Internally, the RcppTOML packages use the splendid cpptoml parser by Chase Geigle. This brings in modern C++11 and makes it that CRAN simply cannot build a binary for R on Windows as the g++ version (still, as of April 2015) in Rtools is too old. There is word of an update to Rtools and that point should we able to support Windows as well. Until then, no mas. A bit more information is on the package page here as well as as the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

---

---

---

---