Get involved After this happened, my next step was to get re-involved in Debian Live to help it carry on after the loss of Daniel. Here s a quick update on some team progress, notes that could help people building Stretch images right now, and what to expect next. Team progress

• Iain uploaded live-config, incorporating an important fix, #bc8914bc, that prevented images from booting.
• I want to get live-images ready for an upload, including #8f234605 to fix wrong config/bootloaders that prevented images from building.

Test build notes

• As always, build Stretch images with latest live-build from Sid (i.e. 5.x).
• Build Stretch images, not Sid, as there s less of a chance of dependency issues spoiling the build, and that s the default anyway.
• To make build iterations faster, make sure the config is modified to not build source & not include installer (edit auto/config before lb config ) and use an apt caching proxy.
• Don t forget to inject fixed packages (e.g. live-config) into each config. Use apt pinning as per live-manual, or drop the debs into config/packages.chroot.

Test boot notes

• Use kvm, giving it enough ram (-m 1024 works for me).
• For gnome-desktop and kde-desktop, use -vga qxl, or else the desktop will crash and restart repeatedly.
• When using qxl, edit boot params to add qxl.modeset=1 (workaround for #779515, which will be fixed in kernel >= 4.3).
• My gnome image test was spoiled by #802929. The mouse doesn t work (pointer moves, but no buttons work). Waiting on a new kernel to fix this. This is a test environment related bug only, i.e. should work fine on hardware. (Test pending.)
• The Stretch standard, lxde-desktop, cinnamon-desktop, xfce-desktop, and gnome-desktop images all built and booted fine (except for the gnome issue noted above).
• The Stretch kde-desktop and mate-desktop images are next on my list to test, along with Jessie images.
• I ve only tested on the standard and lxde-desktop images that if the installer is included, booting from the Install boot menu option starts the installer (i.e. didn t do an actual install).

Coming soon See the TODO in the wiki. We re knocking these off steadily. It will be faster with more people helping (hint, hint).

Finally, bzed.de is back online and I m planning to start blogging again! Part of the reason why I became inactive was the usage of ikiwiki, which is great, but at end unnecessarily complicated. So I ve migrated by page to gohugo.io - a static website generator, written in go. Hugo has an active community and it is easy to create themes for it or to enhance it. Also it is using plain Markdown syntax instead of special ikiwiki syntax mixed into it - should make it easy to migrate away again if necessary. In case somebody else would like to convert from ikiwiki to Hugo, here is the script I ve hacked together to migrate my old blog posts.

#!/bin/bash
find . -type f -name '*.mdwn'   while read i; do
    tmp= mktemp 
     
        echo '+++'
        slug="$(echo $i   sed 's,.*/,,;s,\.mdwn$,,')"
        echo "slug = \"$ slug \""
        echo "title = \"$(echo $i   sed 's,.*/,,;s,\.mdwn$,,;s,_, ,g;s/\b\(.\)/\u\1/;s,debian,Debian,g')\""
        if grep -q 'meta updated' $i; then
            echo -n 'date = '
            sed '/meta updated/!d;/.*meta updated.*/s,.*=",,;s,".*,,;s,^,",;s,$,",' $i
        else
            echo -n 'date = '
            git log --diff-filter=A --follow --format='"%aI"' -1 -- $i
        fi
        if grep -q '\[\[!tag' $i; then
            echo -n 'tags ='
            sed '/\[\[!tag/!d;s,[^ ]*tag ,,;s,\]\],,;s,\([^ ]*\),"\1",g;s/ /,/g;s,^,[,;s,$,],' $i
        fi
        echo 'categories = ["linux"]'
        echo 'draft = false'
        echo '+++'
        echo ''
        sed -e '/\[\[!tag/d' \
            -e '/meta updated/d' \
            -e '/\[\[!plusone *\]\]/d' \
            -e 's,\[\[!img files[0-9/]*/\([^ ]*\) alt="\([^"]*\).*,![\2](../\1),g' \
            -e 's,\[\([^]]*\)\](\([^)]*\)),[\1](\2),g' \
            -e 's,\[\[\([^ ]*\) \([^]]*\)\]\],[\1](\2),g' \
            $i
      > $tmp
    #cat $tmp; rm $tmp 
    mv $tmp  echo $i   sed 's,\.mdwn,.md,g' 
done

For the planet Debian readers - only linux related posts will show up on the planet. If you are interested in my mountain activities and other things I post, please follow my blog on bzed.de directly.

You have a database, a tweet pattern and wants to automatically tweet on a regular basis? No need for RSS, fancy tricks, 3rd party website to translate RSS to Twitter or whatever. Just use db2twitter.

• db2twitter on Github (star it on Github if you like it :) )
• Official documentation of db2twitter on readthedocs
• db2twitter is in Debian Sid already! YaY!

db2twitter is pretty easy to use! First define your Twitter credentials:

screen_name_of_the_user_to_retweet=journalduhacker
consumer_key=ml9jaiBnf3pmU9uIrKNIxAr3v
consumer_secret=8Cmljklzerkhfer4hlj3ljl2hfvc123rezrfsdctpokaelzerp
access_token=213416590-jgJnrJG5gz132nzerl5zerwi0ahmnwkfJFN9nr3j
access_token_secret=3janlPMqDKlunJ4Hnr90k2bnfk3jfnwkFjeriFZERj32Z

Then your database information:

dbconnector=mysql+mysqlconnector
dbhost=localhost
database=jobboard
dbuser=jobboard
dbpass=V3rYS3cr3t
dbtables=jobs,
jobs_rows=companyname,jobtitle,id

Then the pattern of your tweet, a Python-style formatted string:

tweet=  hires a   https://www.linuxjobs.fr/jobs/ 

Add db2twitter in your crontab:

*/10 * * * * db2witter db2twitter db2twitter.ini

And you re all set! db2twitter will generate and tweet the following tweets:

MyGreatCompany hires a web developer https://www.linuxjobs.fr/jobs/51
CoolStartup hires a devops skilled in Docker https://www.linuxjobs.fr/jobs/52

db2twitter is developed by and run for LinuxJobs.fr, the job board of th french-speaking Free Software and Opensource community. db2twitter also has cool options like;

• only tweet during user-specified time (e.g 9AM-6PM)
• use user-specified SQL filter in order to get data from the database (e.g only fetch rows where status == edited )

db2twitter is coded in Python 3.4, uses SQlAlchemy (see supported database types) and Tweepy. The official documentation is available on readthedocs.

Yesterday, Wammu 0.40 has been released. The list of changes is not really huge:

• Correctly escape XML output.
• Make error message selectable.
• Fixed spurious D-Bus error message.
• Translation updates.

I will not make any promises for future releases (if there will be any) as the tool is not really in active development.

Filed under: English Gammu Wammu 0 comments

What happened in the reproducible builds effort this week: Toolchain fixes

• Markus Koschany uploaded antlr3/3.5.2-3 which includes a fix by Emmanuel Bourg to make the generated parser reproducible.
• Markus Koschany uploaded maven-bundle-plugin/2.4.0-2 which includes a fix by Emmanuel Bourg to use the date in the DEB_CHANGELOG_DATETIME variable in the pom.properties file embedded in the jar files.
• Niels Thykier uploaded debhelper/9.20151116 which makes the timestamp of directories created by dh_install, dh_installdocs, and dh_installexamples reproducible. Patch by Niko Tyni.

Mattia Rizzolo uploaded a version of perl to the reproducible repository including the patch written by Niko Tyni to add support for SOURCE_DATE_EPOCH in Pod::Man. Dhole sent an updated version of his patch adding support for SOURCE_DATE_EPOCH in GCC to the upstream mailing list. Several comments have been made in response which have been quickly addressed by Dhole. Dhole also forwarded his patch adding support for SOURCE_DATE_EPOCH in libxslt upstream. Packages fixed The following packages have become reproducible due to changes in their build dependencies: antlr3/3.5.2-3, clusterssh, cme, libdatetime-set-perl, libgraphviz-perl, liblingua-translit-perl, libparse-cpan-packages-perl, libsgmls-perl, license-reconcile, maven-bundle-plugin/2.4.0-2, siggen, stunnel4, systemd, x11proto-kb. The following packages became reproducible after getting fixed:

• bindex/2.2+svn101-3 by Markus Koschany.
• glyr/1.0.8-2 by Etienne Millon.
• jenkins-json/2.4-jenkins-3-4 by Emmanuel Bourg.
• pkg-config/0.29-1 uploaded by Tollef Fog Heen, original patch by Juan Picca.
• plexus-containers1.5/1.6-1 by Emmanuel Bourg.
• polyglot-maven/0.8~tobrien+git20120905-5 by Emmanuel Bourg.
• sigil/0.9.0+dfsg-3 uploaded by Mattia Rizzolo, original patch by Reiner Herrmann.
• simutrans/120.1.1+repack-2 uploaded by J rg Frings-F rst, fix by Markus Koschany.
• torrus/2.08-4 by Bernhard Schmidt.
• trigger-rally-data/0.6.1-2 uploaded by Bertrand Marc, patch by Mattia Rizzolo.

Some uploads fixed some reproducibility issues, but not all of them:

• castle-game-engine/5.2.0-1 by Paul Gevers.
• libam7xxx/0.1.6-2 by Antonio Ospite.
• libpdfbox-java/1:1.8.10-1 by Emmanuel Bourg.
• xfaces/3.3-29 uploaded by Hakan Ardo, original patch by Chris Lamb.

reproducible.debian.net Vagrant Cascadian has set up a new armhf node using a Raspberry Pi 2. It should soon be added to the Jenkins infrastructure. diffoscope development diffoscope version 42 was release on November 20th. It adds a missing dependency on python3-pkg-resources and to prevent similar regression another autopkgtest to ensure that the command line is functional when Recommends are not installed. Two more encoding related problems have been fixed (#804061, #805418). A missing Build-Depends has also been added on binutils-multiarch to make the test suite pass on architectures other than amd64. Package reviews 180 reviews have been removed, 268 added and 59 updated this week. 70 new fail to build from source bugs have been reported by Chris West, Chris Lamb and Niko Tyni. New issue this week: randomness_in_ocaml_preprocessed_files. Misc. Jim MacArthur started to work on a system to rebuild and compare packages built on reproducible.debian.net using .buildinfo and snapshot.debian.org. On December 1-3rd 2015, a meeting of about 40 participants from 18 different free software projects will be held in Athens, Greece with the intent of improving the collaboration between projects, helping new efforts to be started, and brainstorming on end-user aspects of reproducible builds.

Last Friday, I compiled a list of keyboards I'm interested in, and received a lot of incredible feedback, thank you all! This allowed me to shorten the list considerably, two basically two pieces. I'm reasonably sure by now which one I want to buy (both), but will spend this week calming down to avoid impulse-buying. My attention was also brought to a few keyboards originally not on my list, and I'll take this opportunity to present my thoughts on those too.The FinalistsErgoDoxPros

• Great design, by the looks of it.
• Mechanical keys.
• Open source hardware and firmware, thus programmable.
• Thumb keys.
• Available as an assembled product, from multiple sources.

Cons

• Primarily a kit, but assembled available.
• Assembled versions aren't as nice as home-made variants.

SummaryThe keyboard looks interesting, primarily due to the thumb keys. From the ErgoDox EZ campaign, I'm looking at $270. That's friendly, and makes ErgoDox a viable option! (Thanks @miffe!)There's also another option, FalbaTech, which ships sooner, I can customize the keyboard to some extent, and Poland is much closer to Hungary than the US. With this option, I'm looking at $205 + shipping, a very low price for what the keyboard has to offer. (Thanks @pkkolos for the suggestion!)Keyboardio M01Pros

• Mechanical keyboard.
• Hardwood body.
• Blank and dot-only keycaps option.
• Open source: firmware, hardware, and so on. Comes with a screwdriver.
• The physical key layout has much in common with my TypeMatrix.
• Numerous thumb-accessible keys.
• A palm key, that allows me to use the keyboard as a mouse.
• Fully programmable LEDs.
• Custom macros, per-application even.

Cons

• Fairly expensive.
• Custom keycap design, thus rearranging them physically is not an option, which leaves me with the blank or dot-only keycap options only.
• Available late summer, 2016.

SummaryWith shipping cost and whatnot, I'm looking at something in the $370 ballpark, which is on the more expensive side. On the other hand, I get a whole lot of bang for my buck: LEDs, two center bars (tripod mounting sounds really awesome!), hardwood body, and a key layout that is very similar to what I came to love on the TypeMatrix.I also have a thing for wooden stuff. I like the look of it, the feel of it.The VerdictRight now, I'm seriously considering the Model 01, because even if it is about twice the price of the ErgoDox, it also offers a lot more: hardwood body (I love wood), LEDs, palm key. I also prefer the layout of the thumb keys on the Model 01.The Model 01 also comes pre-assembled, looks stunning, while the ErgoDox pales a little in comparsion. I know I could make it look stunning too, but I do not want to build things. I'm not good at it, I don't want to be good at it, I don't want to learn it. I hate putting things together. I'm the kind of guy who needs three tries to put together a set of IKEA shelves, and I'm not exaggerating. I also like the shape of the keys better on the Model 01.Nevertheless, the ErgoDox is still an option, due to the price. I'd love to buy both, if I could. Which means that once I'm ready to replace my keyboard at work, I will likely buy an ErgoDox. But for home, Model 01 it is, unless something even better comes along before my next pay.The Kinesis Advantage was also a strong contender, but I ended up removing it from my preferred options, because it doesn't come with blank keys, and is not a split keyboard. And similar to the ErgoDox, I prefer the Model 01's thumb-key layout. Despite all this, I'm very curious about the key wells, and want to try it someday.Suggested optionsYogitypeSuggested by Andred Carter, a very interesting keyboard with a unique design.Pros

• Portable, foldable.
• Active support for forearm and hand.
• Hands never obstruct the view.

Cons

• Not mechanical.
• Needs a special inlay.
• Best used for word processing, programmers may run into limitations.

SummaryI like the idea of the keyboard, and if it wouldn't need a special inlay, but used a small screen or something to show the keys, I'd like it even more. Nevertheless, I'm looking for a mechanical keyboard right now, which I can also use for coding.But I will definitely keep the Yogitype in mind for later!Matias Ergo ProPros

• Mechanical keys.
• Simple design.
• Split keyboard.

Cons

• Doesn't seem to come with a blank keys option, nor in Dvorak.
• No thumb key area.
• Neither open source, nor open hardware.
• I have no need for the dedicated undo, cut, paste keys.
• Does not appear to be programmable.

SummaryThis keyboard hardly meets any of my desired properties, and doesn't have anything standing out in comparison with the others. I had a quick look at this when compiling my original list, but was quickly discarded. Nevertheless, people asked me why, so I'm including my reasoning here:While it is a split keyboard, with a fairly simple design, it doesn't come in the layout I'd prefer, nor with blank keys. It lacks the thumb key area that ErgoDox and the Model 01 have, and which I developed an affection for.Microsoft Sculpt Ergonomic KeyboardPros

• Numpad is a separate unit.
• Reverse tilt.
• Well positioned, big Alt keys.
• Cheap.

Cons

• Not a split keyboard.
• Not mechanical.
• No blank or Dvorak option as far as I see.

SummaryThis keyboard does not buy me much over my current TypeMatrix 2030. If I'd be looking for the cheapest possible among ergonomic keyboards, this would be my choice. But only because of the price.Truly Ergonomic KeyboardPros

• Mechanical.
• Detachable palm rest.
• Programmable firmware.

Cons

• Not a split keyboard.
• Layouts are virtual only, the printed keycaps stay QWERTY, as far as I see.
• Terrible navigation key setup.

SummaryTwo important factors for me are physical layout and splittability. This keyboard fails both. While it is a portable device, that's not a priority for me at this time.

Long over due post It s been a long time I haven t written here. And lots of things happened in the OpenStack planet. As a full time employee with the mission to package OpenStack in Debian, it feels like it is kind of my duty to tell everyone about what s going on. Liberty is out, uploaded to Debian Since my last post, OpenStack Liberty, the 12th release of OpenStack, was released. In late August, Debian was the first platform which included Liberty, as I proudly outran both RDO and Canonical. So I was the first to make the announcement that Liberty passed most of the Tempest tests with the beta 3 release of Liberty (the Beta 3 is always kind of the first pre-release, as this is when feature freeze happens). Though I never made the announcement that Liberty final was uploaded to Debian, it was done just a single day after the official release. Before the release, all of Liberty was living in Debian Experimental. Following the upload of the final packages in Experimental, I uploaded all of it to Sid. This represented 102 packages, so it took me about 3 days to do it all. Tokyo summit I had the pleasure to be in Tokyo for the Mitaka summit. I was very pleased with the cross-project sessions during the first day. Lots of these sessions were very interesting for me. In fact, I wish I could have attended them all, but of course, I can t split myself in 3 to follow all of the 3 tracks. Then there was the 2 sessions about Debian packaging on upstream OpenStack infra. The goal is to setup the OpenStack upstream infrastructure to allow packaging using Gerrit, and gating each git commit using the usual tools: building the package and checking there s no FTBFS, running checks like lintian, piuparts and such. I knew already the overview of what was needed to make it happen. What I didn t know was the implementation details, which I hoped we could figure out during the 1:30 slot. Unfortunately, this didn t happen as I expected, and we discussed more general things than I wished. I was told that just reading the docs from the infra team was enough, but in reality, it was not. What currently needs to happen is building a Debian based image, using disk-image-builder, which would include the usual tools to build packages: git-buildpackage, sbuild, and so on. I m still stuck at this stage, which would be trivial if I knew a bit more about how upstream infra works, since I already know how to setup all of that on a local machine. I ve been told by Monty Tailor that he would help. Though he s always a very busy man, and to date, he still didn t find enough time to give me a hand. Nobody replied to my request for help in the openstack-dev list either. Hopefully, with a bit of insistence, someone will help. Keystone migration to Testing (aka: Debian Stretch) blocked by python-repoze.who Absolutely all of OpenStack Liberty, as of today, has migrated to Stretch. All? No. Keystone is blocked by a chain of dependency. Keystone depends on python-pysaml2, itself blocked by python-repoze.who. The later, I upgraded it to version 2.2. Though python-repoze.what depends on version <= 1.9, which is blocking the migration. Since python-repoze.who-plugins, python-repoze.what and python-repoze.what-plugins aren t used by any package anymore, I asked for them to be removed from Debian (see #805407). Until this request is processed by the FTP masters, Keystone, which is the most important piece of OpenStack (it does the authentication) will be blocked for migration to Stretch. New OpenStack server packages available On my presentation at Debconf 15, I quickly introduced new services which were released upstream. I have since packaged them all:

• Barbican (Key management as a Service)
• Congress (Policy as a Service)
• Magnum (Container as a Service)
• Manila (Filesystem share as a Service)
• Mistral (Workflow as a Service)
• Zaqar (Queuing as a Service)

Congress, unfortunately, was not accepted to Sid yet, because of some licensing issues, especially with the doc of python-pulp. I will correct this (remove the non-free files) and reattempt an upload. I hope to make them all available in jessie-backports (see below). For the previous release of OpenStack (ie: Kilo), I skipped the uploads of services which I thought were not really critical (like Ironic, Designate and more). But from the feedback of users, they would really like to have them all available. So this time, I will upload them all to the official jessie-backports repository. Keystone v3 support For those who don t know about it, Keystone API v3 means that, on top of the users and tenant, there s a new entity called a domain . All of the Liberty is now coming with Keystone v3 support. This includes the automated Keystone catalog registration done using debconf for all *-api packages. As much as I could tell by running tempest on my CI, everything still works pretty well. In fact, Liberty is, to my experience, the first release of OpenStack to support Keystone API v3. Uploading Liberty to jessie-backports I have rebuilt all of Liberty for jessie-backports on my laptop using sbuild. This is more than 150 packages (166 packages currently). It took me about 3 days to rebuild them all, including unit tests run at build time. As soon as #805407 is closed by the FTP masters, all what s remaining will be available in Stretch (mostly Keystone), and the upload will be possible. As there will be a lot of NEW packages (from the point of view of backports), I do expect that the approval will take some time. Also, I have to warn the original maintainers of the packages that I don t maintain (for example, those maintained within the DPMT), that because of the big number of packages, I will not be able to process the usual communication to tell that I m uploading to backports. However, here s the list of package. If you see one that you maintain, and that you wish to upload the backport by yourself, please let me know. Here s the list of packages, hopefully, exhaustive, that I will upload to jessie-backports, and that I don t maintain myself: alabaster contextlib2 kazoo python-cachetools python-cffi python-cliff python-crank python-ddt python-docker python-eventlet python-git python-gitdb python-hypothesis python-ldap3 python-mock python-mysqldb python-pathlib python-repoze.who python-setuptools python-smmap python-unicodecsv python-urllib3 requests routes ryu sphinx sqlalchemy turbogears2 unittest2 zzzeeksphinx. More than ever, I wish I could just upload these to a PPA^W Bikeshed, to minimize the disruption for both the backports FTP masters, other maintainers, and our OpenStack users. Hopefully, Bikesheds will be available soon. I am sorry to give that much approval work to the backports FTP masters, however, using the latest stable system with the latest release, is what most OpenStack users really want to do. All other major distributions have specific repositories too (ie: RDO for CentOS / Red Hat, and cloud archive for Ubuntu), and stable-backports is currently the only place where I can upload support for the Stable release. Debian listed as supported distribution on openstack.org Good news! If you go at http://www.openstack.org/marketplace/distros/ you will see a list of supported distributions. I am proud to be able to tell that, after 6 months of lobbying from my side, Debian is also listed there. The process of having Debian there included talking with folks from the OpenStack foundation, and having Bdale to sign an agreement so that the Debian logo could be reproduced on openstack.org. Thanks to Bdale Garbee, Neil McGovern, Jonathan Brice, and Danny Carreno, without who this wouldn t have happen.

With the latest release the Wireshark Project decided to make the Qt GUI the default interface. In line with Debian s Policy the packages shipped by Debian also switched the default GUI to minimize the difference from upstream. The GTK+ interface which was the previous default is still available from the wireshark-gtk package. You can read more about the new 2.0.0 release in the release notes or on the Wireshark Blog featuring some of the improvements. Happy sniffing! update: Wireshark 2.0.0 will be available from testing and jessie-backports in a week. Ubuntu users can already download binary packages from the Wireshark stable releases PPA maintained by the Wireshark Project (including me:-)).

It's been at least a year since I last did any work on Debian, but this week I finally uploaded a new version of squishyball, an audio sample comparison tool, incorporating a patch from Thibaut Girka which fixes the X/X/Y test method. Shamefully Thibaut's patch is nearly a year old too. Better late than never... I've also uploaded a new version of smartmontools which updates the package to the new upstream version. I'm not the regular maintainer for this package, but it is in the set of packages covered by the collab-maint team. To be polite I uploaded it to DELAYED-7, so it will take a week to hit unstable. I've temporarily put a copy of the package here in the meantime.

I am so saddened by the news this week. The attacks in Paris, Beirut, and Mali. The reaction of fear, anger, and hate. Governors racing to claim they will keep out refugees, even though they lack the power to do so. Congress voting to keep out refugees. Emotions are a powerful thing. They can cause people to rise up and accomplish stunning things that move humanity forward. And they can move us back. Fear, and the manipulation of it, is one of those. What have I to fear? Even if the United States accepted half a million Syrian refugees tomorrow, I would be far more likely to die in a car accident than at the hands of a Syrian terrorist. I am a careful and cautious person, but I understand that life is not lived unless risk is balanced. I know there is a risk of being in a car crash every time I drive somewhere but if that kept me at home, I would never see my kids violin concert, the beautiful painted canyon of Texas, or the Flint Hills of Kansas. So I drive smart and carefully, but I still drive without fear. I accept this level of risk as necessary to have a life worth living in this area (where there are no public transit options and the nearest town is miles away). I have had pain in my life. I ve seen grandparents pass away, I ve seen others with health scares. These things are hard to think about, but they happen to us all at some point. What have I to fear? I do not fear giving food to the hungry, shelter to the homeless, comfort to those that have spent the last years being shot at. I do not fear helping someone that is different than me. If I fail to do these things for someone because of where they come from or what their holy book is, then I have become less human. I have become consumed by fear. I have let the terrorists have control over my life. And I refuse to do that. If governors really wanted to save lives, they would support meaningful mass transit alternatives that would prevent tens of thousands of road deaths a year. They would support guaranteed health care for all. They would support good education, science-based climate change action, clean water and air, mental health services for all, and above all, compassion for everyone. By supporting Muslim registries, we look like Hitler to them. By discriminating against refugees based on where they re from or their religion, we support the terrorists, making it easy for them to win hearts and minds. By ignoring the fact that entering the country as a refugee takes years, as opposed to entering as a tourist taking only minutes, we willfully ignore the truth about where dangers lie. So what do I have to fear? Only, as the saying goes, fear. Fear is making this country turn its backs on the needy. Fear is making not just the US but much of Europe turn its backs on civil liberties and due process. Fear gives the terrorists control, and that helps them win. I refuse. I simply refuse to play along. No terrorist, no politician, no bigot gets to steal MY humanity. Ultimately, however, I know that the long game is not one of fear. The arc of the universe bends towards justice, and ultimately, love wins. It takes agonizingly long sometimes, but in the end, love wins. So I do not fear.

Even though I spend more time staring at the screen than typing, there are times when I - after lots and lots of prior brain work - sit down and start typing, a lot. A couple of years ago, I started to feel pain in my wrists, and there were multiple occasions when I had to completely stop writing for longer periods of time. These were situations I obviously did not want repeated, so I started to look for remedies. First, I bought a new keyboard, a TypeMatrix 2300, which while not ergonomic, was a huge relief for my hands and wrists. I also started to learn Dvorak, but that's still something that is kind-of in progress: my left hand can write Dvorak reasonably fast, but my right one seems to be Qwerty-wired, even after a month of typing Dvorak almost exclusively.This keyboard served me well for the past five year or so. But recently, I started to look for a replacement, partly triggered by a Clojure/conj talk I watched. I got as far as assembling a list of keyboards I'm interested in, but I have a hard time choosing. This blog post here serves two purposes then: first to make a clear pros/cons list for myself, second, to solicit feedback from others who may have more experience with any of the options below.Update: There is a [follow up post], with a few more keyboards explored, and a semi-final verdict. Thanks everyone for the feedback and help, much appreciated!Lets start with the current keyboard!TypeMatrix 2030Pros

• The Matrix architecture, with straight vertical key columns has been incredibly convenient.
• Enter and Backspace in the middle, both large: loving it.
• Skinnable (easier to clean, and aids in learning a new layout).
• Optional dvorak skin, and a hardware Dvorak switch.
• The layout (cursor keys, home/end, page up/down, etc) is something I got used to very fast.
• Multimedia keys close by with Fn.
• Small, portable, lightweight - ideal for travel.

Cons

• Small: while also a feature, this is a downside too. Shoulder position is not ideal.
• Skins: while they are a terrific aid when learning a new layout, and make cleaning a lot easier, they wear off quickly. Sometimes fingernails are left to grow too long, and that doesn't do good to the skin. One of my two QWERTY layouts has a few holes already, sadly.
• Not a split keyboard, which is starting to feel undesirable.

SummaryAll in all, this is a keyboard I absolutely love, and am very happy with. Yet, I feel I'm ready to try something different. With my skins aging, and the aforementioned Clojure/conj talk, the desire to switch has been growing for a while now.Desired propertiesThere are a few desired properties of the keyboard I want next. The perfect keyboard need not have all of these, but the more the merrier.

• Ergonomic design.
• Available in Dvorak, or with blank keys.
• Preferably a split keyboard, so I can position the two parts as I see fit.
• Ships to Hungary, or Germany, in a reasonable time frame. (If all else fails, shipping to the US may work too, but I'd rather avoid going through extra hoops.)
• Mechanical keys preferred. But not the loud clicky type: I work in an office; and at home, I don't want to wake my wife either.

I plan to buy one keyboard for a start, but may end up buying another to bring to work (like I did with the TypeMatrix, except my employer at the time bought the second one for me). At work, I will continue using the TypeMatrix, most likely, but I'm not sure yet.Anyhow, there are a number of things I do with my computer that require a keyboard:

• I write code, a considerable amount.
• I write prose, even more than code. Usually in English, sometimes in Hungarian.
• I play games. Most of them, with a dedicated controller, but there are some where I use the keyboard a lot.
• I browse the web, listen to music, and occasionally edit videos.
• I multi-task all the time.
• 90% of my time is spent within Emacs (recently switched to Spacemacs).
• I hate the mouse, with a passion. Trackballs, trackpoints and touchpads even more. If I can use my keyboard to do mouse-y stuff well enough to control the browser, and do some other things that do not require precise movement (that is, not games), I'll be very happy.

I am looking for a keyboard that helps me do these things. A keyboard that will stay with me not for five years or a decade, but pretty much forever.The optionsUltimate Hacking KeyboardPros

• Split keyboard.
• Mechanical keys (with a quiet option).
• Ships to Hungary. Made in Hungary!
• Optional addons: three extra buttons and a small trackball for the left side, and a trackball for the right side. While I'm not a big fan of the mouse, the primary reasons is that I have to move my hand. If it's in the middle, that sounds much better.
• Four layers of the factory keymap: I love the idea of these layers, especially the mouse layer.
• Programmable, so I can define any layout I want.
• Open source firmware, design and agent!
• An optional palm rest is available as well.
• Blank option available.

Cons

• Likely not available before late summer, 2016.
• No thumb keys.
• Space/Mod arrangement feels alien.
• The LED area is useless to me, and bothers my eye. Not a big deal, but still.
• While thumb keys are available for the left side, not so for the right one. I'd rather have keys there than a trackball. The only reason I'd want the $50 addon set, is the left thumb-key module (which also seems to have a trackpoint, another pointless gadget).

SummaryThe keyboard looks nice, has a lot of appealing features. It is programmable, so much so that by the looks of it, I could emulate the hardware dvorak switch my TypeMatrix has. However, I'm very unhappy with the addons, so there's that too.All in all, this would cost me about $304 (base keyboard, modules, palm rest and shipping). Not too bad, certainly a strong contender, despite the shortcomings.ErgoDoxPros

• Great design, by the looks of it.
• Mechanical keys.
• Open source hardware and firmware, thus programmable.
• Thumb keys.
• Available via ErgoDox EZ as an assembled product.

Cons

• Primarily a kit, but assembled available.
• Not sure when it'd ship (december shipments are sold out).

SummaryThe keyboard looks interesting, primarily due to the thumb keys. From the ErgoDox EZ campaign, I'm looking at $270. That's friendly, and makes ErgoDox a viable option! (Thanks @miffe!)Kinesis AdvantagePros

• Mechanical keys, Cherry-MX brown.
• Separate thumb keys.
• Key wells look interesting.
• Available right now.
• QWERTY/Dvorak layout available.

Cons

• Not a split keyboard.
• Not open source, neither hardware, nor firmware.
• Shipping to Hungary may be problematic.
• The QWERTY/Dvorak layout is considerably more expensive.
• Judging by some of the videos I saw, keys are too loud.

SummaryThe key wells look interesting, but it's not a split keyboard, nor is it open source. The cost come out about $325 plus shipping and VAT and so on, so I'm probably looking at something closer to $400. Nah. I'm pretty sure I can rule this out.Kinesis FreeStyle2Pros

• Split keyboard.
• Available right now.
• Optional accessory, to adjust the slope of the keyboard.

Cons

• Not open source, neither hardware, nor firmware.
• Doesn't seem to be mechanical.
• Shipping to Hungary may be problematic.
• No Dvorak layout.
• No thumb keys.

SummaryWhile a split keyboard, at a reasonably low cost ($149 + shipping + VAT), it lacks too many things to be considered a worthy contender.MaltronPros

• Mechanical keyboard.
• Key wells.
• Thumb keys.
• Built in palm rest.
• Available in Dvorak too.

Cons

• Not a split keyboard.
• The center numeric area looks weird.
• Not sure about programmability.
• Not open source.
• Expensive.

SummaryWithout shipping, I'm looking at 450. That's a very steep price. I love the wells, and the thumb keys, but it's not split, and customisability is a big question here.AtreusPros

• Sleek, compact design.
• No keycaps.
• Mechanical keyboard.
• Open source firmware.
• More keys within thumbs reach.
• Available right now.

Cons

• Ships as a DIY kit.
• Not a split keyboard.

SummaryWhile not a split keyboard, it does look very interesting, and the price is much lower than the rest: $149 + shipping ($50 or so). It is similar - in spirit - to my existing TypeMatrix. It wouldn't take much to get used to, and is half the price of the alternatives. A strong option, for sure.Keyboardio M01Pros

• Mechanical keyboard.
• Hardwood body.
• Blank and dot-only keycaps option.
• Open source: firmware, hardware, and so on. Comes with a screwdriver.
• The physical key layout has much in common with my TypeMatrix.
• Numerous thumb-accessible keys.
• A palm key, that allows me to use the keyboard as a mouse.
• Fully programmable LEDs.
• Custom macros, per-application even.

Cons

• Fairly expensive.
• Custom keycap design, thus rearranging them physically is not an option, which leaves me with the blank or dot-only keycap options only.
• Available late summer, 2016.

SummaryWith shipping cost and whatnot, I'm looking at something in the $370 ballpark, which is on the more expensive side. On the other hand, I get a whole lot of bang for my buck: LEDs, two center bars (tripod mounting sounds really awesome!), hardwood body, and a key layout that is very similar to what I came to love on the TypeMatrix.I also have a thing for wooden stuff. I like the look of it, the feel of it.The Preference ListAfter writing this all up, I think I prefer the Model 01, but the UHK and ErgoDox come close too!The UHK is cheaper, but not by a large margin. It lacks the thumb keys and the palm key the M01 has. It also looks rather dull (sorry). They'd both ship about the same time, but, the M01 is already funded, while the UHK is not (mind you, there's a pretty darn high chance it will be).The ErgoDox has thumb keys, split keyboard, and is open source. Compared to the UHK, we have the thumb keys, and less distraction, for a better price. But the case is not so nice. Compared to the Model 01: no leds, or center bar, and an inferior case. But, much better price, which is an important factor too.Then, there's the Atreus. While it's a DIY kit, it is much more affordable than the rest, and I could have it far sooner. Yet... it doesn't feel like a big enough switch from my current keyboard. I might as well continue using the TypeMatrix then, right?The rest, I ruled out earlier, while I was reviewing them anyway.So, the big question is: should I invest close to $400 into a keyboard that looks stunning, and will likely grow old with me? Or should I give up some of the features, and settle for one of the $300 ones, that'll also grow old with me. Or is there an option I did not consider, that may match my needs and preferences better?If you, my dear reader, got this far, and have a suggestion, please either tweet at me, or write an email, or reach me over any other medium I am reachable at (including IRC, hanging out as algernon on FreeNode and OFTC).Thank you in advance, to all of you who contact me, and help me choose a keyboard!

Going back to Android recently, I saw that all tools binaries from the Android project are now click-wrapped by a quite ugly proprietary license, among others an anti-fork clause (details below). Apparently those T&C are years old, but the click-wrapping is newer. This applies to the SDK, the NDK, Android Studio, and all the essentials you download through the Android SDK Manager. Since I keep my hands clean of smelly EULAs, I'm working on rebuilding the Android tools I need.
We're talking about hours-long, quad-core + 8GB-RAM + 100GB-disk-eating builds here, so I'd like to publish them as part of a project who cares. As a proof-of-concept, the Replicant project ships a 4.2 SDK and I contributed build instructions for ADT and NDK (which I now use daily). (Replicant is currently stuck to a 2013 code base though.) I also have in-progress instructions on my hard-drive to rebuild various newer versions of the SDK/API levels, and for the NDK whose releases are quite hard to reproduce (no git tags, requires fixes committed after the release, updates are partial rebuilds, etc.) - not to mention that Google doesn't publish the source code until after the official release (closed development) And in some cases like Android Support Repository [not Library] I didn't even find the proper source code, only an old prebuilt. Would you be interested in contributing, and would you recommend a structure that would promote Free, rebuilt Android *DK? The legalese Anti-fork clause:

3.4 You agree that you will not take any actions that may cause or result in the fragmentation of Android, including but not limited to distributing, participating in the creation of, or promoting in any way a software development kit derived from the SDK.

So basically the source is Apache 2 + GPL, but the binaries are non-free. By the way this is not a GPL violation because right after:

3.5 Use, reproduction and distribution of components of the SDK licensed under an open source software license are governed solely by the terms of that open source software license and not this License Agreement.

Still, AFAIU by clicking "Accept" to get the binary you still accept the non-free "Terms and Conditions". (Incidentally, if Google wanted SDK forks to spread and increase fragmentation, introducing an obnoxious EULA is probably the first thing I'd have recommended. What was its legal team thinking?) Indemnification clause:

12.1 To the maximum extent permitted by law, you agree to defend, indemnify and hold harmless Google, its affiliates and their respective directors, officers, employees and agents from and against any and all claims, actions, suits or proceedings, as well as any and all losses, liabilities, damages, costs and expenses (including reasonable attorneys fees) arising out of or accruing from (a) your use of the SDK, (b) any application you develop on the SDK that infringes any copyright, trademark, trade secret, trade dress, patent or other intellectual property right of any person or defames any person or violates their rights of publicity or privacy, and (c) any non-compliance by you with this License Agreement.

Usage restriction:

3.1 Subject to the terms of this License Agreement, Google grants you a limited, worldwide, royalty-free, non-assignable and non-exclusive license to use the SDK solely to develop applications to run on the Android platform. 3.3 You may not use the SDK for any purpose not expressly permitted by this License Agreement. Except to the extent required by applicable third party licenses, you may not: (a) copy (except for backup purposes), modify, adapt, redistribute, decompile, reverse engineer, disassemble, or create derivative works of the SDK or any part of the SDK; or (b) load any part of the SDK onto a mobile handset or any other hardware device except a personal computer, combine any part of the SDK with other software, or distribute any software or device incorporating a part of the SDK.

If you know the URLs, you can still direct-download some of the binaries which don't embed the license, but all this feels fishy. GNU licensing didn't answer me (yet). Maybe debian-legal has an opinion? In any case, the difficulty to reproduce the *DK builds is worrying enough to warrant an independent rebuild. Did you notice this?

This is a burst of notes that I wrote in an e-mail in June when asked about it, and I'm not going to have any better steps since I don't remember even that amount as back then. I figured it's better to have it out than not.

So... if you want to use LUKS In-Place Conversion Tool, the notes below on converting a shipped-with-Ubuntu Dell XPS 13 Developer Edition (2015 Intel Broadwell model) may help you. There were a couple of small learnings to be had...

The page http://www.johannes-bauer.com/linux/luksipc/ itself is good and without errors, although funnily uses reiserfs as an example. It was only a bit unclear why I did save the initial_keyfile.bin since it was then removed in the next step (I guess it's for the case you want to have a recovery file hidden somewhere in case you forget the passphrase).

For using the tool I booted from a 14.04.2 LTS USB live image and operated there, including downloading and compiling luksipc in the live session. The exact reason of resizing before luksipc was a bit unclear to me at first so I simply indeed resized the main rootfs partition and left unallocated space in the partition table.

Then finally I ran ./luksipc -d /dev/sda4 etc.

I realized I want /boot to be on an unencrypted partition to be able to load the kernel + initrd from grub before entering into LUKS unlocking. I couldn't resize the luks partition anymore since it was encrypted... So I resized what I think was the empty small DIAGS partition (maybe used for some system diagnostic or something, I don't know), or possibly the next one that is the actual recovery partition one can reinstall the pre-installed Ubuntu from. And naturally I had some problems because it seems vfatresize tool didn't do what I wanted it to do and gparted simply crashed when I tried to use it first to do the same. Anyway, when done with getting some extra free space somewhere, I used the remaining 350MB for /boot where I copied the rootfs's /boot contents to.

After adding the passphrase in luks I had everything encrypted etc and decryptable, but obviously I could only access it from a live session by manual cryptsetup luksOpen + mount /dev/mapper/myroot commands. I needed to configure GRUB, and I needed to do it with the grub-efi-amd64 which was a bit unfamiliar to me. There's also grub-efi-amd64-signed I have installed now but I'm not sure if it was required for the configuration. Secure boot is not enabled by default in BIOS so maybe it isn't needed.

I did GRUB installation I think inside rootfs chroot where I also mounted /dev/sda6 as /boot (inside the rootfs chroot), ie mounted dev, sys with -o bind to under the chroot (from outside chroot) and mount -t proc proc proc too. I did a lot of trial and effort so I surely also tried from outside the chroot, in the live session, using some parameters to point to the mounted rootfs's directories...

I needed to definitely install cryptsetup etc inside the encrypted rootfs with apt, and I remember debugging for some time if they went to the initrd correctly after I executed mkinitramfs/update-initramfs inside the chroot.

At the end I had grub asking for the password correctly at bootup. Obviously I had edited the rootfs's /etc/fstab to include the new /boot partition, I changed / to be "UUID=/dev/mapper/myroot / ext4 errors=remount-ro 0 ", kept /boot/efi as coming from the /dev/sda1 and so on. I had also added "myroot /dev/sda4 none luks" to /etc/crypttab. I seem to also have GRUB_CMDLINE_LINUX="cryptdevice=/dev/sda4:myroot root=/dev/mapper/myroot" in /etc/default/grub.

FOSDEM 2016 is going to be great (again!) and you still have the chance to be one of the stars. Have you submitted your talk to the Desktops DevRoom yet? No? Remember: we will only accept proposals until December 6th. After that, the Organization Team will get busy and vote and choose the talks. Here is the full Call for Participation, in case you need to check the details on how to submit: FOSDEM Desktops DevRoom 2016 Call for Participation Topics include anything related to the Desktop: desktop environments, software development for desktop/cross-platform, applications, UI, etc

I've previously written about Canonical's obnoxious IP policy and how Mark Shuttleworth admits it's deliberately vague. After spending some time discussing specific examples with Canonical, I've been explicitly told that while Canonical will gladly give me a cost-free trademark license permitting me to redistribute unmodified Ubuntu binaries, they will not tell me what Any redistribution of modified versions of Ubuntu must be approved, certified or provided by Canonical if you are going to associate it with the Trademarks. Otherwise you must remove and replace the Trademarks and will need to recompile the source code to create your own binaries actually means.

Why does this matter? The free software definition requires that you be able to redistribute software to other people in either unmodified or modified form without needing to ask for permission first. This makes it clear that Ubuntu itself isn't free software - distributing the individual binary packages without permission is forbidden, even if they wouldn't contain any infringing trademarks[1]. This is obnoxious, but not inherently toxic. The source packages for Ubuntu could still be free software, making it fairly straightforward to build a free software equivalent.

Unfortunately, while true in theory, this isn't true in practice. The issue here is the apparently simple phrase you must remove and replace the Trademarks and will need to recompile the source code. "Trademarks" is defined later as being the words "Ubuntu", "Kubuntu", "Juju", "Landscape", "Edubuntu" and "Xubuntu" in either textual or logo form. The naive interpretation of this is that you have to remove trademarks where they'd be infringing - for instance, shipping the Ubuntu bootsplash as part of a modified product would almost certainly be clear trademark infringement, so you shouldn't do that. But that's not what the policy actually says. It insists that all trademarks be removed, whether they would embody an infringement or not. If a README says "To build this software under Ubuntu, install the following packages", a literal reading of Canonical's policy would require you to remove or replace the word "Ubuntu" even though failing to do so wouldn't be a trademark infringement. If an @ubuntu.com email address is present in a changelog, you'd have to change it. You wouldn't be able to ship the juju-core package without renaming it and the application within. If this is what the policy means, it's so impractical to be able to rebuild Ubuntu that it's not free software in any meaningful way.

This seems like a pretty ludicrous interpretation, but it's one that Canonical refuse to explicitly rule out. Compare this to Red Hat's requirements around Fedora - if you replace the fedora-logos, fedora-release and fedora-release-notes packages with your own content, you're good. A policy like this satisfies the concerns that Dustin raised over people misrepresenting their products, but still makes it easy for users to distribute modified code to other users. There's nothing whatsoever stopping Canonical from adopting a similarly unambiguous policy.

Mark has repeatedly asserted that attempts to raise this issue are mere FUD, but he won't answer you if you ask him direct questions about this policy and will insist that it's necessary to protect Ubuntu's brand. The reality is that if Debian had had an identical policy in 2004, Ubuntu wouldn't exist. The effort required to strip all Debian trademarks from the source packages would have been immense[2], and this would have had to be repeated for every release. While this policy is in place, nobody's going to be able to take Ubuntu and build something better. It's grotesquely hypocritical, especially when the Ubuntu website still talks about their belief that people should be able to distribute modifications without licensing fees.

All that's required for Canonical to deal with this problem is to follow Fedora's lead and isolate their trademarks in a small set of packages, then tell users that those packages must be replaced if distributing a modified version of Ubuntu. If they're serious about this being a branding issue, they'll do it. And if I'm right that the policy is deliberately obfuscated so Canonical can encourage people to buy licenses, they won't. It's easy for them to prove me wrong, and I'll be delighted if they do. Let's see what happens.

[1] The policy is quite clear on this. If you want to distribute something other than an unmodified Ubuntu image, you have two choices:

1. Gain approval or certification from Canonical
2. Remove all trademarks and recompile the source code

Note that option 2 requires you to rebuild even if there are no trademarks to remove.

[2] Especially when every source package contains a directory called "debian"

comments

I'm pleased with the way that Lumail2 development is proceeding, and it is reaching a point where there will be a second source-release. I've made a lot of changes to the repository recently, and most of them boil down to moving code from the C++ side of the application, over to the Lua side. This morning, for example, I updated the handing of index.limit to be entirely Lua based.

When you open a Maildir folder you see the list of messages it contains, as you would expect. The notion of the index.limit is that you can limit the messages displayed, for example:

• See all messages: Config:set( "index.limit", "all")
• See only new/unread messages: Config:set( "index.limit", "new")
• See only messages which arrived today: Config:set( "index.limit", "today")
• See only messages which contain "Steve" in their formatted version: Config:set( "index.limit", "steve")

These are just examples that are present as defaults, but they give an idea of how things can work. I guess it isn't so different to Mutt's "limit" facilities - but thanks to the dynamic Lua nature of the application you can add your own with relative ease.

One of the biggest changes, recently, was the ability to display coloured text! That was always possible before, but a single line could only be one colour. Now colours can be mixed within a line, so this works as you might imagine: Panel:append( "$[RED]This is red, $[GREEN]green, $[WHITE]white, and $[CYAN]cyan!" ) Other changes include a persistant cache of "stuff", which is Lua-based, the inclusion of at least one luarocks library to parse Date: headers, and a simple API for all our objects. All good stuff. Perhaps time for a break in the next few weeks, but right now I think I'm making useful updates every other evening or so.

Writing programs is fun, but making them fast can be a pain. Python programs are no exception to that, but the basic profiling toolchain is actually not that complicated to use. Here, I would like to show you how you can quickly profile and analyze your Python code to find what part of the code you should optimize. What's profiling? Profiling a Python program is doing a dynamic analysis that measures the execution time of the program and everything that compose it. That means measuring the time spent in each of its functions. This will give you data about where your program is spending time, and what area might be worth optimizing. It's a very interesting exercise. Many people focus on local optimizations, such as determining e.g. which of the Python functions range or xrange is going to be faster. It turns out that knowing which one is faster may never be an issue in your program, and that the time gained by one of the functions above might not be worth the time you spend researching that, or arguing about it with your colleague. Trying to blindly optimize a program without measuring where it is actually spending its time is a useless exercise. Following your guts alone is not always sufficient. There are many types of profiling, as there are many things you can measure. In this exercise, we'll focus on CPU utilization profiling, meaning the time spent by each function executing instructions. Obviously, we could do many more kind of profiling and optimizations, such as memory profiling which would measure the memory used by each piece of code something I talk about in The Hacker's Guide to Python. cProfile Since Python 2.5, Python provides a C module called cProfile which has a reasonable overhead and offers a good enough feature set. The basic usage goes down to:
Though you can also run a script with it, which turns out to be handy:
This prints out all the function called, with the time spend in each and the number of times they have been called. Advanced visualization with KCacheGrind While being useful, the output format is very basic and does not make easy to grab knowledge for complete programs. For more advanced visualization, I leverage KCacheGrind. If you did any C programming and profiling these last years, you may have used it as it is primarily designed as front-end for Valgrind generated call-graphs. In order to use, you need to generate a cProfile result file, then convert it to KCacheGrind format. To do that, I use pyprof2calltree.
And the KCacheGrind window magically appears!

Concrete case: Carbonara optimization I was curious about the performances of Carbonara, the small timeserie library I wrote for Gnocchi. I decided to do some basic profiling to see if there was any obvious optimization to do. In order to profile a program, you need to run it. But running the whole program in profiling mode can generate a lot of data that you don't care about, and adds noise to what you're trying to understand. Since Gnocchi has thousands of unit tests and a few for Carbonara itself, I decided to profile the code used by these unit tests, as it's a good reflection of basic features of the library. Note that this is a good strategy for a curious and naive first-pass profiling. There's no way that you can make sure that the hotspots you will see in the unit tests are the actual hotspots you will encounter in production. Therefore, a profiling in conditions and with a scenario that mimics what's seen in production is often a necessity if you need to push your program optimization further and want to achieve perceivable and valuable gain. I activated cProfile using the method described above, creating a cProfile.Profile object around my tests (I actually started to implement that in testtools). I then run KCacheGrind as described above. Using KCacheGrind, I generated the following figures.

The test I profiled here is called test_fetch and is pretty easy to understand: it puts data in a timeserie object, and then fetch the aggregated result. The above list shows that 88 % of the ticks are spent in set_values (44 ticks over 50). This function is used to insert values into the timeserie, not to fetch the values. That means that it's really slow to insert data, and pretty fast to actually retrieve them. Reading the rest of the list indicates that several functions share the rest of the ticks, update, _first_block_timestamp, _truncate, _resample, etc. Some of the functions in the list are not part of Carbonara, so there's no point in looking to optimize them. The only thing that can be optimized is, sometimes, the number of times they're called.

The call graph gives me a bit more insight about what's going on here. Using my knowledge about how Carbonara works, I don't think that the whole stack on the left for _first_block_timestamp makes much sense. This function is supposed to find the first timestamp for an aggregate, e.g. with a timestamp of 13:34:45 and a period of 5 minutes, the function should return 13:30:00. The way it works currently is by calling the resample function from Pandas on a timeserie with only one element, but that seems to be very slow. Indeed, currently this function represents 25 % of the time spent by set_values (11 ticks on 44). Fortunately, I recently added a small function called _round_timestamp that does exactly what _first_block_timestamp needs that without calling any Pandas function, so no resample. So I ended up rewriting that function this way:
And then I re-run the exact same test to compare the output of cProfile.

The list of function seems quite different this time. The number of time spend used by set_values dropped from 88 % to 71 %.

The call stack for set_values shows that pretty well: we can't even see the _first_block_timestamp function as it is so fast that it totally disappeared from the display. It's now being considered insignificant by the profiler. So we just speed up the whole insertion process of values into Carbonara by a nice 25 % in a few minutes. Not that bad for a first naive pass, right?

The second update in the 0.12.* series of Rcpp is now on the CRAN network for GNU R. As usual, I will also push a Debian package. This follows the 0.12.0 release from late July which started to add some serious new features, and builds upon the 0.12.1 release in September. It also marks the sixth release this year where we managed to keep a steady bi-montly release frequency. Rcpp has become the most popular way of enhancing GNU R with C or C++ code. As of today, 512 packages on CRAN depend on Rcpp for making analytical code go faster and further. That is up by more than fifty package from the last release in September (and we recently blogged about crossing 500 dependents). This release once again features pull requests from two new contributors with Nathan Russell and Tianqi Chen joining in. As shown below, other recent contributors (such as such as Dan) are keeping at it too. Keep'em coming! Luke Tierney also email about a code smell he spotted and which we took care of. A big Thank You! to everybody helping with code, bug reports or documentation. See below for a detailed list of changes extracted from the NEWS file.

Changes in Rcpp version 0.12.2 (2015-11-14)

• Changes in Rcpp API:
  • Correct return type in product of matrix dimensions (PR #374 by Florian)
  • Before creating a single String object from a SEXP, ensure that it is from a vector of length one (PR #376 by Dirk, fixing #375).
  • No longer use STRING_ELT as a left-hand side, thanks to a heads-up by Luke Tierney (PR #378 by Dirk, fixing #377).
  • Rcpp Module objects are now checked more carefully (PR #381 by Tianqi, fixing #380)
  • An overflow in Matrix column indexing was corrected (PR #390 by Qiang, fixing a bug reported by Allessandro on the list)
  • Nullable types can now be assigned R_NilValue in function signatures. (PR #395 by Dan, fixing issue #394)
  • operator<<() now always shows decimal points (PR #396 by Dan)
  • Matrix classes now have a transpose() function (PR #397 by Dirk fixing #383)
  • operator<<() for complex types was added (PRs #398 by Qiang and #399 by Dirk, fixing #187)
• Changes in Rcpp Attributes:
  • Enable export of C++ interface for functions that return void.
• Changes in Rcpp Sugar:
  • Added new Sugar function cummin(), cummax(), cumprod() (PR #389 by Nathan Russell fixing #388)
  • Enabled sugar math operations for subsets; e.g. x[y] + x[z]. (PR #393 by Kevin and Qiang, implementing #392)
• Changes in Rcpp Documentation:
  • The NEWS file now links to GitHub issue tickets and pull requests.
  • The Rcpp.bib file with bibliographic references was updated.

Thanks to CRANberries, you can also look at a diff to the previous release As always, even fuller details are on the Rcpp Changelog page and the Rcpp page which also leads to the downloads page, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.