Sometimes waiting for a delayed flight helps to implement things. I added some basic support for the new Debian SSO Client Certificate feature to paste.debian.net. If you are using such a certificate most anti-spam restrictions, code limitations and so on won t count for you anymore.

Categories:

• Debian-Blog
• Programming
• Tools

Keywords:

• laptop-mode-tools
• LMT
• systemd

Like:

It's 2004 and I'm in Oldenburg DE, working on the Debian Installer. Colin and I pair program on partman, its new partitioner, to get it into shape. We've somewhat reluctantly decided to use it. Partman is in some ways a beautful piece of work, a mass of semi-object-oriented, super extensible shell code that sprang fully formed from the brow of Anton. And in many ways, it's mad, full of sector alignment twiddling math implemented in tens of thousands of lines of shell script scattered amoung hundreds of tiny files that are impossible to keep straight. In the tiny Oldenburg Developers Meeting, full of obscure hardware and crazy intensity of ideas like porting Debian to VAXen, we hack late into the night, night after night, and crash on the floor. It's 2015 and I'm at a Chinese bakery, then at the Berkeley pier, then in a SF food truck lot, catching half an hour here and there in my vacation to add some features to Propellor. Mostly writing down data types for things like filesystem formats, partition layouts, and then some small amount of haskell code to use them in generic ways. Putting these peices together and reusing stuff already in Propellor (like chroot creation). Before long I have this, which is only 2 undefined functions away from (probably) working: This is at least a replication of vmdebootstrap, generating a bootable disk image from that config and 400 lines of code, with enormous customizability of the disk image contents, using all the abilities of Propellor. But is also, effectively, a replication of everything partman is used for (aside from UI and RAID/LVM). What a difference a decade and better choices of architecture make! In many ways, this is the loosely coupled, extensible, highly configurable system partman aspired to be. Plus elegance. And I'm writing it on a lark, because I have some spare half hours in my vacation. Past Debian Installer team lead Tollef stops by for lunch, I show him the code, and we have the conversation old d-i developers always have about partman. I can't say that partman was a failure, because it's been used by millions to install Debian and Ubuntu and etc for a decade. Anything that deletes that many Windows partitions is a success. But it's been an unhappy success. Nobody has ever had a good time writing partman recipes; the code has grown duplication and unmaintainability. I can't say that these extensions to Propellor will be a success; there's no plan here to replace Debian Installer (although with a few hundred more lines of code, propellor is d-i 2.0); indeed I'm just adding generic useful stuff and building further stuff out of it without any particular end goal. Perhaps that's the real difference.

Follow me on Identi.ca or Twitter or Diaspora* Don t know Retweet? My last post about it introduced this small Twitter bot whichs just retweets (for now) every tweets from a Twitter account to another one. Retweet

• Retweet 0.2 on Github
• The Retweet project on Github
• The official documentation of the Retweet project

Retweet was created in order to improve the Journal du hacker Twitter account. The Journal du hacker is a Hacker News-like French-speaking website. Especially useful to broadcast news through a network of Twitter accounts, Retweet was improved to bump Python version to 3.4 and to improve pep8 compliance (work in progress). The project is also well documented and should be quite simple to install, configure and use. After my first blog post about Retweet, new users gave me feedback about it and I now have great ideas for future features for the next release. What about you? If you try it, please tell me what you think about it, opening a bug request or ask for new features. Or just write your comment here ;)

Like each month, here comes a report about the work of paid contributors to Debian LTS. Individual reports In July, 79.50 work hours have been dispatched among 7 paid contributors. Their reports are available:

• Ben Hutchings did 14.75 hours.
• Guido G nther did 8 hours.
• Mike Gabriel did 8 hours.
• Rapha l Hertzog did 15 hours.
• Santiago Ruano Rinc n did 14.75 hours.
• Scott Kitterman did 4 hours.
• Thorsten Alteholz did 15 hours.

Evolution of the situation August has seen a small decrease in terms of sponsored hours (71.50 hours per month) because two sponsors did not pay their renewal invoice on time. That said they reconfirmed their willingness to support us and things should be fixed after the summer. And we should be able to reach our first milestone of funding the equivalent of a half-time position, in particular since a new platinum sponsor might join the project. DebConf 15 happened this month and Debian LTS was featured in a talk and in a work session. Have a look at the video recordings:

• Debian Long Term Support: Past Present and Future (slides)
• Preparing for Wheezy LTS

In terms of security updates waiting to be handled, the situation is better than last month: the dla-needed.txt file lists 20 packages awaiting an update (4 less than last month), the list of open vulnerabilities in Squeeze shows about 22 affected packages in total (11 less than last month). The new LTS frontdesk ensures regular triage of CVE reports and the difference between both counts dropped significantly. That s good! Thanks to our sponsors Thanks to Sig-I/O, a new bronze sponsor, which joins our 35 other sponsors.

• Gold sponsors:
• The Positive Internet (for 14 months already)
• Linode LLC (for 3 months already)
• Silver sponsors:
• David Ayers IntarS Austria (for 14 months already)
• Blablacar (for 13 months already)
• Domeneshop AS (for 13 months already)
• Universit Lille 3 (for 13 months already)
• Trollweb Solutions (for 11 months already)
• Gandi SAS (for 8 months already)
• University of Luxembourg (for 5 months already)
• Rentabiliweb Group (for 4 months already)
• Bronze sponsors:
• Evolix (for 14 months already)
• Offensive Security (for 14 months already)
• Seznam.cz, a.s. (for 14 months already)
• Freeside Internet Service (for 13 months already)
• MyTux (for 13 months already)
• Intevation GmbH (for 11 months already)
• Linuxhotel GmbH (for 11 months already)
• Nantes M tropole (for 11 months already)
• Daevel SARL (for 9 months already)
• FOSSter (for 9 months already)
• Bitfolk LTD (for 8 months already)
• Megaspace Internet Services GmbH (for 8 months already)
• Gree, Inc. (for 7 months already)
• Greenbone Networks GmbH (for 7 months already)
• NUMLOG (for 7 months already)
• WinGo AG (for 7 months already)
• Ecole Centrale de Nantes LHEEA (for 3 months already)
• Sig-I/O

One comment Liked this article? Click here. My blog is Flattr-enabled.

A new maintenance release 0.4.3 of RProtoBuf is now on CRAN. RProtoBuf provides R bindings for the Google Protocol Buffers ("Protobuf") data encoding library used and released by Google, and deployed as a language and operating-system agnostic protocol by numerous projects. This release comes upon the request of CRAN and adds additional import statements to the NAMESPACE file. While we were at it, a few more things got cleaned up and edited---but no new code was added. Full details are below.

Changes in RProtoBuf version 0.4.3 (2015-08-25)

• Declare additional imports from methods in NAMESPACE.
• Travis CI tests now run faster as all CRAN dependencies are installed as binaries.
• The tools/winlibs.R script now tests for R (< 3.3.0) before calling the (soon-to-be phased out) setInternet2() function.
• Several small edits were made to DESCRIPTION to clarify library dependencies, provide additonal references and conform to now-current R packaging standards.

CRANberries also provides a diff to the previous release. The RProtoBuf page has a package vignette, a a 'quick' overview vignette, and a unit test summary vignette. Questions, comments etc should go to the GitHub issue tracker off the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Debian/sid is going through a big restructuring with the switch to a new gcc and libstc++. Furthermore, libcec3 is now the default. So I have updated my PHT builds for Debian/sid to build and install on the current status, both for amd64 and i386. Add the following lines to your sources.list: You can also grab the binary for amd64 directly here for amd64 and i386, you can get the source package with The release file and changes file are signed with my official Debian key 0x860CDC13. For Debian/testing I am waiting until the transition has settled. Please wait a bit more. Now be ready for enjoying the next movie!

A good amount of the Debian reproducible builds team had the chance to enjoy face-to-face interactions during DebConf15.

Names in red and blue were all present at DebConf15

Hugging people with whom one has been working tirelessly for months gives a lot of warm-fuzzy feelings. Several recorded and hallway discussions paved the way to solve the remaining issues to get reproducible builds part of Debian proper. Both talks from the Debian Project Leader and the release team mentioned the effort as important for the future of Debian. A forty-five minutes talk presented the state of the reproducible builds effort. It was then followed by an hour long roundtable to discuss current blockers regarding dpkg, .buildinfo and their integration in the archive. Toolchain fixes

• Kenneth J. Pronovici uploaded epydoc/3.0.1+dfsg-12 which makes class and modules ordering predictable (#795835) and fixes __repr__ so memory addresses don't appear in docs (#795826). Patches by Val Lorentz.
• Sergei Golovan uploaded erlang/1:18.0-dfsg-2 which adds support for SOURCE_DATE_EPOCH to erlc. Patch by Chris West (Faux) and Chris Lamb.
• Dmitry Shachnev uploaded sphinx/1.3.1-5 which make grammar, inventory, and JavaScript locales generation deterministic. Original patch by Val Lorentz.
• St phane Glondu uploaded ocaml/4.02.3-2 to experimental, making startup files and native packed libraries deterministic. The patch adds deterministic .file to the assembler output.
• Enrico Tassi uploaded lua-ldoc/1.4.3-3 which now pass the -d option to txt2man and add the --date option to override the current date.

Reiner Herrmann submitted a patch to make rdfind sort the processed files before doing any operation. Chris Lamb proposed a new patch for wheel implementing support for SOURCE_DATE_EPOCH instead of the custom WHEEL_FORCE_TIMESTAMP. akira sent one making man2html SOURCE_DATE_EPOCH aware.

• #796330 on d-shlibs by Reiner Herrmann: sort with LC_ALL set to C.

St phane Glondu reported that dpkg-source would not respect tarball permissions when unpacking under a umask of 002. After hours of iterative testing during the DebConf workshop, Sandro Knau created a test case showing how pdflatex output can be non-deterministic with some PNG files. Packages fixed The following 65 packages became reproducible due to changes in their build dependencies: alacarte, arbtt, bullet, ccfits, commons-daemon, crack-attack, d-conf, ejabberd-contrib, erlang-bear, erlang-cherly, erlang-cowlib, erlang-folsom, erlang-goldrush, erlang-ibrowse, erlang-jiffy, erlang-lager, erlang-lhttpc, erlang-meck, erlang-p1-cache-tab, erlang-p1-iconv, erlang-p1-logger, erlang-p1-mysql, erlang-p1-pam, erlang-p1-pgsql, erlang-p1-sip, erlang-p1-stringprep, erlang-p1-stun, erlang-p1-tls, erlang-p1-utils, erlang-p1-xml, erlang-p1-yaml, erlang-p1-zlib, erlang-ranch, erlang-redis-client, erlang-uuid, freecontact, givaro, glade, gnome-shell, gupnp, gvfs, htseq, jags, jana, knot, libconfig, libkolab, libmatio, libvsqlitepp, mpmath, octave-zenity, openigtlink, paman, pisa, pynifti, qof, ruby-blankslate, ruby-xml-simple, timingframework, trace-cmd, tsung, wings3d, xdg-user-dirs, xz-utils, zpspell. The following packages became reproducible after getting fixed:

• apr/1.5.2-3 by Stefan Fritsch.
• aprx/2.08.svn593+dfsg-2 uploaded by Colin Tuckley, original patch by Chris Lamb.
• blkreplay/1.0-3 uploaded by Andrew Shadura, patch by Dhole.
• cal3d/0.11.0-6 uploaded by Manuel A. Fernandez Montecelo, patch by akira.
• cgsi-gsoap/1.3.8-1 by Mattias Ellert.
• eyefiserver/2.4+dfsg-1 by Jean-Michel Vourg re.
• gnujump/1.0.8-3 uploaded by Evgeni Golov, original patch by Chris Lamb.
• hkgerman/1:2-29 by Roland Rosenfeld.
• jove/4.16.0.73-4 by Cord Beermann.
• libevhtp/1.2.10-3 by Vincent Bernat.
• libmkdoc-xml-perl/0.75-4 uploaded by gregor herrmann, original patch by Niko Tyni.
• libparse-debianchangelog-perl/1.2.0-6 by Niko Tyni.
• librostlab-blast/1.0.1-5 uploaded by Andreas Tille, original patch by Chris Lamb.
• libxray-absorption-perl/3.0.1-3 uploaded by gregor herrmann, original patch by Niko Tyni.
• lua-penlight/1.3.2-2 by Enrico Tassi.
• mosquitto/1.4.3-1 by Roger A. Light.
• nagios-plugins-contrib/15.20150818 by Jan Wagner and Bernd Zeimetz.
• nn/6.7.3-10 uploaded by Cord Beermann, original patch by Chris Lamb.
• pybik/2.1-1 by B. Clausius.
• pyepr/0.9.3-1 uploaded by Antonio Valentino, original patch by Juan Picca.
• python-xlrd/0.9.4-1 by Vincent Bernat.
• transmissionrpc/0.11-2 uploaded by Vincent Bernat, original patch by Juan Picca.
• unoconv/0.7-1.1 sponsored by Vincent Bernat, fix by Dhole.
• vim-latexsuite/20141116.812-1 uploaded by Johann Felix Soden, original patch by Chris Lamb.
• volk/1.0.2-2 by A. Maitland Bottoms.
• xbmc/2:13.2+dfsg1-5 by Balint Reczey.
• xdotool/1:3.20150503.1-2 uploaded by Daniel Kahn Gillmor, initial patch by Chris Lamb.
• xfig/1:3.2.5.c-5 by Roland Rosenfeld.
• xfireworks/1.3-10 uploaded by Yukiharu YABUKI, original patch by Chris Lamb.
• xul-ext-monkeysphere/0.8-2 uploaded by Daniel Kahn Gillmor, original patch by Dhole.

Uploads that might have fixed reproducibility issues:

• brian/1.4.1-3 uploaded by Yaroslav Halchenko, original patch by Juan Picca.
• opennebula/4.12.3+dfsg-1 by Dmitry Smirnov.
• pcsx2/1.3.1-1008-g9f291a6+dfsg-1 by Miguel A. Col n V lez.
• webassets/3:0.11-1 uploaded by Agustin Henze, original patch by Reiner Herrmann.

Some uploads fixed some reproducibility issues but not all of them:

• apache2/2.4.16-3 uploaded by Stefan Fritsch, original patch by Jean-Michel Vourg re.
• gerris/20131206+dfsg-6 uploaded by Anton Gladky, original patch by Reiner Herrmann.
• kodi/15.1+dfsg1-1 by Balint Reczey.
• zshdb/0.05+git20101031-4 uploaded by Iain R. Learmonth, original patch by Chris Lamb.

Patches submitted which have not made their way to the archive yet:

• #795861 on fakeroot by Val Lorentz: set the mtime of all files to the time of the last debian/changelog entry.
• #795870 on fatresize by Chris Lamb: set build date to the time of the latest debian/changelog entry.
• #795945 on projectl by Reiner Herrmann: sort with LC_ALL set to C.
• #795977 on dahdi-tools by Dhole: set the timezone to UTC before calling asciidoc.
• #795981 on x11proto-input by Dhole: set the timezone to UTC before calling asciidoc.
• #795983 on dbusada by Dhole: set the timezone to UTC before calling asciidoc.
• #795984 on postgresql-plproxy by Dhole: set the timezone to UTC before calling asciidoc.
• #795985 on xorg by Dhole: set the timezone to UTC before calling asciidoc.
• #795987 on pngcheck by Dhole: set the date in the man pages to the latest debian/changelog entry.
• #795997 on python-babel by Val Lorentz: make build timestamp independent from the timezone and remove the name of the build system locale from the documentation.
• #796092 on a7xpg by Reiner Herrmann: sort with LC_ALL set to C.
• #796212 on bittornado by Chris Lamb: remove umask-varying permissions.
• #796251 on liblucy-perl by Niko Tyni: generate lib/Lucy.xs in a deterministic order.
• #796271 on tcsh by Reiner Herrmann: sort with LC_ALL set to C.
• #796275 on hspell by Reiner Herrmann: remove timestamp from aff files generated by mk_he_affix.
• #796324 on fftw3 by Reiner Herrmann: remove date from documentation files.
• #796335 on nasm by Val Lorentz: remove extra timestamps from the build system.
• #796360 on libical by Chris Lamb: removes randomess caused Perl in generated icalderivedvalue.c.
• #796375 on wcd by Dhole: set the date in the man pages to the latest debian/changelog entry.
• #796376 on mapivi by Dhole: set the date in the man pages to the latest debian/changelog entry.
• #796527 on vserver-debiantools by Dhole: set the date in the man pages to the latest debian/changelog entry.

St phane Glondu reported two issues regarding embedded build date in omake and cduce. Aur lien Jarno submitted a fix for the breakage of make-dfsg test suite. As binutils now creates deterministic libraries by default, Aur lien's patch makes use of a wrapper to give the U flag to ar. Reiner Herrmann reported an issue with pound which embeds random dhparams in its code during the build. Better solutions are yet to be found. reproducible.debian.net Package pages on reproducible.debian.net now have a new layout improving readability designed by Mattia Rizzolo, h01ger, and Ulrike. The navigation is now on the left as vertical space is more valuable nowadays. armhf is now enabled on all pages except the dashboard. Actual tests on armhf are expected to start shortly. (Mattia Rizzolo, h01ger) The limit on how many packages people can schedule using the reschedule script on Alioth has been bumped to 200. (h01ger) mod_rewrite is now used instead of JavaScript for the form in the dashboard. (h01ger) Following the rename of the software, debbindiff has mostly been replaced by either diffoscope or differences in generated HTML and IRC notification output. Connections to UDD have been made more robust. (Mattia Rizzolo) diffoscope development diffoscope version 31 was released on August 21^st. This version improves fuzzy-matching by using the tlsh algorithm instead of ssdeep. New command line options are available: --max-diff-input-lines and --max-diff-block-lines to override limits on diff input and output (Reiner Herrmann), --debugger to dump the user into pdb in case of crashes (Mattia Rizzolo). jar archives should now be detected properly (Reiner Herrman). Several general code cleanups were also done by Chris Lamb. strip-nondeterminism development Andrew Ayer released strip-nondeterminism version 0.010-1. Java properties file in jar should now be detected more accurately. A missing dependency spotted by St phane Glondu has been added. Testing directory ordering issues: disorderfs During the reproducible builds workshop at DebConf, participants identified that we were still short of a good way to test variations on filesystem behaviors (e.g. file ordering or disk usage). Andrew Ayer took a couple of hours to create disorderfs. Based on FUSE, disorderfs in an overlay filesystem that will mount the content of a directory at another location. For this first version, it will make the order in which files appear in a directory random. Documentation update Dhole documented how to implement support for SOURCE_DATE_EPOCH in Python, bash, Makefiles, CMake, and C. Chris Lamb started to convert the wiki page describing SOURCE_DATE_EPOCH into a Freedesktop-like specification in the hope that it will convince more upstream to adopt it. Package reviews 44 reviews have been removed, 192 added and 77 updated this week. New issues identified this week: locale_dependent_order_in_devlibs_depends, randomness_in_ocaml_startup_files, randomness_in_ocaml_packed_libraries, randomness_in_ocaml_custom_executables, undeterministic_symlinking_by_rdfind, random_build_path_by_golang_compiler, and images_in_pdf_generated_by_latex. 117 new FTBFS bugs have been reported by Chris Lamb, Chris West (Faux), and Niko Tyni. Misc. Some reproducibility issues might face us very late. Chris Lamb noticed that the test suite for python-pykmip was now failing because its test certificates have expired. Let's hope no packages are hiding a certificate valid for 10 years somewhere in their source! Pictures courtesy and copyright of Debian's own paparazzi: Aigars Mahinovs.

During Jacob Applebaum's talk at DebConf15, he noted that Debian should TLS-enable all services, especially the mirrors. His reasoning was that when a high-value target downloads a security update for package foo, an adversary knows that they are still using a vulnerable version of foo and try to attack before the security update has been installed. In this specific case, TLS is not of much use though. If the target downloads 4.7 MiB right after a security update with 4.7 MiB has been released, or downloads from security.debian.org, it's still obvious what's happening. Even padding won't help much as the 5 MiB download will also be suspicious. The mere act of downloading anything from the mirrors after an update has been released is reason enough to try an attack. The solution, is, of course, Tor. weasel was nice enough to set up a hidden service on Debian's infrastructure; initally we agreed that he would just give me a VM and I would do the actual work, but he went the full way on his own. Thanks :) This service is not redundant, it uses a key which is stored on the local drive, the .onion will change, and things are expected to break. But at least this service exists now and can be used, tested, and put under some load:

http://vwakviie2ienjx6t.onion/

I couldn't get apt-get to be content with a .onion in /etc/apt/sources.list and Acquire::socks::proxy "socks://127.0.0.1:9050"; in /etc/apt/apt.conf, but the torify wrapper worked like a charm. What follows is, to the best of my knowledge, the first ever download from Debian's "official" Tor-enabled mirror:

~ # apt-get install torsocks
~ # mv /etc/apt/sources.list /etc/apt/sources.list.backup
~ # echo 'deb http://vwakviie2ienjx6t.onion/debian/ unstable main non-free contrib' > /etc/apt/sources.list
~ # torify apt-get update
Get:1 http://vwakviie2ienjx6t.onion unstable InRelease [215 kB]
Get:2 http://vwakviie2ienjx6t.onion unstable/main amd64 Packages [7548 kB]
Get:3 http://vwakviie2ienjx6t.onion unstable/non-free amd64 Packages [91.9 kB]
Get:4 http://vwakviie2ienjx6t.onion unstable/contrib amd64 Packages [58.5 kB]
Get:5 http://vwakviie2ienjx6t.onion unstable/main i386 Packages [7541 kB]
Get:6 http://vwakviie2ienjx6t.onion unstable/non-free i386 Packages [85.4 kB]
Get:7 http://vwakviie2ienjx6t.onion unstable/contrib i386 Packages [58.1 kB]
Get:8 http://vwakviie2ienjx6t.onion unstable/contrib Translation-en [45.7 kB]
Get:9 http://vwakviie2ienjx6t.onion unstable/main Translation-en [5060 kB]
Get:10 http://vwakviie2ienjx6t.onion unstable/non-free Translation-en [80.8 kB]
Fetched 20.8 MB in 2min 0s (172 kB/s)
Reading package lists... Done
~ # torify apt-get install vim
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  vim-common vim-nox vim-runtime vim-tiny
Suggested packages:
  ctags vim-doc vim-scripts cscope indent
The following packages will be upgraded:
  vim vim-common vim-nox vim-runtime vim-tiny
5 upgraded, 0 newly installed, 0 to remove and 661 not upgraded.
Need to get 0 B/7719 kB of archives.
After this operation, 2048 B disk space will be freed.
Do you want to continue? [Y/n] 
Retrieving bug reports... Done
Parsing Found/Fixed information... Done
Reading changelogs... Done
(Reading database ... 316427 files and directories currently installed.)
Preparing to unpack .../vim-nox_2%3a7.4.826-1_amd64.deb ...
Unpacking vim-nox (2:7.4.826-1) over (2:7.4.712-3) ...
Preparing to unpack .../vim_2%3a7.4.826-1_amd64.deb ...
Unpacking vim (2:7.4.826-1) over (2:7.4.712-3) ...
Preparing to unpack .../vim-tiny_2%3a7.4.826-1_amd64.deb ...
Unpacking vim-tiny (2:7.4.826-1) over (2:7.4.712-3) ...
Preparing to unpack .../vim-runtime_2%3a7.4.826-1_all.deb ...
Unpacking vim-runtime (2:7.4.826-1) over (2:7.4.712-3) ...
Preparing to unpack .../vim-common_2%3a7.4.826-1_amd64.deb ...
Unpacking vim-common (2:7.4.826-1) over (2:7.4.712-3) ...
Processing triggers for man-db (2.7.0.2-5) ...
Processing triggers for mime-support (3.58) ...
Processing triggers for desktop-file-utils (0.22-1) ...
Processing triggers for hicolor-icon-theme (0.13-1) ...
Setting up vim-common (2:7.4.826-1) ...
Setting up vim-runtime (2:7.4.826-1) ...
Processing /usr/share/vim/addons/doc
Setting up vim-nox (2:7.4.826-1) ...
Setting up vim (2:7.4.826-1) ...
Setting up vim-tiny (2:7.4.826-1) ...
~ # 

More services will follow. noodles, weasel, and me agreed that the project as a whole should aim to Tor-enable the complete package lifecycle, package information, and the website. Maybe a more secure install option on the official images which, amongst others, sets up apt, apt-listbugs, dput, reportbug, et al up to use Tor without further configuration could even be a realistic stretch goal.

I tried to start to write this blog entry like I usually do: Type along what goes through my mind and see where I'm heading. This won't work out right now for various reasons, mostly because there is so much going on that I don't have the time to finish that in a reasonable time and I want to publish this today still. So please excuse me for being way more brief than I usually am, and hopefully I'll find the time to expand some things when asked or come back to that later. Part of the reason of me being short on time is different stuff going on in my private life which requires additional attention. A small part of this is also something that I hinted in a former blog entry: I switched my job in June. I really was looking forward to this. I made them aware of what the name Rhonda means to me and it's definitely extremely nice to be addressed with female pronouns at work. And also I'm back in a system administration job which means there is an interest overlap with my work on Debian, so a win-win situation on sooo many levels! I'm at DebConf15 since almost two weeks now. On my way here I was complimented on my outfit by a security guard at the Vienna airport which surprised me but definitely made my day. I was wearing one of these baggy hippie pants (which was sent to me by a fine lady I met at MiniDebConf Bucharest) but pulled up the leg parts to the knees so it could be perceived as a skirt instead. Since I came here I was pretty busy with taking care of DCschedule bot adjustments (like, changing topic and twittering from @DebConf at the start of the talks), helping out with the video team when I noticed there was a lack of people (which is a hint for that you might want to help with the video team in the future too, it's important for remote people but also for yourself because you can't attend multiple sessions at the same time). And I have to repeat myself, this is the place I feel home amongst my extended family, even though I it still is sometimes for me to get to speak up in certain groups. I though believe it's more an issue of certain individuals taking up a lot of space in discussions without giving (more shy) people in the round the space to also join in. I guess it might be the time that we need a session on dominant talking patterns for next year and how to work against them. I absolutely enjoyed such a session during last year's FemCamp in Vienna which set the tone for the rest of the conference, and it was simply great. And then there was the DebConf Poetry Night. I'm kinda disappointed with the outcome this year. It wasn't able to attract as much people anticipated, which I to some degree account to me not making people aware of it well enough, overlapping with a really great band playing at the same time in competition, and even though the place where we did it sounded like a good idea at first, it didn't had enough light for someone to read something from a book (but that was solved through smartphone lights). I know that most people did enjoy it, so it was good to do it, but I'm still a fair bit disappointed with the outcome and will try to work on improving on that grounds for next year. :) With all this going on there unfortunately wasn't as much time as I would have liked to spend with people I haven't seen for a long time, or new people I haven't met yet. Given that this year's DebConf had an height in attendees (526 being here at certain times during the two weeks, and just today someone new arrived too, so that doesn't even have to be the final number) it makes it a bit painful to have picked up so many tasks and thus lost some chances to socialize as much as I would have liked to. So, if you are still here and have the feeling we should have talked more, please look for me. As Bdale pointed out correctly in the New to DebConf BoF (paraphrased): When you see us DebConf old timers speaking to someone else and you feel like you don't want to disturb, please do disturb and speak to us. I always enjoyed to get to know new people. This for me always is one of the important aspects of DebConf. Also, I am very very happy to have received feedback from different people about both my tweets and my blog, thank you a lot of that. It is really motivating to keep going. So, lets enjoy the last few hours of DebConf! Another last side notice: While my old name in the Debian LDAP did manage to find some wrongly displayed names in the DebConf website, like for speakers, or volunteers, it was clear to me that having it exposed through SSO.debian.org isn't really something I appreciate. So I took the chance and spoke to Luca from the DSA team right here today, and ... got it fixed. I love it! Next step is getting my gpg key exchanged, RT ticket is coming up. :)

/debian permanent link Comments: 1 Flattr this

Based on a discussion around breakfast, and encouraged by the people at the table, I hereby call for a new release goal (or challenge, whatever you prefer to call it):


Every package maintainer should remove one of their packages from the archive.


It's dead simple. It is acceptable to adopt a package to replace the one that has been removed, or to add a new one to the archive.
For tracking purposes please include "for RG" (release goal) in the removal request to ftp.debian.org.


And how about a debconf challenge? how about filing over 100 removal requests before the end of Debconf 15 on Saturday night? blog about it, dent/twit about it, spam IRC about it!


The idea came up after discussing about how us as package maintainers refuse to remove our obsolete or unused packages. So yes, that may also include the very first package that you got into the archive.


Sad news, good news.

A bi-lingual poem created on inspiration from Debconf15 and in honor of Debconf Poetry Night by Rhonda

Du ... Du hast ... Du hast apt ... Du hast apt gebrochen! Reconsider your disk usage, And APT might work again.

(as usual - licenced as CC-BY V3+ or GPLv2+)

Another relatively minor release, this of a software package that I've technically orphaned. But, well, it needed a fix, and it only took a few hours. krb5-sync is a system for synchronizing passwords from a Kerberos KDC to an Active Directory realm (which I no longer personally use). The primary change in this release is some tweaks to the silent mode of krb5-sync-backend to keep it from spamming output about transient errors if accounts are slow to materialize on the Active Directory side. It also incorporates changes from the Debian packaging to relax the timing on some tests. You can get the latest version from the krb5-sync distribution page.

Some years ago Barwon South Water gave LUV 3 old 1RU Sun servers for any use related to free software. We gave one of those servers to the Canberra makerlab and another is used as the server for the LUV mailing lists and web site and the 3rd server was put aside for training. The servers have hot-swap 15,000rpm SAS disks IE disks that have a replacement cost greater than the budget we have for hardware. As we were given a spare 70G disk (and a 140G disk can replace a 70G disk) the LUV server has 2*70G disks and the 140G disks (which can t be replaced) are in the server for training. On Saturday I ran a BTRFS and ZFS training session for the LUV Beginners SIG. This was inspired by the amount of discussion of those filesystems on the mailing list and the amount of interest when we have lectures on those topics. The training went well, the meeting was better attended than most Beginners SIG meetings and the people who attended it seemed to enjoy it. One thing that I will do better in future is clearly documenting commands that are expected to fail and documenting how to login to the system. The users all logged in to accounts on a Xen server and then ssh d to root at their DomU. I think that it would have saved a bit of time if I had aliased commands like btrfs to echo you must login to your virtual server first or made the shell prompt at the Dom0 include instructions to login to the DomU. Each user or group had a virtual machine. The server has 32G of RAM and I ran 14 virtual servers that each had 2G of RAM. In retrospect I should have configured fewer servers and asked people to work in groups, that would allow more RAM for each virtual server and also more RAM for the Dom0. The Dom0 was running a BTRFS RAID-1 filesystem and each virtual machine had a snapshot of the block devices from my master image for the training. Performance was quite good initially as the OS image was shared and fit into cache. But when many users were corrupting and scrubbing filesystems performance became very poor. The disks performed well (sustaining over 100 writes per second) but that s not much when shared between 14 active users. The ZFS part of the tutorial was based on RAID-Z (I didn t use RAID-5/6 in BTRFS because it s not ready to use and didn t use RAID-1 in ZFS because most people want RAID-Z). Each user had 5*4G virtual disks (2 for the OS and 3 for BTRFS and ZFS testing). By the end of the training session there was about 76G of storage used in the filesystem (including the space used by the OS for the Dom0), so each user had something like 5G of unique data. We are now considering what other training we can run on that server. I m thinking of running training on DNS and email. Suggestions for other topics would be appreciated. For training that s not disk intensive we could run many more than 14 virtual machines, 60 or more should be possible. Below are the notes from the BTRFS part of the training, anyone could do this on their own if they substitute 2 empty partitions for /dev/xvdd and /dev/xvde. On a Debian/Jessie system all that you need to do to get ready for this is to install the btrfs-tools package. Note that this does have some risk if you make a typo. An advantage of doing this sort of thing in a virtual machine is that there s no possibility of breaking things that matter.

1. Making the filesystem
   1. Make the filesystem, this makes a filesystem that spans 2 devices (note you must use the-f option if there was already a filesystem on those devices):
      mkfs.btrfs /dev/xvdd /dev/xvde
   2. Use file(1) to see basic data from the superblocks:
      file -s /dev/xvdd /dev/xvde
   3. Mount the filesystem (can mount either block device, the kernel knows they belong together):
      mount /dev/xvdd /mnt/tmp
   4. See a BTRFS df of the filesystem, shows what type of RAID is used:
      btrfs filesystem df /mnt/tmp
   5. See more information about FS device use:
      btrfs filesystem show /mnt/tmp
   6. Balance the filesystem to change it to RAID-1 and verify the change, note that some parts of the filesystem were single and RAID-0 before this change):
      btrfs balance start -dconvert=raid1 -mconvert=raid1 -sconvert=raid1 force /mnt/tmp
      btrfs filesystem df /mnt/tmp
   7. See if there are any errors, shouldn t be any (yet):
      btrfs device stats /mnt/tmp
   8. Copy some files to the filesystem:
      cp -r /usr /mnt/tmp
   9. Check the filesystem for basic consistency (only checks checksums):
      btrfs scrub start -B -d /mnt/tmp
2. Online corruption
   1. Corrupt the filesystem:
      dd if=/dev/zero of=/dev/xvdd bs=1024k count=2000 seek=50
   2. Scrub again, should give a warning about errors:
      btrfs scrub start -B /mnt/tmp
   3. Check error count:
      btrfs device stats /mnt/tmp
   4. Corrupt it again:
      dd if=/dev/zero of=/dev/xvdd bs=1024k count=2000 seek=50
   5. Unmount it:
      umount /mnt/tmp
   6. In another terminal follow the kernel log:
      tail -f /var/log/kern.log
   7. Mount it again and observe it correcting errors on mount:
      mount /dev/xvdd /mnt/tmp
   8. Run a diff, observe kernel error messages and observe that diff reports no file differences:
      diff -ru /usr /mnt/tmp/usr/
   9. Run another scrub, this will probably correct some errors which weren t discovered by diff:
      btrfs scrub start -B -d /mnt/tmp
3. Offline corruption
   1. Umount the filesystem, corrupt the start, then try mounting it again which will fail because the superblocks were wiped:
      umount /mnt/tmp
      dd if=/dev/zero of=/dev/xvdd bs=1024k count=200
      mount /dev/xvdd /mnt/tmp
      mount /dev/xvde /mnt/tmp
   2. Note that the filesystem was not mountable due to a lack of a superblock. It might be possible to recover from this but that s more advanced so we will restore the RAID.
      Mount the filesystem in a degraded RAID mode, this allows full operation.
      mount /dev/xvde /mnt/tmp -o degraded
   3. Add /dev/xvdd back to the RAID:
      btrfs device add /dev/xvdd /mnt/tmp
   4. Show the filesystem devices, observe that xvdd is listed twice, the missing device and the one that was just added:
      btrfs filesystem show /mnt/tmp
   5. Remove the missing device and observe the change:
      btrfs device delete missing /mnt/tmp
      btrfs filesystem show /mnt/tmp
   6. Balance the filesystem, not sure this is necessary but it s good practice to do it when in doubt:
      btrfs balance start /mnt/tmp
   7. Umount and mount it, note that the degraded option is not needed:
      umount /mnt/tmp
      mount /dev/xvdd /mnt/tmp
4. Experiment
   1. Experiment with the btrfs subvolume create and btrfs subvolume delete commands (which act like mkdir and rmdir).
   2. Experiment with btrfs subvolume snapshot SOURCE DEST and btrfs subvolume snapshot -r SOURCE DEST for creating regular and read-only snapshots of other subvolumes (including the root).

A new Armadillo release 5.400.2 was just released by Conrad. Consequently, a new RcppArmadillo release 0.5.400.2.0 went out to CRAN and into Debian. Armadillo is a powerful and expressive C++ template library for linear algebra aiming towards a good balance between speed and ease of use with a syntax deliberately close to a Matlab. This release contains the following extensions:

Changes in RcppArmadillo version 0.5.400.2.0 (2015-08-17)

• Upgraded to Armadillo 5.400.2 ("Plutocracy Incorporated Deluxe")
  • added find_unique() for finding indices of unique values
  • added diff() for calculating differences between consecutive elements
  • added cumprod() for calculating cumulative product
  • added null() for finding the orthonormal basis of null space
  • expanded interp1() to handle repeated locations
  • expanded unique() to handle complex numbers
  • faster flupud()
  • faster row-wise cumsum()
  • fix for k-means clustering in gmm_diag class
• corrected use of kitten() thanks to Grant Brown

Courtesy of CRANberries, there is also a diffstat report for the most recent CRAN release. As always, more detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

<Sigh/> For the second time since I joined Debian, I missed DebConf. This time, mostly for two completely happy reasons (which have, of course, grown up quite significatively): But adding to that very important factor, early-to-mid August is possibly the worst date for me to attend DebConf As that's when we start classes at UNAM, and I really don't want to delegate introducing my class to somebody else (besides it being or not possible). Life has been quite busy here. Besides starting my seventh semester as a teacher (finally being able to present my book on operating systems to the students!), I also took on teaching two modules for a diploma-course on embedded Linux. I'm co- teaching the "User-space Linux" module together with Gabriel Salda a, and will teach the "Boot process" module with Sandino Araico. There are many things that should be improved about the covered program, but we are having good fun working our way on it; Gabriel and I have given two (out of six) 6hr sessions each, and have reached the point where it is actually becoming fun :) As for Debian... I have been doing minimal maintenance work. Answering to some keyring tickets, or taking care of only high-profile important issues. I do not have time to do much. I have been wanting to do some interesting analysis on the evolution of our keyring for several months, but have not had the time to follow it up. Will do at some point, /mehopes. Anyway... This post is probably just because I need to somehow be there at our community's highest social, technical exchange period in the year. That is, there's not much to report, it's only me waving from home!

The Asus t100 tablet is this amazing and odd little thing: it sells for under $200, yet has a full-featured Atom 64-bit CPU, 2GB RAM, 32 or 64GB SSD, etc. By default, it ships with Windows 8.1. It has a detachable keyboard, so it can be used as a tablet or a very small 10 laptop. I have never been a fan of Windows on it. It does the trick for web browsing and email, but I d like to ssh into my machines sometimes, and I just can t bring myself to type sensitive passwords into Windows. I decided to try installing Debian on it. After a lot of abortive starts due to the UEFI-only firmware, I got jessie installed. (The installer was fine; it was Debian Live that wouldn t boot.) I got wifi and battery status working via an upgrade to the 4.1 kernel. A little $10 Edimax USB adapter was handy to spare a bunch of copying via USB disks. I have been using XFCE with XMonad for so many years that I am somewhat a stranger to other desktop environments. XMonad isn t really suitable for a tablet, however, so I thought I d try Gnome, especially after a fairly glowing review about its use on a tablet. I am already disappointed after just a few minutes. There is no suspend button on the menu. Some Googling showed that holding Alt while hovering over the power off button will change it to a suspend button. And indeed it does. But uh, what? That is so common and so non-obvious. And pushing the power button does nothing. That s right, nothing. Apparently the way to enable some action when you push the power button is to type in a settings command in a terminal. There s no setting in the settings panel. I initially ditched Gnome some years ago due to its penchant for removing features. I had hoped that this much time later, it would have passed that stage, but I m already disappointed. I was hoping for some really nice integration with the system. But my XFCE setup has a very clear When power button is pressed setting. I have no idea why Gnome doesn t. Also, the touch screen works fine and it registers my touches, but whenever I touch anywhere, the cursor disappears. Weird, eh? There are some things to fix yet on the tablet (sound, brightness adjustment, and making suspend reliable) but others have solved these in Ubuntu so I don t think it ll be too hard. In the meantime, any suggestions regarding Gnome? Is it just going to annoy me? Maybe I should try KDE also. I ve heard good things about Plasma Active, but don t see it in Debian though.

I wrote a Bash one-liner to open the source code (in Debsources) of any file on your system (if it belongs to a Debian package). It will simply retrieve the associated package and point your default browser to its source code. Add this somewhere in your $PATH, and name this file debsrc: And try something like debsrc /usr/share/doc/acpi/AUTHORS. Enjoy! Update: improved the one-liner thanks to josch's advice.

As with all the previous versions, I have prepared a mega update for the Kobo firmware 3.17.0. Big warning for a start this is for Mark6 hardware, thus GloHD only. The update includes all my favorite patches and features. Please see details (although for for Kobo Glo) for 3.16.0, 3.15.0, 3.13.1 and 3.12.1. As before, the following addons are included: Kobo Start Menu, koreader, coolreader, pbchess, ssh access, custom dictionaries, and some side-loaded fonts. What I dropped this time is most of the kobohack part, as the libraries seem to get outdated. But I included the dropbear ssh server from the kobohack package. So what are all these items:

• firmware (thread): the basic software of the device, shipped by Kobo company
• Metazoa firmware patches (thread): fix some layout options and functionalities, see below for details.
• Kobo Start Menu (V07, update 4 thread): a menu that pops up before the reading software (nickel) starts, which allows to start alternative readers (like koreader) etc.
• KOreader (koreader-nightly-20150808, thread): an alternative document reader that supports epub, azw, pdf, djvu and many more
• pbchess and CoolReader (2015.7, thread): a chess program and another alternative reader, bundled together with several other games
• kobohack (web site): I only use the ssh server
• ssh access (old post: makes a full computer from your device by allowing you to log into it via ssh
• custom dictionaries (thread): this fix updates dictionaries from the folder customdicts to the Kobo dictionary folder. For creating your own Japanese-English dictionary, see this blog entry
• side-loaded fonts: GentiumBasic and GentiumBookBasic, Verdana, DroidSerif, and Charter-eInk

Install procedure Latest firmware Warning: Sideloading or crossloading the incorrect firmware can break/brick your device. The link below is for Kobo GloHD ONLY. The first step is to update the Kobo to the latest firmware. This can easily be done by just getting the latest firmware (thread, direct link for Kobo 3.17.0 for GloHD) and unpacking the zip file into the .kobo directory on your device. Eject and enjoy the updating procedure. Mega update Get the combined Kobo-3.17.0-combined/KoboRoot.tgz and put it into the .kobo directory, then eject and enjoy the updating procedure again. After this the device should reboot and you will be kicked into KSM, from where after some time of waiting Nickel will be started. If you consider the fonts too small, select Configure, then the General, and add item, then select kobomenuFontsize=55 and save. Remarks to some of the items included The full list of included things is above, here are only some notes about what specific I have done.

• Metazoa firmware patches I have included the experimental sickel patch to extend the waiting time until sickel reboots your device. The patches I have enabled in this build are:
  • libadobe.so.patch: none
  • librmsdk.so.1.0.0.patch: default ( Fix page breaks bug , Default ePub monospace font (Courier) )
  • libnickel.so.1.0.0.patch: Custom reading footer style , My 15 line spacing values , Custom left & right margins , Brightness fine control , Search in Library by default , Disable pinch-to-zoom font resizing
  • sickel.patch: default ( SickelService::Ping() timer , SickelService::Resume() timer , SickelService::SickelService() timer )
  If you need any other patches, you need to do the patching yourself.
• kobohack-h Kobohack (latest version 20150110) originally provided updated libraries and optimizations, but unfortunately it is now completely outdated and using it is not recommended for the library part. I only include the ssh server (dropbear) so that connections to the Kobo via ssh.
• ssh fixes See the detailed instructions here, the necessary files are already included in the mega upload. It updates the /etc/inittab to run also /etc/init.d/rcS2, and this one again starts the inetd server and run user supplied commands in /mnt/onboard/run.sh which is where your documents are.
• Custom dictionaries The necessary directories and scripts are already included in the above combined KoboRoot.tgz, so nothing to be done but dropping updated, fixed, changed dictionaries into your Kobo root into the directory customdict. After this you need to reboot to get the actual dictionaries updated. See this thread for more information. The adaptions and script mentioned in this post are included in the mega update.

WARNINGS After installing this patch, you need to fix the password for root and disable telnet. This is an important step, here are the steps you have to take (taken from this old post):

1. Turn on Wifi on the Kobo and find IP address
   Go to Settings Connect and after this is done, go to Settings Device Information where you will see something like
IP Address: 192.168.1.NN
   (numbers change!)
2. telnet into your device
   telnet 192.168.1.NN
   it will ask you the user name, enter root (without the quotes) and no password
3. (ON THE GLO) change home directory of root
   edit /etc/passwd with vi and change the entry for root by changing the 6th field from: / to /root (without the quotes). After this procedure the line should look like
   root::0:0:root:/root:/bin/sh
   don t forget to save the file
4. (ON THE GLO) create ssh keys for dropbear
   [root@(none) ~]# mkdir /etc/dropbear
[root@(none) ~]# cd /etc/dropbear
[root@(none) ~]# dropbearkey -t dss -f dropbear_dss_host_key
[root@(none) ~]# dropbearkey -t rsa -f dropbear_rsa_host_key
5. (ON YOUR PERSONAL COMPUTER) check that you can log in with ssh
   ssh root@192.168.1.NN
   You should get dropped into your device again
6. (ON THE GLO) log out of the telnet session (the first one you did)
   [root@(none) ~]# exit
7. (ON THE GLO) in your ssh session, change the password of root
   [root@(none) ~]# passwd
   you will have to enter the new password two times. Remember it well, you will not be easily able to recover it without opening your device.
8. (ON THE GLO) disable telnet login
   edit the file /etc/inetd.conf.local on the GLO (using vi) and remove the telnet line (the line starting with 23).
9. restart your device

The combined KoboRoot.tgz is provided without warranty. If you need to reset your device, don t blame me!

Transcoding video from one format to another seems to be a bit of a black art. There are many tools that allow doing this kind of stuff, but one issue that most seem to have is that they're not very well documented. I ran against this a few years ago, when I was first doing video work for FOSDEM and did not yet have proper tools to do the review and transcoding workflow. At the time, I just used mplayer to look at the .dv files, and wrote a text file with a simple structure to remember exactly what to do with it. That file was then fed to a perl script which wrote out a shell script that would use the avconv command to combine and extract the "interesting" data from the source DV files into a single DV file per talk, and which would then call a shell script which used gst-launch and sox to do a multi-pass transcode of those intermediate DV files into a WebM file. While all that worked properly, it was a rather ugly hack, never cleaned up, and therefore I never really documented it properly either. Recently, however, someone asked me to do so anyway, so here goes. Before you want to complain about how this ate the videos of your firstborn child, however, note the above. The perl script spent a somewhat large amount of code reading out the text file and parsing it into an array of hashes. I'm not going to reproduce that, since the actual format of the file isn't all that important anyway. However, here's the interesting bits:

foreach my $pfile(keys %parts)  
        my @files = @ $parts $pfile ;
        say "#" x (length($pfile) + 4);
        say "# " . $pfile . " #";
        say "#" x (length($pfile) + 4);
        foreach my $file(@files)  
                my $start = "";
                my $stop = "";
                if(defined($file-> start ))  
                        $start = "-ss " . $file-> start ;
                 
                if(defined($file-> stop ))  
                        $stop = "-t " . $file-> stop ;
                 
                if(defined($file-> start ) && defined($file-> stop ))  
                        my @itime = split /:/, $file-> start ;
                        my @otime = split /:/, $file-> stop ;
                        $otime[0]-=$itime[0];
                        $otime[1]-=$itime[1];
                        if($otime[1]<0)  
                                $otime[0]-=1;
                                $otime[1]+=60;
                         
                        $otime[2]-=$itime[2];
                        if($otime[2]<0)  
                                $otime[1]-=1;
                                $otime[2]+=60;
                         
                        $stop = "-t " . $otime[0] . ":" . $otime[1] .  ":" . $otime[2];
                 
                if(defined($file-> start )   defined($file-> stop ))  
                        say "ln " . $file-> name  . ".dv part-pre.dv";
                        say "avconv -i part-pre.dv $start $stop -y -acodec copy -vcodec copy part.dv";
                        say "rm -f part-pre.dv";
                  else  
                        say "ln " . $file-> name  . ".dv part.dv";
                 
                say "cat part.dv >> /tmp/" . $pfile . ".dv";
                say "rm -f part.dv";
         
        say "dv2webm /tmp/" . $pfile . ".dv";
        say "rm -f /tmp/" . $pfile . ".dv";
        say "scp /tmp/" . $pfile . ".webm video.fosdem.org:$uploadpath   true";
        say "mv /tmp/" . $pfile . ".webm .";
 

That script uses avconv to read one or more .dv files and transcode them into a single .dv file with all the start- or end-junk removed. It uses /tmp rather than the working directory, since the working directory was somewhere on the network, and if you're going to write several gigabytes of data to an intermediate file, it's usually a good idea to write them to a local filesystem rather than to a networked one. Pretty boring. It finally calls dv2webm on the resulting .dv file. That script looks like this:

#!/bin/bash
set -e
newfile=$(basename $1 .dv).webm
wavfile=$(basename $1 .dv).wav
wavfile=$(readlink -f $wavfile)
normalfile=$(basename $1 .dv)-normal.wav
normalfile=$(readlink -f $normalfile)
oldfile=$(readlink -f $1)
echo -e "\033]0;Pass 1: $newfile\007"
gst-launch-0.10 webmmux name=mux ! fakesink \
  uridecodebin uri=file://$oldfile name=demux \
  demux. ! ffmpegcolorspace ! deinterlace ! vp8enc multipass-cache-file=/tmp/vp8-multipass multipass-mode=1 threads=2 ! queue ! mux.video_0 \
  demux. ! progressreport ! audioconvert ! audiorate ! tee name=t ! queue ! vorbisenc ! queue ! mux.audio_0 \
  t. ! queue ! wavenc ! filesink location=$wavfile
echo -e "\033]0;Audio normalize: $newfile\007"
sox --norm $wavfile $normalfile
echo -e "\033]0;Pass 2: $newfile\007"
gst-launch-0.10 webmmux name=mux ! filesink location=$newfile \
  uridecodebin uri=file://$oldfile name=video \
  uridecodebin uri=file://$normalfile name=audio \
  video. ! ffmpegcolorspace ! deinterlace ! vp8enc multipass-cache-file=/tmp/vp8-multipass multipass-mode=2 threads=2 ! queue ! mux.video_0 \
  audio. ! progressreport ! audioconvert ! audiorate ! vorbisenc ! queue ! mux.audio_0
rm $wavfile $normalfile

... and is a bit more involved. Multi-pass encoding of video means that we ask the encoder to first encode the file but store some statistics into a temporary file (/tmp/vp8-multipass, in our script), which the second pass can then reuse to optimize the transcoding. Since DV uses different ways of encoding things than does VP8, we also need to do a color space conversion (ffmpegcolorspace) and deinterlacing (deinterlace), but beyond that the video line in the first gstreamer pipeline isn't very complicated. Since we're going over the file anyway and we need the audio data for sox, we add a tee plugin at an appropriate place in the audio line in the first gstreamer pipeline, so that we can later on pick up that same audio data an write it to a wav file containing linear PCM data. Beyond the tee, we go on and do a vorbis encoding, as is needed for the WebM format. This is not actually required for a first pass, but ah well. There's some more conversion plugins in the pipeline (specifically, audioconvert and audiorate), but those are not very important. We next run sox --norm on the .wav file, which does a fully automated audio normalisation on the input. Audio normalisation is the process of adjusting volume levels so that the audio is not too loud, but also not too quiet. Sox has pretty good support for this; the default settings of its --norm parameter make it adjust the volume levels so that the highest peak will just about reach the highest value that the output format can express. As such, you have no clipping anywhere in the file, but also have an audio level that is actually useful. Next, we run a second-pass encoding on the input file. This second pass uses the statistics gathered in the first pass to decide where to put its I- and P-frames so that they are placed at the most optimal position. In addition, rather than reading the audio from the original file, we now read the audio from the .wav file containing the normalized audio which we produced with sox, ensuring the audio can be understood. Finally, we remove the intermediate audio files we created; and the shell script which was generated by perl also contained an rm command for the intermediate .dv file. Some of this is pretty horrid, and I never managed to clean it up enough so it would be pretty (and now is not really the time). However, it Just Works(tm), and I am happy to report that it continues to work with gstreamer 1.0, provided you replace the ffmpegcolorspace by an equally simple videoconvert, which performs what ffmpegcolorspace used to perform in gstreamer 0.10.