• Bugs in/Updates to auto-generated snippets require a rebuild of all packages (not to mention the obvious code-duplication in all packages).
• In case of circular dependencies[1] all having postinst scripts, dpkg will have to guess which package to configure first.
• They require forking a shell at least once for each maintscript.
• They complicate the implementations of e.g. detached chroot creation.

• no-ctrl-scripts in 2.5.44
• ctrl-script in 2.5.44
• debhelper-autoscript-in-maintainer-scripts in 2.5.45

1. dhpython with 3775 instances
   • This is an umbrella for all helpers using dh-python s python module, see #827774.
2. dh_installmenu with 1861 instances
   • This will presumably drop as menu files should be replaced with desktop files.
3. dh_makeshlibs with 1396 remaining instances
   • This should be in rapid decline since we decided to replace its scripts with a dpkg trigger. Unfortunately, it can take quite a while until all of them have been rebuilt with a new version of debhelper
4. dh_installinit with 1224 instances
5. dh_python2 with 1168 instances
6. dh_installdebconf with 772 instances
7. dh_installdeb with 754 instances
   • These are the dpkg-maintscript-helper snippets for rm_conffile , mv_conffile etc. Hopefully in the near future, dpkg will support these directly.
8. dh_systemd_enable with 447 instances
9. dh_installemacsen with 179 instances
10. dh_icons with 165 instances
11. dh_installtex with 137 instances
12. dh_apache2 with 117 instances
13. dh_installudev with 98 instances
14. dh_installxfonts with 87 instances
15. dh_systemd_start with 79 instances

• Replacing the python scripts (I presume it is the byte-code handling, but I have not looked at this at all)
• Migrating away from menu files
• Support enabling + starting/stopping/restarting a service declaratively.
  • This might have a hidden requirement on declaratively creating service users if we want these packages to become truly maintscript-less .

• I researched different diffoscope outputs. In reproducible-builds testing framework only differences of .buildinfo files are given but I needed diffoscope outputs for .changes files. Hence, I had to build packages locally using our experimental tool chain. My goal was to generate different outputs and to see how I can hide .buildinfo files from .changes.
• I updated argument completion patch as per suggestions given by Paul Wise (pabs). Patch has been reviewed by Mattia Rizzolo, Holger Levsen and merged by Reiner Herrmann (deki) into diffoscope master. This patch closes #826711. Thanks all for support.
• For Ignore .buildinfo files when comparing .changes files, we finally decided to enable this by default and without having any command line option to hide.
• Last week I researched more on .changes and .buildinfo files. After getting guidelines from Lunar I was able to understand the need of this feature. I am in the middle of implementation of this particular problem.

• Finish the implementation of hiding .buildinfo from .changes
• Start thinking on interfaces and discuss about different use cases.

• Resize the image to fit Facebook's preferred 960 pixel box.
• Insert Copyright, Byline and Bylinetitle into the EXIF data.
• Optionally, paste my watermark onto a predefined corner of the image.
• Optionally, adjust the white balance.
• Rename the file according to a specific syntax.
• Save the result to a predefined folder.

(It is a gross oversimplification, but for the purposes of OpenStack CI, Jenkins is pretty much used as a glorified ssh/scp wrapper. Zuul Version 3, under development, is working to remove the need for Jenkins to be involved at all).

OpenStack CI Overview While previous post was really focused on the image-building components of the OpenStack CI system, overview is the same but more focused on the launchers that run the tests.

1. The process starts when a developer uploads their code to gerrit via the git-review tool. There is no further action required on their behalf and the developer simply waits for results of their jobs.

2. Gerrit provides a JSON-encoded "fire-hose" output of everything happening to it. New reviews, votes, updates and more all get sent out over this pipe. Zuul is the overall scheduler that subscribes itself to this information and is responsible for managing the CI jobs appropriate for each change.

3. Zuul has a configuration that tells it what jobs to run for what projects. Zuul can do lots of interesting things, but for the purposes of this discussion we just consider that it puts the jobs it wants run into gearman for a launcher to consume. gearman is a job-server; as they explain it "[gearman] provides a generic application framework to farm out work to other machines or processes that are better suited to do the work". Zuul puts into gearman basically a tuple (job-name, node-type) for each job it wants run, specifying the unique job name to run and what type of node it should be run on.

4. A group of Zuul launchers are subscribed to gearman as workers. It is these Zuul launchers that will consume the job requests from the queue and actually get the tests running. However, a launcher needs two things to be able to run a job a job definition (what to actually do) and a worker node (somewhere to do it). The first part what to do is provided by job-definitions stored in external YAML files. The Zuul launcher knows how to process these files (with some help from Jenkins Job Builder, which despite the name is not outputting XML files for Jenkins to consume, but is being used to help parse templates and macros within the generically defined job definitions). Each Zuul launcher gets these definitions pushed to it constantly by Puppet, thus each launcher knows about all the jobs it can run automatically. Of course Zuul also knows about these same job definitions; this is the job-name part of the tuple we said it put into gearman. The second part somewhere to run the test takes some more explaining. To the next point...

5. Several cloud companies donate capacity in their clouds for OpenStack to run CI tests. Overall, this capacity is managed by a customized management tool called nodepool (you can see the details of this capacity at any given time by checking the nodepool configuration). Nodepool watches the gearman queue and sees what requests are coming out of Zuul. It looks at node-type of jobs in the queue (i.e. what platform the job has requested to run on) and decides what types of nodes need to start and which cloud providers have capacity to satisfy demand. Nodepool will start fresh virtual machines (from images built daily as described in the prior post), monitor their start-up and, when they're ready, put a new "assignment job" back into gearman with the details of the fresh node. One of the active Zuul launchers will pick up this assignment job and register the new node to itself.

6. At this point, the Zuul launcher has what it needs to actually get jobs started. With an fresh node registered to it and waiting for something to do, the Zuul launcher can advertise its ability to consume one of the waiting jobs from the gearman queue. For example, if a ubuntu-trusty node is provided to the Zuul launcher, the launcher can now consume from gearman any job it knows about that is intended to run on an ubuntu-trusty node type. If you're looking at the launcher code this is driven by the NodeWorker class you can see this being created in response to an assignment via LaunchServer.assignNode. To actually run the job where the "job hits the metal" as it were the Zuul launcher will dynamically construct an Ansible playbook to run. This playbook is a concatenation of common setup and teardown operations along with the actual test scripts the jobs wants to run. Using Ansible to run the job means all the flexibility an orchestration tool provides is now available to the launcher. For example, there is a custom console streamer library that allows us to live-stream the console output for the job over a plain TCP connection, and there is the possibility to use projects like ARA for visualisation of CI runs. In the future, Ansible will allow for better coordination when running multiple-node testing jobs after all, this is what orchestration tools such as Ansible are made for! While the Ansible run can be fairly heavyweight (especially when you're talking about launching thousands of jobs an hour), the system scales horizontally with more launchers able to consume more work easily. When checking your job results on logs.openstack.org you will see a _zuul_ansible directory now which contains copies of the inventory, playbooks and other related files that the launcher used to do the test run.

7. Eventually, the test will finish. The Zuul launcher will put the result back into gearman, which Zuul will consume (log copying is interesting but a topic for another day). The testing node will be released back to nodepool, which destroys it and starts all over again nodes are not reused and also have no sensitive details on them, as they are essentially publicly accessible. Zuul will wait for the results of all jobs for the change and post the result back to Gerrit; it either gives a positive vote or the dreaded negative vote if required jobs failed (it also handles merges to git, but that is also a topic for another day).

Work will continue within OpenStack Infrastructure to further enhance Zuul; including better support for multi-node jobs and "in-project" job definitions (similar to the https://travis-ci.org/ model); for full details see the spec.

#jinja2: lstrip_blocks: True

global
 % for setting in global_settings % 
     % if setting ... % 
    option   setting  
     % endif % 
 % endfor % 

Server setup The first thing I did on the server-side is to follow Sean Whitton's advice and configure postfix so that it keeps undelivered emails for 10 days (instead of 5 days, the default):

postconf -e maximal_queue_lifetime=10d

Then I created a new user:

adduser mailq-check

with a password straight out of pwgen -s 32. I gave ssh permission to that user:

adduser mailq-check sshuser

and then authorized my new ssh key (see next section):

sudo -u mailq-check -i
mkdir ~/.ssh/
cat - > ~/.ssh/authorized_keys

Laptop setup On my laptop, the machine from where I monitor the server's mail queue, I first created a new password-less ssh key:

ssh-keygen -t ed25519 -f .ssh/egilsstadir-mailq-check
cat ~/.ssh/egilsstadir-mailq-check.pub

which I then installed on the server. Then I added this cronjob in /etc/cron.d/egilsstadir-mailq-check:

0 2 * * * francois /usr/bin/ssh -i /home/francois/.ssh/egilsstadir-mailq-check mailq-check@egilsstadir mailq   grep -v "Mail queue is empty"

and that's it. I get a (locally delivered) email whenever the mail queue on the server is non-empty. There is a race condition built into this setup since it's possible that the server will want to send an email at 2am. However, all that does is send a spurious warning email in that case and so it's a pretty small price to pay for a dirt simple setup that's unlikely to break.

No comment Liked this article? Click here. My blog is Flattr-enabled.

Analyzing data input In the past, I have used Workrave to document how I work too much on the computer, but never went through more serious processing of the vast data store that Workrave accumulates about mouse movements and keystrokes. Interested in knowing how much my leave from Koumbit affected time spent on the computer, I decided to look into this again. It turns out I am working as much, if not more, on the computer since I took that "time off": We can see here that I type a lot on the computer. Normal days range from 10 000 to 60 000 keystrokes, with extremes at around 100 000 keystrokes per day. The average seem to fluctuate around 30 to 40 000 keystrokes per day, but rises sharply around the end of the second quarter of this very year. For those unfamiliar with the underlying technology, one keystroke is roughly one byte I put on the computer. So the average of 40 000 keystrokes is 40 kilobyte (KB) per day on the computer. That means 15 MB over a year or about 150MB (or 100 MiB if you want to be picky about it) over the course of the last decade. That is a lot of typing. I originally thought this could have been only because I type more now, as opposed to use more the mouse previously. Unfortunately, Workrave also tracks general "active time" which we can also examine: Here we see that I work around 4 hours a day continuously on the computer. That is active time: not just login, logout time. In other words, the time where i look away from the computer and think for a while, jot down notes in my paper agenda or otherwise step away from the computer for small breaks is not counted here. Notice how some days go up to 12 hours and how recently the average went up to 7 hours of continuous activity. So we can clearly see that I basically work more on the computer now than I ever did in the last 7 years. This is a problem - one of the reasons of this time off was to step away from the computer, and it seems I have failed.

Update: it turns out the graph was skewed towards the last samples. I went more easy on the keyboard in the last few days and things have significantly improved:

Another interesting thing we can see is when I switched from using my laptop to using the server as my main workstation, around early 2011, which is about the time marcos was built. Now that marcos has been turned into a home cinema downstairs, I went back to using my laptop as my main computing device, in late 2015. We can also clearly see when I stopped using Koumbit machines near the end of 2015 as well.

Further improvements and struggle for meaning The details of how the graph was produced are explained at the end of this article. This is all quite clunky: it doesn't help that the Workrave data structure is not easily parsable and so easily corruptible. It would be best if each data point was on its own separate line, which would be long, granted, but so easier to parse. Furthermore, I do not like the perl/awk/gnuplot data processing pipeline much. It doesn't allow me to do interesting analysis like averages, means and linear regressions easily. It could be interesting to rewrite the tools in Python to allow better graphs and easier data analysis, using the tools I learned in 2015-09-28-fun-with-batteries. Finally, this looks only at keystrokes and non-idle activity. It could be more interesting to look at idle/active times and generally the amount of time spent on the computer each day. And while it is interesting to know that I basically write a small book on the computer every day (according to Wikipedia, 120KB is about the size of a small pocket book), it is mostly meaningless if all that stuff is machine-readable code. Where is, after all, the meaning in all those shell commands and programs we constantly input on our keyboards, in the grand scheme of human existence? Most of those bytes are bound to be destroyed by garbage collection (my shell's history) or catastrophic backup failures.

While the creative works from the 16th century can still be accessed and used by others, the data in some software programs from the 1990s is already inaccessible. - Lawrence Lessig

But is my shell history relevant? Looking back at old posts on this blog, one has to wonder if the battery life of the Thinkpad 380z laptop or how much e-waste I threw away in 2005 will be of any historical value in 20 years, assuming the data survives that long.

How this graph was made I was happy to find that Workrave has some contrib scripts to do such processing. Unfortunately, those scripts are not shipped with the Debian package, so I requested that to be fixed (#825982). There were also some fixes necessary to make the script work at all: first, there was a syntax error in the Perl script. But then since my data is so old, there was bound to be some data corruption in there: incomplete entries or just plain broken data. I had lines that were all NULL characters, typical of power failures or disk corruptions. So I have made a patch to fix that script (#826021). But this wasn't enough: while this processes data on the current machine fine, it doesn't deal with multiple machines very well. In the last 7 years of data I could find, I was using 3 different machines: this server (marcos), my laptop (angela) and Koumbit's office servers (koumbit). I ended up modifying the contrib scripts to be able to collate that data meaningfully. First, I copied over the data from Koumbit in a local fake-koumbit directory. Second, I mounted marcos home directory locally with SSHFS:

sshfs anarc.at:/home/anarcat marcos

I also made this script to sum up datasets:

#!/usr/bin/perl -w
use List::MoreUtils 'pairwise';
$  = 1;
my %data = ();
while (<>)  
    my @fields = split;
    my $date = shift @fields;
    if (defined($data $date ))  
        my @t = pairwise   $a + $b   @ $data $date , @fields;
        $data $date  = \@t;
     
    else  
        $data $date  = \@fields;
     
 
foreach my $d ( sort keys %data )  
    print "$d @ $data $d \n";
 

Then I made a modified version of the Gnuplot script that processes all those files together:

#!/usr/bin/gnuplot
set title "Workrave"
set ylabel "Keystrokes per day"
set timefmt "%Y%m%d%H%M"
#set xrange [450000000:*]
set format x "%Y-%m-%d"
set xtics rotate
set xdata time
set terminal svg
set output "workrave.svg"
plot "workrave-angela.dat" using 1:28 title "angela", \
     "workrave-marcos.dat" using 1:28 title "marcos", \
     "workrave-koumbit.dat" using 1:28 title "koumbit", \
     "workrave-sum.dat" using 1:2 smooth sbezier linewidth 3 title "average"
#plot "workrave-angela.dat" using 1:28 smooth sbezier title "angela", \
#     "workrave-marcos.dat" using 1:28 smooth sbezier title "marcos", \
#     "workrave-koumbit.dat" using 1:28 smooth sbezier title "koumbit"

And finally, I made a small shell script to glue this all together:

#!/bin/sh
perl workrave-dump > workrave-$(hostname).dat
HOME=$HOME/marcos perl workrave-dump > workrave-marcos.dat
HOME=$PWD/fake-koumbit perl workrave-dump > workrave-koumbit.dat
# remove idle days as they skew the average
sed -i '/ 0$/d' workrave-*.dat
# per-day granularity
sed -i 's/^\(........\)....\? /\1 /' workrave-*.dat
# sum up all graphs
cat workrave-*.dat   sort   perl sum.pl > workrave.dat
./gnuplot-workrave-anarcat

I used a different gnuplot script to generate the activity graph:

#!/usr/bin/gnuplot
set title "Workrave"
set ylabel "Active hours per day"
set timefmt "%Y%m%d%H%M"
#set xrange [450000000:*]
set format x "%Y-%m-%d"
set xtics rotate
set xdata time
set terminal svg
set output "workrave.svg"
plot "workrave-angela.dat"  using 1:($23/3600) title "angela", \
     "workrave-marcos.dat"  using 1:($23/3600) title "marcos", \
     "workrave-koumbit.dat" using 1:($23/3600) title "koumbit", \
     "workrave.dat"         using 1:($23/3600) title "average" smooth sbezier linewidth 3
#plot "workrave-angela.dat" using 1:28 smooth sbezier title "angela", \
#     "workrave-marcos.dat" using 1:28 smooth sbezier title "marcos", \
#     "workrave-koumbit.dat" using 1:28 smooth sbezier title "koumbit"

Reinstallation seems to be only option left here, but there are a few checks you can make

1. Call the terminal. Use Ctrl+Alt+F2, you should be able to get your command line.
2. Login as root. This is extreme troubleshooting. You need root
3. Check and see if startx works. (Yay! problem solved if it does!)
4. Try to see if the basic gnome packages are installed type apt-get install with basic gnome packages like gnome-desktop-environment, gdm, gnome, etc.. to see if they are installed. If not, try to install them and then run the startx command.
5. If you can t install them, or if they are already installed, try running the updating the system
   
apt-get update
apt-get upgrade
apt-get dist-upgrade //If OS is at newest version

6. If updating doesn t work and gives Failed to Fetch errors for all the source links, try pinging the network to make sure you have a connection.
   
ping google.com
ping 8.8.8.8

7. If you do have a connection, verify the sources :
   
nano /etc/apt/sources.listdeb http://ftp.us.debian.org/debian jessie main contrib non-free
   #deb-src http://ftp.us.debian.org/debian jessie main contrib non-free
   deb http://security.debian.org/ jessie/updates main contrib non-free
   #deb-src http://security.debian.org/ jessie/updates main contrib non-free
   deb http://ftp.us.debian.org/debian testing main contrib non-free
   #deb-src http://ftp.us.debian.org/debian testing main contrib non-free
   deb http://security.debian.org/ testing/updates main contrib non-free
   #deb-src http://security.debian.org/ testing/updates main contrib non-free
8. If you have no connection, check your routing table with the command netstat -nr . If this routing table is completely empty, then I ve got some bad news for you. Reinstallation is your new friend. Try and see if you can copy your files using command line. cp and ls are basically your new best friends.

• golang-github-hpcloud-tail/1.0.0+git20160415.b294095-3: put versioned dependency & rebuild against golang-fsnotify/1.3.0-3 to fix FTBFS on ppc64el.
• updates: packer/0.10.1-1, pybtex/0.20.1-1, afl/2.12b-1, afl/2.13b-1, pyutilib/5.3.5-1.
• new packages: golang-github-azure-go-ntlmssp/0.0~git20160412.e0b63eb-1 (needed by Packer 0.10.1), and python-latexcodec/1.0.3-1 (needed by Pybtex 0.20).
• prospector/0.11.7-7 fixed for reproducible builds: there were variations in the sorting order of dependencies in prospector.egg-info/requires.txt. I've prepared a patch to make the package reproducible again (that problem began with 0.11.7-5) before the proposed toolchain patch for setuptools (#804249) gets accepted.
• python-latexcodec/1.0.3-3 also fixed for reproducible builds (#824454).

$ pyinfra -i <inventory script/single host> <deploy script>
$ pyinfra -i <inventory script/single host> --run <operation>
$ pyinfra -i <inventory script/single host> --facts <fact>

$ pyinfra -i 192.0.2.10 --run server.user sam --user root --key ~/.ssh/sshkey --key-password 123456 -v

COMPUTE_SERVERS = ['192.0.2.10', '192.0.2.11']
DATABASE_SERVERS = ['192.0.2.20', '192.0.2.21']

$ pyinfra -i inventory/farm1.py  --run server.user sam --user root --key ~/.ssh/sshkey --key-password=123456 -v

user_name = 'sam'
ssh_user = 'root'
ssh_key = '~/.ssh/sshkey'
ssh_key_password = '123456'

from pyinfra import host
from pyinfra.modules import server
server.user(host.data.user_name)

$ pyinfra -i inventory/farm1.py deploy.py

$ pyinfra -i 192.0.2.10 --fact deb_packages --user root --key ~/.ssh/sshkey --key-password=123456
 
    "192.0.2.10":  
        "libdebconfclient0": "0.192",
        "python-debian": "0.1.27",
        "libavahi-client3": "0.6.31-5",
        "dbus": "1.8.20-0+deb8u1",
        "libustr-1.0-1": "1.0.4-3+b2",
        "sed": "4.2.2-4+b1",

from pyinfra import host
from pyinfra.modules import apt
if host.fact.linux_distribution['name'] == 'Debian':
   apt.packages(packages='gummi', present=True, update=True)
elif host.fact.linux_distribution['name'] == 'CentOS':
   pass

• scripts for creating proper HDF5 training data of the MNIST database of handwritten digits and for training a simple neural network on it (create_mnist.py, mnist_pi.py),
• examples for setting up data and training a convolutional neural network (CNN) on the CIFAR-10 dataset of pictures (create_cifa10.py, cifar10_cnn.py),
• as well as example scripts for setting up training data and creating a LSTM (Long short-term memory) recurrent neural network (RNN) on test data used in the Hutter Prize competition (create_hutter.py, hutter_lstm.py).
• And there's also another example script for creating training data of the CIFAR-100 dataset (create_cifar100.py).

• Caffe for image recognition resp. classification⁷ is just around the corner (#823140).
• Theano is currently in experimental, and will be ready together with libgpuarray (OpenCL based GPU accelerated processing) and Keras (abstraction layer) for Stretch. It could already run on NVIDIA graphics card via CUDA⁸ (limited to amd64 and ppc64el, though).
• Lasagne, another abstraction layer for Theano is RFP (#818641).
• Google's Tensorflow, the free successor of Dist-Belief, is currently on ITP (#804612). It's waiting for Google's build system Bazel to become available.
• Torch is also ITP (#794634). It's blocked by a wishlist bug on dh-lua to get closed.
• Amazon's own machine learning workhorse dsstne ("destiny") is now also put under a free license and also will becoming available (#824692) in the near future for Debian (contrib). It's not yet for image recognition applications, though (lacks CNN).
• Mxnet is RFP (#808235).