<?xml version="1.0" encoding="UTF-8"?>
<mime-info xmlns="http://www.freedesktop.org/standards/shared-mime-info">
  <mime-type type="audio/x-rosegarden">
    <sub-class-of type="application/x-gzip"/>
    <comment>Rosegarden project file</comment>
    <glob pattern="*.rg"/>
  </mime-type>
</mime-info>

% grep Mime /usr/share/applications/rosegarden.desktop
MimeType=audio/x-rosegarden-composition;audio/x-rosegarden-device;audio/x-rosegarden-project;audio/x-rosegarden-template;audio/midi;
X-KDE-NativeMimeType=audio/x-rosegarden-composition
%

• Implement end-to-end encryption
• Receive messages the moment they are sent without draining the battery

• Create an entirely new address book by entering addresses for your friends that you don't already have
• Use a new and different app for sending encrypted messages

• Our app could send both XMPP messages and SMS messages
• Everytime you added a new XMPP contact, it added the contact to your address book with a new XMPP field
• Anytime you send a message to a contact with an XMPP field filled in, it would send via XMPP and otherwise it would send a normal SMS message

per_account:set sendmail  = "$HOME/bin/check-mail-gpg /usr/bin/msmtp -a $my_email"

#! /usr/bin/python3
import os
import sys
import subprocess
import email.parser
from email.header import decode_header
from email.utils import parseaddr
STATUS_FILE = '/tmp/check-mail-gpg.tmp'
def dec(header):
    head = decode_header(header)
    if len(head) == 1 and head[0][1] == None:
        return head[0][0]
    else:
        return ''.join([h.decode(enc) if enc else h.decode('ascii') \
                for (h,enc) in head])
def send_mail(mail):
    print('Calling external email client to send the email...')
    #return -1 # testing mode
    p = subprocess.Popen(sys.argv[1:], stdin=subprocess.PIPE)
    p.stdin.write(mail.encode('utf-8'))
    p.stdin.close()
    return p.wait()
def main():
    mail = sys.stdin.read()
    heads = email.parser.Parser().parsestr(mail, headersonly=True)
    content = heads['Content-Type'].split(';')[0].strip()
    print('Content is:', content)
    if content == 'multipart/encrypted':
        print('Ok: encrypted mail, we can return now...')
        sys.exit(send_mail(mail))
    addrs = [parseaddr(addr) for addr in heads['To'].split(',')]
    print('Found emails:', addrs)
    gpg_cnt = 0
    for name, addr in addrs:
        print('Looking for', addr, 'in the keyring...')
        res = subprocess.call(['gpg', '--list-keys', addr],
                stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
        if res == 0:
            gpg_cnt += 1
    if gpg_cnt == 0:
        print('Ok: no email found in the gpg keyring, we can return now...')
        sys.exit(send_mail(mail))
    if not os.path.exists(STATUS_FILE):
        open(STATUS_FILE, 'w').close()
    msg_id = heads['Message-ID']
    msg_id_prev = ''
    with open(STATUS_FILE, 'r') as f:
        msg_id_prev = f.read()
    if msg_id.strip() == msg_id_prev.strip():
        sys.exit(send_mail(mail))
    else:
        with open(STATUS_FILE, 'w') as f:
            f.write(msg_id)
        print('Alert: trying to send an unencrypted email to', addrs,
                ', for which some gpg keys were found in the keyring!')
        print('Try again if you are sure to send this message unencrypted.')
        sys.exit(1)
if __name__ == '__main__':
    main()

Update I ve been told about the option crypt_opportunistic_encrypt in mutt, which provides a feature very similar to what I was looking for. This option will automatically enable encryption when the recipient has a GPG key in your keyring. From mutt s man page:

3.41. crypt_opportunistic_encrypt Type: boolean Default: no Setting this variable will cause Mutt to automatically enable and disable encryption, based on whether all message recipient keys can be located by mutt. When this option is enabled, mutt will determine the encryption setting each time the TO, CC, and BCC lists are edited. If $edit_headers is set, mutt will also do so each time the message is edited. While this is set, encryption settings can t be manually changed. The pgp or smime menus provide an option to disable the option for a particular message. If $crypt_autoencrypt or $crypt_replyencrypt enable encryption for a message, this option will be disabled for the message. It can be manually re-enabled in the pgp or smime menus. (Crypto only)

there are currently at least 3 ways to refer to a gpg key: short key ID (last 8 hex digits of fingerprint), long key ID (last 16 hex digits) and full fingerprint. The short key ID used to be popular, and since 5 years it is known that it is computationally easy to generate a gnupg key with an arbitrary short key id. A mitigation to this is using "keyid-format long" in gpg.conf, and a better thing to do, especially in scripts, is to use the full fingerprint to refer to a key, or just ship the public key for verification and skip the key servers. Note that in case of keyid collision, gpg will download and import all the matching keys, and will use all the matching keys for verifying signatures.

AB41 C1C6 8AFD 668C A045  EBF8 673A 03E4 C1DB 921F

• First of all, configure your local GPG to never show you short IDs. This is done by adding the line keyid-format 0xlong to ~/.gnupg/gpg.conf.
  • I just checked both in /usr/share/gnupg/options.skel and /usr/share/gnupg2/gpg-conf.skel, and that setting is not yet part of the packages' defaults. I'll check a bit deeper and file bugs right away if needed!
• Have you written or maintained any program that deals with GPG keys in any way? Make sure it specifies --keyid-format long or 0xlong in any relevant call. It might even be better to use the machine-oriented representation instead: --with-colons.
  • ...Of course, your computer will not feel tired or confused at comparing 160-bit full fingerprints instead of 64-bit long IDs, so it's better if our scripts use the full version for everything.
• Only sign somebody else's key if you see and verify its full fingerprint (this is not a new issue, but given we are talking about it, and that DebConf is around the corner, and that we will have a KSP as usual...)

Improving distributed and secure communication using free software This the title of my project for this summer. It sounds good, but what am I going to really do? Well, since I m a student at Universit du Qu bec Montr al, I have had the opportunity to meet with the company Savoir-Faire Linux in Montreal last year and that s when I found out about their exciting project: Ring.

Ring is one of the few projects which bring communication confidentiality and freedom in the hands of the users.

OpenDHT Ring is divided in multiple components as explained here: https://ring.cx/en/about/technical. The component I work on is called OpenDHT. This is the distributed hash table which enables Ring users to communicate in a decentralized network. I have already been working on this project for a year now, so you won t see me posting reports where I say I have to find my way around in this project. In fact, I have contributed to rewriting a major part of the code for better maintainability. In the begining of last summer, I was able to be part of a research association between Savoir-Faire Linux and Universit du Qu bec Montr al. We have been working on adding two major features to OpenDHT.

Short introduction to Distributed Hash Tables OpenDHT is based on the Kademlia design. If you know about DHTs, you are aware that they define the notion of distance using the XOR metric. This makes the tuple (H_n, xor) a metric space, where H_n is the space of identifier keys of length n. This way, you can find an identifier be the closest to a target hash than the rest of the nodes in the network.

In order to find some data that is associated with some hash h, you have to find the nodes in the network for which the distance between their hash identifier and the target hash h is minimized. The group of nodes which are the closest to the target hash (OpenDHT allows up to 8 nodes) will hold data for that hash.

Data persistence Let s say you ask a group of 8 nodes to hold some data for a hash and you want the data to hold the whole time until its expiration time as come. If the group of 8 nodes change because that for some reason those nodes leave or others just arrive and have an id that is closer to the target hash than the initial group of 8 nodes. The data would then not be found on the new group of 8 nodes. A first and simple solution was to count on the initial creator of the value to periodically announce the value to the group of 8 nodes. However, what if this node leaves? This is one of the main problem that I ve been working on solving since 2015. A first version of this was produced on September 2015. However, we experienced traffic issues and were forced to redesign the OpenDHT network requests engine. We are presently working on adding a Query feature that would enable filtering of data on remote nodes, hence substantially lowering the overall traffic produced by a response to a get request.

Indexation During the last year, I have also been working on the design of a solution for the use case of indexation. In a more technical way, this is the capability to find data by providing a map where its key is a string field and the mapped value is the value associated to that field, much like in a form. This could help find data for which you don t know the hash, but you have specific information about its content. In other words, this would be a search engine for OpenDHT. I have already created a working, but still in progress, version of this. You can find it on http://opendht.net, on the index branch. Now, Nicolas Reynaud (kaldoran) is contributing to this during the GSOC.

Roadmap As I explained on my page on the Debian Wiki, I am going to work on:

1. Developing new functionalities in OpenDHT aiming at reducing overall generated traffic.
2. Maintenance and optimization of the OpenDHT code in general.
3. Optimizing data persistence solution over the distributed hash table.
4. Merge (1) in the Ring daemon component in order to benefit from lower traffic in Ring.
5. Make OpenDHT use TCP protocol instead of UDP. This is going to reduce code complexity and enhance robsutness of the DHT.

For further details, read my reports!

• Read documentation related to argparse python module.
• Debug the code to find the ways to implement the solution
• Discussed the problem with the community.
• Based on progress I have pushed some code here (work is in progress)

Analyzing data input In the past, I have used Workrave to document how I work too much on the computer, but never went through more serious processing of the vast data store that Workrave accumulates about mouse movements and keystrokes. Interested in knowing how much my leave from Koumbit affected time spent on the computer, I decided to look into this again. It turns out I am working as much, if not more, on the computer since I took that "time off": We can see here that I type a lot on the computer. Normal days range from 10 000 to 60 000 keystrokes, with extremes at around 100 000 keystrokes per day. The average seem to fluctuate around 30 to 40 000 keystrokes per day, but rises sharply around the end of the second quarter of this very year. For those unfamiliar with the underlying technology, one keystroke is roughly one byte I put on the computer. So the average of 40 000 keystrokes is 40 kilobyte (KB) per day on the computer. That means 15 MB over a year or about 150MB (or 100 MiB if you want to be picky about it) over the course of the last decade. That is a lot of typing. I originally thought this could have been only because I type more now, as opposed to use more the mouse previously. Unfortunately, Workrave also tracks general "active time" which we can also examine: Here we see that I work around 4 hours a day continuously on the computer. That is active time: not just login, logout time. In other words, the time where i look away from the computer and think for a while, jot down notes in my paper agenda or otherwise step away from the computer for small breaks is not counted here. Notice how some days go up to 12 hours and how recently the average went up to 7 hours of continuous activity. So we can clearly see that I basically work more on the computer now than I ever did in the last 7 years. This is a problem - one of the reasons of this time off was to step away from the computer, and it seems I have failed.

Update: it turns out the graph was skewed towards the last samples. I went more easy on the keyboard in the last few days and things have significantly improved:

Another interesting thing we can see is when I switched from using my laptop to using the server as my main workstation, around early 2011, which is about the time marcos was built. Now that marcos has been turned into a home cinema downstairs, I went back to using my laptop as my main computing device, in late 2015. We can also clearly see when I stopped using Koumbit machines near the end of 2015 as well.

Further improvements and struggle for meaning The details of how the graph was produced are explained at the end of this article. This is all quite clunky: it doesn't help that the Workrave data structure is not easily parsable and so easily corruptible. It would be best if each data point was on its own separate line, which would be long, granted, but so easier to parse. Furthermore, I do not like the perl/awk/gnuplot data processing pipeline much. It doesn't allow me to do interesting analysis like averages, means and linear regressions easily. It could be interesting to rewrite the tools in Python to allow better graphs and easier data analysis, using the tools I learned in 2015-09-28-fun-with-batteries. Finally, this looks only at keystrokes and non-idle activity. It could be more interesting to look at idle/active times and generally the amount of time spent on the computer each day. And while it is interesting to know that I basically write a small book on the computer every day (according to Wikipedia, 120KB is about the size of a small pocket book), it is mostly meaningless if all that stuff is machine-readable code. Where is, after all, the meaning in all those shell commands and programs we constantly input on our keyboards, in the grand scheme of human existence? Most of those bytes are bound to be destroyed by garbage collection (my shell's history) or catastrophic backup failures.

While the creative works from the 16th century can still be accessed and used by others, the data in some software programs from the 1990s is already inaccessible. - Lawrence Lessig

But is my shell history relevant? Looking back at old posts on this blog, one has to wonder if the battery life of the Thinkpad 380z laptop or how much e-waste I threw away in 2005 will be of any historical value in 20 years, assuming the data survives that long.

How this graph was made I was happy to find that Workrave has some contrib scripts to do such processing. Unfortunately, those scripts are not shipped with the Debian package, so I requested that to be fixed (#825982). There were also some fixes necessary to make the script work at all: first, there was a syntax error in the Perl script. But then since my data is so old, there was bound to be some data corruption in there: incomplete entries or just plain broken data. I had lines that were all NULL characters, typical of power failures or disk corruptions. So I have made a patch to fix that script (#826021). But this wasn't enough: while this processes data on the current machine fine, it doesn't deal with multiple machines very well. In the last 7 years of data I could find, I was using 3 different machines: this server (marcos), my laptop (angela) and Koumbit's office servers (koumbit). I ended up modifying the contrib scripts to be able to collate that data meaningfully. First, I copied over the data from Koumbit in a local fake-koumbit directory. Second, I mounted marcos home directory locally with SSHFS:

sshfs anarc.at:/home/anarcat marcos

I also made this script to sum up datasets:

#!/usr/bin/perl -w
use List::MoreUtils 'pairwise';
$  = 1;
my %data = ();
while (<>)  
    my @fields = split;
    my $date = shift @fields;
    if (defined($data $date ))  
        my @t = pairwise   $a + $b   @ $data $date , @fields;
        $data $date  = \@t;
     
    else  
        $data $date  = \@fields;
     
 
foreach my $d ( sort keys %data )  
    print "$d @ $data $d \n";
 

Then I made a modified version of the Gnuplot script that processes all those files together:

#!/usr/bin/gnuplot
set title "Workrave"
set ylabel "Keystrokes per day"
set timefmt "%Y%m%d%H%M"
#set xrange [450000000:*]
set format x "%Y-%m-%d"
set xtics rotate
set xdata time
set terminal svg
set output "workrave.svg"
plot "workrave-angela.dat" using 1:28 title "angela", \
     "workrave-marcos.dat" using 1:28 title "marcos", \
     "workrave-koumbit.dat" using 1:28 title "koumbit", \
     "workrave-sum.dat" using 1:2 smooth sbezier linewidth 3 title "average"
#plot "workrave-angela.dat" using 1:28 smooth sbezier title "angela", \
#     "workrave-marcos.dat" using 1:28 smooth sbezier title "marcos", \
#     "workrave-koumbit.dat" using 1:28 smooth sbezier title "koumbit"

And finally, I made a small shell script to glue this all together:

#!/bin/sh
perl workrave-dump > workrave-$(hostname).dat
HOME=$HOME/marcos perl workrave-dump > workrave-marcos.dat
HOME=$PWD/fake-koumbit perl workrave-dump > workrave-koumbit.dat
# remove idle days as they skew the average
sed -i '/ 0$/d' workrave-*.dat
# per-day granularity
sed -i 's/^\(........\)....\? /\1 /' workrave-*.dat
# sum up all graphs
cat workrave-*.dat   sort   perl sum.pl > workrave.dat
./gnuplot-workrave-anarcat

I used a different gnuplot script to generate the activity graph:

#!/usr/bin/gnuplot
set title "Workrave"
set ylabel "Active hours per day"
set timefmt "%Y%m%d%H%M"
#set xrange [450000000:*]
set format x "%Y-%m-%d"
set xtics rotate
set xdata time
set terminal svg
set output "workrave.svg"
plot "workrave-angela.dat"  using 1:($23/3600) title "angela", \
     "workrave-marcos.dat"  using 1:($23/3600) title "marcos", \
     "workrave-koumbit.dat" using 1:($23/3600) title "koumbit", \
     "workrave.dat"         using 1:($23/3600) title "average" smooth sbezier linewidth 3
#plot "workrave-angela.dat" using 1:28 smooth sbezier title "angela", \
#     "workrave-marcos.dat" using 1:28 smooth sbezier title "marcos", \
#     "workrave-koumbit.dat" using 1:28 smooth sbezier title "koumbit"

1. Backup all your important data : Your home folder or any external partition will have tons of data if you have been using your OS for some time. Don t forget to double check all the folders such as your Desktop, localhost (/var/www/html/) and anything else you can think of.
2. Keep a list of your favorite applications that you ve installed. I wouldn t recommed copying the lib files from the computer. There is a high chance of getting your files corrupted that way (from personal experience). Instead, go to your package manager to find your installed packages and make a note of all the applications that you would want in your new OS.
3. Backup files from your browser. If you use Google Chrome (Or Chromium) be sure to sign in and save your settings. For firefox users, here s a guide to backup your profile. If your browser is something else, make sure to ask google for help with this,
4. Download the .iso file of your new OS according to your requirement. I prefer to make a live usb for installation. Make sure you double check and download the right image file.

• [1] https://etbe.coker.com.au/2015/09/05/long-term-review-android/
• [2] http://etbe.coker.com.au/2014/03/25/nexus5-armourdillo-hybrid-case/
• [3] https://en.wikipedia.org/wiki/Nexus_6P
• [4] https://etbe.coker.com.au/2014/06/04/android-device-service-life/

• golang-github-hpcloud-tail/1.0.0+git20160415.b294095-3: put versioned dependency & rebuild against golang-fsnotify/1.3.0-3 to fix FTBFS on ppc64el.
• updates: packer/0.10.1-1, pybtex/0.20.1-1, afl/2.12b-1, afl/2.13b-1, pyutilib/5.3.5-1.
• new packages: golang-github-azure-go-ntlmssp/0.0~git20160412.e0b63eb-1 (needed by Packer 0.10.1), and python-latexcodec/1.0.3-1 (needed by Pybtex 0.20).
• prospector/0.11.7-7 fixed for reproducible builds: there were variations in the sorting order of dependencies in prospector.egg-info/requires.txt. I've prepared a patch to make the package reproducible again (that problem began with 0.11.7-5) before the proposed toolchain patch for setuptools (#804249) gets accepted.
• python-latexcodec/1.0.3-3 also fixed for reproducible builds (#824454).

$ pyinfra -i <inventory script/single host> <deploy script>
$ pyinfra -i <inventory script/single host> --run <operation>
$ pyinfra -i <inventory script/single host> --facts <fact>

$ pyinfra -i 192.0.2.10 --run server.user sam --user root --key ~/.ssh/sshkey --key-password 123456 -v

COMPUTE_SERVERS = ['192.0.2.10', '192.0.2.11']
DATABASE_SERVERS = ['192.0.2.20', '192.0.2.21']

$ pyinfra -i inventory/farm1.py  --run server.user sam --user root --key ~/.ssh/sshkey --key-password=123456 -v

user_name = 'sam'
ssh_user = 'root'
ssh_key = '~/.ssh/sshkey'
ssh_key_password = '123456'

from pyinfra import host
from pyinfra.modules import server
server.user(host.data.user_name)

$ pyinfra -i inventory/farm1.py deploy.py

$ pyinfra -i 192.0.2.10 --fact deb_packages --user root --key ~/.ssh/sshkey --key-password=123456
 
    "192.0.2.10":  
        "libdebconfclient0": "0.192",
        "python-debian": "0.1.27",
        "libavahi-client3": "0.6.31-5",
        "dbus": "1.8.20-0+deb8u1",
        "libustr-1.0-1": "1.0.4-3+b2",
        "sed": "4.2.2-4+b1",

from pyinfra import host
from pyinfra.modules import apt
if host.fact.linux_distribution['name'] == 'Debian':
   apt.packages(packages='gummi', present=True, update=True)
elif host.fact.linux_distribution['name'] == 'CentOS':
   pass

• scripts for creating proper HDF5 training data of the MNIST database of handwritten digits and for training a simple neural network on it (create_mnist.py, mnist_pi.py),
• examples for setting up data and training a convolutional neural network (CNN) on the CIFAR-10 dataset of pictures (create_cifa10.py, cifar10_cnn.py),
• as well as example scripts for setting up training data and creating a LSTM (Long short-term memory) recurrent neural network (RNN) on test data used in the Hutter Prize competition (create_hutter.py, hutter_lstm.py).
• And there's also another example script for creating training data of the CIFAR-100 dataset (create_cifar100.py).

• Caffe for image recognition resp. classification⁷ is just around the corner (#823140).
• Theano is currently in experimental, and will be ready together with libgpuarray (OpenCL based GPU accelerated processing) and Keras (abstraction layer) for Stretch. It could already run on NVIDIA graphics card via CUDA⁸ (limited to amd64 and ppc64el, though).
• Lasagne, another abstraction layer for Theano is RFP (#818641).
• Google's Tensorflow, the free successor of Dist-Belief, is currently on ITP (#804612). It's waiting for Google's build system Bazel to become available.
• Torch is also ITP (#794634). It's blocked by a wishlist bug on dh-lua to get closed.
• Amazon's own machine learning workhorse dsstne ("destiny") is now also put under a free license and also will becoming available (#824692) in the near future for Debian (contrib). It's not yet for image recognition applications, though (lacks CNN).
• Mxnet is RFP (#808235).

# apt-get install xserver-xephyr firejail/experimental

$ firejail --x11=xephyr --private --private-tmp openbox
$ DISPLAY=$(firemon --x11   grep "DISPLAY"   sed 's/   DISPLAY //') \
  firejail --private --private-tmp skype

• A private /home/you so that Skype cannot access any of your files (disadvantage is that Skype can t remember your username and password; you can look at --private=directory to do something persistent).
• A private /tmp to avoid it going near any sockets.
• A private X11 server so that Skype cannot access the contents of any of your other windows (X11 inter-application security is virtually non-existent).
• The Firejail profile for Skype restricts the hardware it can access to only what it needs i.e. network, camera, microphone etc.
• The openbox window manager so you can close overlapping windows.

• Some packages update debian/changelog only at release time, e.g. auto-generated from the git changelog using git-dch
• Missing or misplaced release tags are not detected