Search Results: "tar"

23 March 2016

Ingo Juergensmann: Xen randomly crashing server

It's a long story... an oddessey of almost two years... But to start from the beginning: Back then I rented a server at Hetzner until they decided to bill for every IP address you got from them. I got a /26 in the past and so I would have to pay for every IP address of that subnet in addition to the server rent of 79.- EUR/month. That would have meant nearly doubling the monthly costs. So I moved with my server from Hetzner to rrbone Net, which offered me a /26 on a rented Cisco C200 M2 server for a competitve price. After migrating the VMs from Hetzner to rrbone with the same setup that was running just fine at Hetzner I experienced spontaneous reboots of the server, sometimes several times per day and in short time frame. The hosting provider was very, very helpful in debugging this like exchanging the memory, setting up a remote logging service for the CIMC and such. But in the end we found no root cause for this. The CIMC logs showed that the OS was rebooting the machine. Anyway, I then bought my own server and exchanged the Cisco C200 by my own hardware, but the reboots still happen as before. Sometimes the servers runs for weeks, sometimes the server crashes 4-6 times a day, but usually it's like a pattern: when it crashes and reboots, it will do that again within a few hours and after the second reboot the chances are high that the server will run for several days without a reboot - or even weeks. The strange thing is, that there are absolutely no hints in the logs, neither syslog or in the Xen logs, so I assume that's something quite deep in the kernel that causes the reboot. Another hint is, that the reboots fairly often happened, when I used my Squid proxy on one of the VMs to access the net. I'm connecting for example by SSH with portforwarding to one VM, whereas the proxy runs on another VM, which led to network traffic between the VMs. Sometimes the server crashed on the very firsts proxy requests. So, I exchanged Squid by tinyproxy or other proxies, moved the proxy from one VM to that VM I connect to using SSH, because I thought that the inter-VM traffic may cause the machine to reboot. Moving the proxy to another virtual server I rented at another hosting provider to host my secondary nameserver did help a little bit, but with no real hard proof and statistics, just an impression of mine. I moved from xm toolstack to xl toolstack as well, but didn't help either. The reboots are still happening and in the last few days very frequent. Even with the new server I exchanged the memory, used memory mirroring as well, because I thought that it might be a faulty memory module or something, but still rebooting out of the blue. During the last weekend I configured grub to include "noreboot" command line and then got my first proof that somehow the Xen network stack is causing the reboots:

This is a screenshot of the IPMI console, so it's not showing the full information of that kernel oops, but as you can see, there are most likely such parts involved like bridge, netif, xenvif and the physical igb NIC. Here's another screenshot of a crash from this night:

Slightly different information, but still somehow network involved as you can see in the first line (net_rx_action). So the big question is: is this a bug Xen or with my setup? I'm using xl toolstack, the xl.conf is basically the default, I think:

## Global XL config file ##

# automatically balloon down dom0 when xen doesn't have enough free
# memory to create a domain
autoballoon=0

# full path of the lockfile used by xl during domain creation
#lockfile="/var/lock/xl"

# default vif script
#vif.default.script="vif-bridge"

With this the default network scripts of the distribution (i.e. Debian stable) should be used. The network setup consists of two brdiges:

auto xenbr0
iface xenbr0 inet static
address 31.172.31.193
netmask 255.255.255.192
gateway 31.172.31.254
bridge_ports eth0
pre-up brctl addbr xenbr0 auto xenbr1
iface xenbr1 inet static
address 192.168.254.254
netmask 255.255.255.0
pre-up brctl addbr xenbr1

There are some more lines to that config like setting up some iptables rules with up commands and such. But as you can see my eth0 NIC is part of the "main" xen bridge with all the IP addresses that are reachable from the outside. The second bridge is used for internal networking like database connections and such. I would rather like to use a netconsole to capture the full debug output in case of a new crash, but unfortunately this only works until the bridge is brought up and in place:

[ 0.000000] Command line: placeholder root=UUID=c3....22 ro debug ignore_loglevel loglevel=7 netconsole=port@31.172.31.193/eth0,514@5.45.x.y/e0:ac:f1:4c:y:x
[ 32.565624] netpoll: netconsole: local port $port
[ 32.565683] netpoll: netconsole: local IPv4 address 31.172.31.193
[ 32.565742] netpoll: netconsole: interface 'eth0'
[ 32.565799] netpoll: netconsole: remote port 514
[ 32.565855] netpoll: netconsole: remote IPv4 address 5.45.x.y
[ 32.565914] netpoll: netconsole: remote ethernet address e0:ac:f1:4c:y:x
[ 32.565982] netpoll: netconsole: device eth0 not up yet, forcing it
[ 36.126294] netconsole: network logging started
[ 49.802600] netconsole: network logging stopped on interface eth0 as it is joining a master device

So, the first question is: how to use netconsole with an interface that is used on a bridge? The second question is: is the setup with two bridges with Xen ok? I've been using this setup for years now and it worked fairly well on the Hetzner server as well, although I used there xm toolstack with a mix of bridge and routed setup, because Hetzner didn't like to see the MAC addresses of the other VMs on the switch and shut the port down if that happens.

Kategorie:

Tags:

Daniel Pocock: GSoC 2016 opportunities for Voice, Video and Chat Communication

I've advertised a GSoC project under Debian for improving voice, video and chat communication with free software. Replacing Skype, Viber and WhatsApp is a big task, however, it is quite achievable by breaking it down into small chunks of work. I've been cataloguing many of the key improvements needed to make Free RTC products work together. Many of these chunks are within the scope of a GSoC project. If you can refer any students, if you would like to help as a mentor or if you are a student, please come and introduce yourself on the FreeRTC mailing list. If additional mentors volunteer, there is a good chance we can have more than one student funded to work on this topic. The deadline is Friday, 25 March 2016 The student application deadline is 25 March 2016 19:00 UTC. This is a hard deadline for students. Mentors can still join after the deadline, during the phase where student applications are evaluated. The Google site can be very busy in the hours before the deadline so it is recommended to try and complete the application at least 8 hours before the final deadline. Action items for students:

Register yourself on the Google Site and submit an application. You can submit applications to multiple organizations. For example, if you wish to focus on the DruCall module for Drupal, you can apply to both Debian and Drupal.
Join the FreeRTC mailing list and send a message introducing yourself. Tell us which topics you are interested in, which programming languages your are most confident with and which organizations you applied to through the Google site.
Create an application wiki page on the Debian wiki. You are permitted to edit the page after the 25 March deadline, so if you are applying at the last minute, just create a basic list of things you will work on and expand it over the following 2-3 days

Introducing yourself and making a strong application When completing the application form for Google, the wiki page and writing the email to introduce yourself, consider including the following details:

Link to any public profile you have on sites like Github or bug trackers
Tell us about your programming language skills, list the top three programming languages you are comfortable with and tell us how many years you have used each
other skills you have or courses you have completed
any talks you have given at conferences
any papers you have had published
any conferences you have attended or would like to attend
where you are located and where you study, including timezone
any work experience you already have
any courses, exams or employment commitments you have between 22 May and 24 August
anybody from your local free software community or university who may be willing to help as an additional mentor

Further reading Please also see my other project idea, for ham radio / SDR projects and my blog Want to be selected for Google Summer of Code 2016?. If you are not selected in 2016 We try to make contact with all students who apply and give some feedback, in particular, we will try to let you know what to do to increase your chances of selection in the next year, 2017. Applying for GSoC and being interviewed by mentors is a great way to practice for applying for other internships and jobs.

22 March 2016

Ben Hutchings: I'm super, thanks for asking!

Some of you (~~but~~ though probably not many) will know that I've suffered from depression at times. It has made it very difficult for me to do my job, to keep my home in order, even to have real conversations. Many important tasks seemed to require more energy than I had, and every setback left me feeling hopeless. I have sought treatment on several occasions, with very limited results. Certainly I would get better eventually, but it was hard to tell whether this had anything to do with the medication or therapy I received. And even while I felt better, I still didn't feel particularly happy or hopeful except when going to my favourite social events like DebConf. As I was recovering from the 'flu a weeks ago, I realised that I was feeling enthusiastic and energetic for no particular reason. Obviously I was recovering from physical illness, but that was brief. I had also just started a new project at work, but it isn't that exciting (at this stage) and I'm already aware of plenty of problems with it. So nothing external seemed to have changed to make me feel better, and yet my mood was so high that I worried that my illness could have changed to bipolar disorder and that I was entering a manic phase.

Feeling better mentally than I have in at least 6 months. Unfortunately my head's been buzzing with thoughts that kept me awake all night. Ben Hutchings (@benhutchingsuk) 29 February 2016

Thankfully, I'm not behaving manically. I have been doing a lot more shopping than usual, but almost the first purchase was a whiteboard to maintain the to-do list that I had been keeping rather unreliably in my head, and mostly I've been buying things off that list - that I've intended to get for months or years. It's just been so long since I could feel generally positive and optimistic about my life - outside of any special event - that I didn't immediately recognise this state of mind as normality.

@benhutchingsuk Did I say 6 months? I think it's more like a decade. There's a pretty deep pile of stuff I'm finally dealing with... Ben Hutchings (@benhutchingsuk) 7 March 2016

It's now been ~~two~~ three weeks since my recovery, and I still feel so much better:

I'm bothered far less by discomforts like feeling a little cold or hungry. I used to become almost paralysed by this, while feeling too short of energy to fix myself some food.
Pleasure is more intense. Food tastes and smells wonderful. Music feels more immersive. Sex made me cry.
I can easily have normal conversations with people I know and people I don't. I'm having important conversations I had been putting off.
I'm also able to write personal things like this, that I've shied away from for years.
When I see minor problems with the flat, I either fix them straight away or put them on the whiteboard to be fixed soon.
I'm far more persistent. If I run into an obstacle I'll either try a different approach or add an item to the whiteboard for whatever tools or supplies I'm missing.

As I feel better in myself I also find myself less self-centred and more able to focus on other people's needs, particularly my wife Nattie. Frankly I've been an asshole pretty often while I was depressed, so I apologise to those who've been the brunt of that. I hope you'll find me more pleasant to deal with from now on. There's lots more I could write, but I'll end this entry now and maybe say more at a later date.

20 March 2016

Jose M. Calhariz: at daemon, now with proper support for --without-selinux

Thanks to Wenzong Fan, now the configure flag --without-selinux works properly. So here is a new release at 3.1.19. Please test it.

Sean Whitton: Spring Break in San Francisco

Last night I got back from spending around 5 days in the Bay Area for Spring Break. I stayed in a hostel in downtown SF for three nights and then I stayed with a friend who is doing a PhD at Stanford. When initially planning this trip my aim was just to visit somewhere interesting on the west coast of the continental United States. I chose the Bay Area because I wanted to get my PGP key signed by some Debian Developers and that area has a high concentration of DDs, and because I wanted to see my friend at Stanford. But in the end I liked San Francisco a lot more than expected to and am very glad that I had an opportunity to visit. The first thing that I liked was how easy it seemed to be to find people interested in the same kind of tech stuff that I am. I spent my first afternoon in the city exploring the famous Mission district, and at one point while sitting in the original Philz Coffee I found that the person sitting next to me was running Debian on her laptop and blogs about data privacy. We had an discussion about how viable OpenPGP is as a component of a technically unsophisticated user s attempts to stay safe online. Later that same day while riding the subway train, someone next to me fired up Emacs on their laptop. And over the course of my trip I met five Debian Developers doing all sorts of different kinds of work both in and outside of Debian, and some Debian users including one of Stanford s UNIX sysadmins. This is a far cry from my day-to-day life down in the Sonoran Desert where new releases of iOS are all anyone seems to be interested in. Perhaps I should have expected this before my trip, but I think I had assumed that most of the work being done in San Francisco was writing web apps, so I was pleased to find people working on the same kind of things that I am currently putting time into. And in saying the above, I don t mean to demean the interests of the people around me in Arizona for a moment (nor those writing web apps; I d like to learn how to write good ones at some point). I m very grateful to be able to discuss my philosophical interests with the other graduate students. It s just that I miss being able to discuss tech stuff. I guess you can t have everything you want! One particular encouraging meeting I had was with a Debian Developer employed by Google and working on Git. While my maths background sets me up with the right thinking skills to write programs, I don t have knowledge typically gained from an education in computer science that enables one to work on the most interesting software. In particular, low-level programming in C is something that I had thought it wouldn t be possible for me to get started with. So it was encouraging to meet the DD working on Git at Google because his situation was similar: his undergraduate background is in maths and he was able to learn how to code in C by himself and is now working on a exciting project at a company that it is hard to get hired by. I don t mean that doing exactly what he s doing is something that I aiming for, just that it is very encouraging to know the field is more open to me than I had thought. I was also reminded of how fortunate I am to have the Internet to learn from and projects like Debian to get involved with. Moving on from tech, I enjoyed the streets of San Francisco, and the Stanford campus. San Francisco is fantastically multicultural though with clear class and wealth divisions. A very few minutes walk from the Twitter headquarters with its tech bros , as the maths PhD students I met at Stanford call them, are legions of the un- and barely-employed passing their time on the concrete. I enjoyed riding one of the old cable cars through the aesthetically revealing and stark combination of a west coast grid system on some very steep hills. I was fortunate to be able to walk across the Golden Gate Bridge on a perfectly clear and mist-free day. Meeting people involved with Debian and meeting my old friend at Stanford had me reflecting on and questioning my life in the desert even more than usual. I try to remind myself that there is an end date in sight and I will regret spending my time here just thinking about leaving. I sometimes worry that I could easily find myself moving to the big city London, San Francisco or elsewhere and letting myself be carried by the imagined self-importance of that, sidelining and procrastinating things that I should prize more highly. I should remember that the world of writing software in big cities isn t going away and my time in the desert is an opportunity to prepare myself better for that, building my resistance to being swept away by the tides of fashion.

19 March 2016

Petter Reinholdtsen: UsingQR - "Electronic" paper invoices using JSON and QR codes

Back in 2013 I proposed a way to make paper and PDF invoices easier to process electronically by adding a QR code with the key information about the invoice. I suggested using vCard field definition, to get some standard format for name and address, but any format would work. I did not do anything about the proposal, but hoped someone one day would make something like it. It would make it possible to efficiently send machine readable invoices directly between seller and buyer. This was the background when I came across a proposal and specification from the web based accounting and invoicing supplier Visma in Sweden called UsingQR. Their PDF invoices contain a QR code with the key information of the invoice in JSON format. This is the typical content of a QR code following the UsingQR specification (based on a real world example, some numbers replaced to get a more bogus entry). I've reformatted the JSON to make it easier to read. Normally this is all on one long line:

 
 "vh":500.00,
 "vm":0,
 "vl":0,
 "uqr":1,
 "tp":1,
 "nme":"Din Leverand r",
 "cc":"NO",
 "cid":"997912345 MVA",
 "iref":"12300001",
 "idt":"20151022",
 "ddt":"20151105",
 "due":2500.0000,
 "cur":"NOK",
 "pt":"BBAN",
 "acc":"17202612345",
 "bc":"BIENNOK1",
 "adr":"0313 OSLO"

The interpretation of the fields can be found in the format specification (revision 2 from june 2014). The format seem to have most of the information needed to handle accounting and payment of invoices, at least the fields I have needed so far here in Norway. Unfortunately, the site and document do not mention anything about the patent, trademark and copyright status of the format and the specification. Because of this, I asked the people behind it back in November to clarify. Ann-Christine Savlid (ann-christine.savlid (at) visma.com) replied that Visma had not applied for patent or trademark protection for this format, and that there were no copyright based usage limitations for the format. I urged her to make sure this was explicitly written on the web pages and in the specification, but unfortunately this has not happened yet. So I guess if there is submarine patents, hidden trademarks or a will to sue for copyright infringements, those starting to use the UsingQR format might be at risk, but if this happen there is some legal defense in the fact that the people behind the format claimed it was safe to do so. At least with patents, there is always a chance of getting sued... I also asked if they planned to maintain the format in an independent standard organization to give others more confidence that they would participate in the standardization process on equal terms with Visma, but they had no immediate plans for this. Their plan was to work with banks to try to get more users of the format, and evaluate the way forward if the format proved to be popular. I hope they conclude that using an open standard organisation like IETF is the correct place to maintain such specification. Update 2016-03-20: Via Twitter I became aware of some comments about this blog post that had several useful links and references to similar systems. In the Czech republic, the Czech Banking Association standard #26, with short name SPAYD, uses QR codes with payment information. More information is available from the Wikipedia page on Short Payment Descriptor. And in Germany, there is a system named BezahlCode, (specification v1.8 2013-12-05 available as PDF), which uses QR codes with URL-like formatting using "bank:" as the URI schema/protocol to provide the payment information. There is also the ZUGFeRD file format that perhaps could be transfered using QR codes, but I am not sure if it is done already. Last, in Bolivia there are reports that tax information since november 2014 need to be printed in QR format on invoices. I have not been able to track down a specification for this format, because of my limited language skill sets.

18 March 2016

Enrico Zini: Adjusting with Moore's Law

In my last post I used zxcvbn to check the strength of passwords, and it gave estimates in hundreds of centuries. However, there's Moore's law. If we assume that every 18 months the effective computing power will double...

#!/usr/bin/python3
import sys
years = int(sys.argv[1])
# How many 18 months intervals we spent
count = 0
# How many original-computation-time-months we compute each 18 months interval
step = 18
# How many original-computation-time-months we computed so far
total = 0
# How many original-computation-time-months we need
target = years * 12
# Iterate each 18 months interval
while True:
    count += 1
    total += step
    if total > target: break
    step *= 2
print(" :.2f  years needed".format(count / 12 * 18))

...then something that is now estimated in 152 centures is likely to take, well:

$ ./actual_years 15200
9.33 years needed

Update Fixed algorithm (thanks John MacGregor). Now the estimate is a little longer, but still:

$ ./actual_years 15200
21.00 years needed

ukasz Stelmach proposed an analytic solution:

There are two cars (remember phisics in primary school?). They both have the same speed V0. It is going to take "54.697 centuries" for car A to travel the desired distance at constant speed. Car B is accelerating, its speed is 2^(t/18m)*V0 (18 [m]onths). s = V0 * T (car A, T is 54.697 centuries) s = 2^(t/18)*V0 dt (car B, we're looking for t) It's been some time but as far as I can tell it is s = V0 * 2^(t/18) * 1/((1/18) * ln 2) 2^(t/18) * 1/((1/18) * ln 2) = 54.697 * 100 * 12 [months] 2^(t/18) = 2527.54 t = 18*log2(2527.54) t = 16.96 years (still reasonable strength).

17 March 2016

Patrick Matth i: Debian Jessie 8.3: Short howto for Corosync+Pacemaker Active/Passive Cluster with two nodes and DRBD/LVM

Hello, since I had to change my old heartbeat v1 setup to an more modern Corosync+Pacemaker setup, because heartbeat v1 does not support systemd (it first looks like it is working, but it fails on service start/stops), I want to share a simple setup:

Two nodes (node1-1 and node1-2)
Active/Passive setup
Shared IP (here: 123.123.123.123/24)
Internal network on eth1 (here: 192.168.99.0/24)
DRBD shared storage
LVM on top of DRBD
Multiple services, depending also on the DRBD/LVM storage

First you have to activate the jessie-backports repository, because the cluster stack is not available/broken in Debian Jessie. Install the required packages with:

apt-get install -t jessie-backports libqb0 fence-agents pacemaker corosync pacemaker-cli-utils crmsh drbd-utils

After that configure your DRBD and LVM (VG+LV) on it (there are enough tutorials for it). Then deploy this configuration to /etc/corosync/corosync.conf:

totem
version: 2
token: 3000
token_retransmits_before_loss_const: 10
clear_node_high_bit: yes
crypto_cipher: none
crypto_hash: none
transport: udpu
interface
ringnumber: 0
bindnetaddr: 192.168.99.0

logging
to_logfile: yes
logfile: /var/log/corosync/corosync.log
debug: off
timestamp: on
logger_subsys
subsys: QUORUM
debug: off

quorum
provider: corosync_votequorum
two_node: 1
wait_for_all: 1
nodelist
node
ring0_addr: node1-1

node
ring0_addr: node1-2

Both nodes require a passwordless keypair, which is copied to the another node, so that you can ssh from one to each other. Then you can start with crm configure:

property stonith-enabled=no
property no-quorum-policy=ignore
property default-resource-stickiness=100 primitive DRBD_r0 ocf:linbit:drbd params drbd_resource= r0 op start interval= 0 timeout= 240 \
op stop interval= 0 timeout= 100 \
op monitor role=Master interval=59s timeout=30s \
op monitor role=Slave interval=60s timeout=30s
primitive LVM_r0 ocf:heartbeat:LVM params volgrpname= data1 op monitor interval= 30s
primitive SRV_MOUNT_1 ocf:heartbeat:Filesystem params device= /dev/mapper/data1-lv1 directory= /srv/storage fstype= ext4 options= noatime,nodiratime,nobarrier op monitor interval= 40s primitive IP-rsc ocf:heartbeat:IPaddr2 params ip= 123.123.123.123 nic= eth0 cidr_netmask= 24 meta migration-threshold=2 op monitor interval=20 timeout=60 on-fail=restart
primitive IPInt-rsc ocf:heartbeat:IPaddr2 params ip= 192.168.99.4 nic= eth1 cidr_netmask= 24 meta migration-threshold=2 op monitor interval=20 timeout=60 on-fail=restart primitive MariaDB-rsc lsb:mysql meta migration-threshold=2 op monitor interval=20 timeout=60 on-fail=restart
primitive Redis-rsc lsb:redis-server meta migration-threshold=2 op monitor interval=20 timeout=60 on-fail=restart
primitive Memcached-rsc lsb:memcached meta migration-threshold=2 op monitor interval=20 timeout=60 on-fail=restart
primitive PHPFPM-rsc lsb:php5-fpm meta migration-threshold=2 op monitor interval=20 timeout=60 on-fail=restart
primitive Apache2-rsc lsb:apache2 meta migration-threshold=2 op monitor interval=20 timeout=60 on-fail=restart
primitive Nginx-rsc lsb:nginx meta migration-threshold=2 op monitor interval=20 timeout=60 on-fail=restart group APCLUSTER LVM_r0 SRV_MOUNT_1 IP-rsc IPInt-rsc MariaDB-rsc Redis-rsc Memcached-rsc PHPFPM-rsc Apache2-rsc Nginx-rsc
ms ms_DRBD_APCLUSTER DRBD_r0 meta master-max= 1 master-node-max= 1 clone-max= 2 clone-node-max= 1 notify= true colocation APCLUSTER_on_DRBD_r0 inf: APCLUSTER ms_DRBD_APCLUSTER:Master
order APCLUSTER_after_DRBD_r0 inf: ms_DRBD_APCLUSTER:promote APCLUSTER:start commit

The last (bold marked) lines made me some headache. In short they define that the DRBD device on the active node has to be the primary one and that it is required to start the APCLUSTER on the host, since the LVM, filesystem and services require to access its data. Just a short copy paste howto for an simple use case with not so much deep explanaitions..

16 March 2016

Enrico Zini: Postprocessing files saved by vim

Processing vim buffers through an external program on save, is a technique that is probably in use, and that I have not found documented anywhere, so here we go. Skip to the details for the entertaining example. I wanted to integrate egt with TaskWarrior so that every time I am editing an egt project file in vim, if I write a line in a certain way then a new taskwarrior task is created. Also, I have never written a vim plugin, and I don't want to dive into another rabbit hole just yet. Lynoure brought to my attention that taskwiki processes things when the file is saved, so I thought that I can probably get a long way by piping the file from vim into egt and reading it back before saving. I created a new egt annotate command that reads a project file, tweaks it, and then prints it out; I opened a project file in vim; I typed :%!egt annotate %:p --stdin and saw that it could be done. I find the result quite fun: I type 15 March: 18:00-21:00 in a log in egt, save the file, and it becomes 15 March: 18:00-21:00 3h. I do something in TaskWarrior, save the file in egt, and the lines that are TaskWarrior tasks update with the new task states. Details Here's a step by step example of how to hook into vim in this way. First thing, create a filter script that processes text files in some way. Let's call this /usr/local/bin/pyvimshell:

#!/usr/bin/python3
import sys
import subprocess
for line in sys.stdin:
    print(line.rstrip(), file=sys.stdout)
line = line.strip()
if line == "$":
    pass
elif line.startswith("$"):
    out = subprocess.check_output(["sh", "-c", line[1:].strip()], universal_newlines=True)
    sys.stdout.write(out)
    if not out.endswith("\n"):
        sys.stdout.write("\n")
    print("$ ", file=sys.stdout)

Then let's create a new filetype in ~/.vim/filetype.vim for our special magic text files:

if exists("did_load_filetypes")
  finish
endif
augroup filetypedetect
  au! BufNewFile,BufRead *.term setf SillyShell
augroup END

If you create a file example.term, open it in vim and type :set ft it should say SillyShell. Finally, the hook in ~/.vim/after/ftplugin/SillyShell.vim:

function! SillyShellRun()
    :%!/home/enrico/dev/egt/pyvimshell
    :$
endfunction
autocmd BufWritePre,FileWritePre <buffer> :silent call SillyShellRun()

Now you can create a file example.term, open it in vim, type $ ls, save it, and suddenly you have a terminal with infinite scrollback. For egt, I actually want to preserve my cursor position across saves, and egt also needs to know the path to the project file, so here is the egt version of the hook:

function! EgtAnnotate()
    let l:cur_pos = getpos(".")
    :%!egt annotate --stdin %:p
    call setpos(".", l:cur_pos)
endfunction
autocmd BufWritePre,FileWritePre <buffer> :silent call EgtAnnotate()

I find this adorably dangerous. Have fun.

14 March 2016

Julian Andres Klode: Dropping SHA-1 support in APT

Tomorrow ~~is the anniversary of Caesar s assassination~~ APT will see a new release, turning of support for SHA-1 checksums in Debian unstable and in Ubuntu xenial, the upcoming LTS release. While I have no knowledge of an imminent attack on our use of SHA1, Xenial (Ubuntu 16.04 LTS) will be supported for 5 years, and the landscape may change a lot in the next 5 years. As disabling the SHA1 support requires a bit of patching in our test suite, it s best to do that now rather than later when we re forced to do it. This will mean that starting tomorrow, some third party repositories may stop working, such as the one for the web browser I am writing this with. Debian Derivatives should be mostly safe for that change, if they are registered in the Consensus, as that has checks for that. This is a bit unfortunate, but we have no real choice: Technical restrictions prevent us from just showing a warning in a sensible way. There is one caveat, however: GPG signatures may still use SHA1. While I have prepared the needed code to reject SHA1-based signatures in APT, a lot of third party repositories still ship Release files signed with signatures using SHA-1 as the digest algorithms. Some repositories even still use 1024-bit DSA keys. I plan to enforce SHA2 for GPG signatures some time after the release of xenial, and definitely for Ubuntu 16.10, so around June-August (possibly during DebConf). For xenial, I plan to have a SRU (stable release update) in January to do the same (it s just adding one member to an array). This should give 3rd party providers a reasonable time frame to migrate to a new digest algorithm for their GPG config and possibly a new repository key. Summary

Tomorrow: Disabling SHA1 support for Release, Packages, Sources files
June/July/August: Disabling SHA1 support for GPG signatures (InRelease/Release.gpg) in development releases
January 2017: Disabling SHA1 support for GPG signatures in Ubuntu 16.04 LTS via a stable-release-update.

Filed under: Debian, Ubuntu

13 March 2016

Ingo Juergensmann: Letsencrypt: challenging challenges solved

A few weeks ago I was wondering in Letsencrypt: challenging challenges about how to setup Letsencrypt when a domain is spread across several virtual machines (VM). One of the possible solutions would be to consolidate everything on one single VM, which is nothing I would like to do. The second option would need to generate the Letsencrypt certs on the webserver and copy over the certs to the appropriate VM on a regular basis or event driven. The third option is to use a network share - and this is what I'm using right now. So, my setup is as following after I solved the GlusterFS issue with rpcbind binding to all interfaces, although it has been configured to only listen to certain interfaces (solution was: simply remove all NFS related stuff): On Dom0 (or the host machine) I run GlusterFS as a server on a small 1 GB LVM as part of a replicate with the VM that will do the actual Letsencrypt work:

Volume Name: le
Type: Replicate
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: 192.168.x.254:/srv/gfs/le
Brick2: 192.168.x.1:/srv/gfs/le

This is to ensure that on reboot of the machine every other VM using Letsencrypt certs can mount the GlusterFS share, because the host machine will be there for sure whereas the other VM generating the certs with the letsencrypt.sh script might still be booting. And when the GlusterFS share is missing services will not start on the other VMs because of the missing certs, of course. So, the replica on the virtualization host (Dom0) is only acting as some kind of always-being-available network share, because, well, the other VM will not always be there... for example during a kernel update when a reboot is required. The same setup is on my mailserver, acting as the second GlusterFS brick of that replica drive. The mailserver hosts the bind9 nameserver as well and I might do something that new domains with Letsencrypt certs get added to my DNSSEC setup as well. Of course, when the letsencrypt.sh script creates or updates the certs, it needs the certs being mounted in that configured location, so I needed to add a line to /etc/fstab:

192.168.x.254:/le /etc/letsencrypt.sh/certs glusterfs noexec,nodev,_netdev 0 0

Basically the same needs to be done on the other VMs where you want to use the certs as well, but you may want to mount the share as read-only there. The next step was a little more tricky. When letsencrypt.sh generates new certs, Letsencrypt will contact the webserver for that domain to respond to the ACME challenge. This requires that on each VM you want to use letsencrypt you have to run a webserver. Well, actually at least that there is somewhere a webserver that can answer these requests for that specific domain... Now, the setup of the webserver (Apache in my case) is like this: I'm using the Apache macro module to make it more easy, so I generated two small configs in /etc/apache/conf-available and enabled them bei a2enconf: letsencrypt-proxy.conf to do some setup for proxying the ACME challenges to a common website called acme.example.org. And then letsencrypt-sslredir.conf to setup SSL redirection when everything is in place and the domain can be switched over to HTTPS-only. letsencrypt-proxy.conf:

<Macro le_proxy>
ProxyRequests Off
<Proxy *>
Order deny,allow
Allow from all
</Proxy>
ProxyPass /.well-known/acme-challenge/ http://acme.windfluechter.net/
ProxyPassReverse / http://% HTTP_HOST /.well-known/acme-challenge/
</Macro>

letsencrypt-sslredir.conf:

<Macro le_sslredir>
RewriteEngine on
RewriteCond % HTTPS !=on
RewriteRule . https://% HTTP_HOST % REQUEST_URI [L]
</Macro>

So, after all the setup of a virtual host for Apache looks like this:

<Macro example.org>
(lots of setup stuff)
</Macro>
<VirtualHost 31.172.31.x:443 [2a01:a700:4629:x::1]:443>
Header always set Strict-Transport-Security "max-age=31556926; includeSubDomains"
SSLEngine on
# letsencrypt certs:
SSLCertificateFile /etc/letsencrypt.sh/certs/example.org/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt.sh/certs/example.org/privkey.pem
SSLHonorCipherOrder On
Use example.org
Use le_proxy
</VirtualHost>
<VirtualHost 31.172.31.x:80 [2a01:a700:4629:x::1]:80>
Use example.org
Use le_proxy
Use le_sslredir
</VirtualHost>

le_sslredir is only needed when you are sure that you want all traffic being redirected to HTTPS. For example when your blog is listed on planet.debian.org or other Planets you might want to omit this from your HTTP config because bug #813313 is not yet solved. In the end, when creating a new Letsencrypt cert, you need to add the le_proxy macro to your website, add the domain to letsencrypt.sh config in /etc/letsencrypt.sh/domains.txt and then the scripts will request a new cert from Letsencrypt, handling the ACME challenge stuff via the URL redirection in le_proxy being redirected to your acme.exmaple.org site and finally writes your new cert to the GlusterFS share. From that share you can then use the new cert on all needed VMs, be it your mailserver, webserver or XMPP/SIP server VMs. At least this works for me.

Kategorie:

Tags:

12 March 2016

Daniel Pocock: Google Summer of Code opportunities for ham radio and SDR

I've started preparing some ideas for Google Summer of Code projects I'd be willing to help mentor this year and one of them is for ham radio, with a focus on software defined radio (SDR). Can you help? If you can help as a co-mentor or simply help refer students for this project please get in touch with me. Ander the terms of the program students are paid a $US 5,500 stipend by Google and source code is fully published under a genuine free software license . Details about the project and how students can apply Students applying for this project are invited to submit two applications, one under the GNU Radio project and another under the Debian Project. The aim of this project is to make ready-to-run solutions for ham radio enthusiasts. The typical use case is a ham who has a spare computer in his shack, he should be able to boot the computer from DVD or USB stick using the Debian Ham Radio Live Blend or the GNU Radio Live SDR and have a functional transceiver within a few minutes. A student may not be able to do everything required for this project in one summer. We are looking for a student who can make any incremental improvement to bring us closer to this goal. Here are some of the tasks that may be involved:

survey the existing GNU Radio samples for ham radio, many are listed on the HamRadio page of the GNU Radio wiki.
design user interface improvements for the samples to make them more intuitive to new users and traditional radio operators. Consider how they can interact with Hardware such as a VFO tuning knob, PTT microphone switch and even a morse key.
look through the other packages in the Debian Ham Radio metapackage list and consider how they could interact with GNU Radio. In particular, we are interested in the use of message bus solutions, such as ZeroMQ or D-Bus - for example, GNU Radio could send alerts on the bus when incoming signals exceed the squelch threshold. GNU Radio could also receive events over a message bus, for example, patching GPredict to send Doppler shift information.
developing and packaging libraries needed to process digital voice transmissions
look at how one or more of the samples can be deployed as a Debian package so users can just install the package and have a working radio

Prerequisites The following experience is highly desirable:

ham radio license
GNU/Linux skills (Debian or Ubuntu or another distribution like Fedora)
use of version control systems (Git)
C++ or Python or both

Mentor(s)

Daniel Pocock (VK3TQR/M0GLR/HB9FZT)

Application process To apply

please introduce yourself on both the GNU Radio mailing list and the Debian Hams mailing list
Fill in the formal application for both GNU Radio and Debian
Pick some items from the list above or feel free to suggest another piece of work relevant to this theme. Give us a detailed, week-by-week plan for completing the task over the summer.
find at least one other member of the GNU Radio or Debian community who is willing to be a co-mentor on the project. Please try communicating with us over IRC or email and give us examples of your existing work on Github or elsewhere.

Lars Wirzenius: Not-platform for Debian project leader elections 2016

After some serious thinking, I've decided not to nominate myself in the Debian project leader elections for 2016. While I was doing that, I wrote the beginnings of a platform, below. I'm publishing it to have a record of what I was thinking, in case I change my mind in the future, and perhaps it can inspire other other people to do something I would like to happen. Why not run? I don't think I want to deal with the stress. I already have more than enough stress in my life, from work. I enjoy my obscurity in Debian. It allows me to go away for long periods of time, and to ignore any discussions, topics, and people that annoy or frustrate me, if I don't happen to want to tackle them at any one time. I couldn't do that if I was DPL. NOT a platform for Debian project leader election, 2016 Apart from what the Debian constitution formally specifies, I find that the important duties of the Debian project leader are:

Inspire, motivate, and enable the Debian community to make Debian better: to be the grease that makes the machinery run smoother. It is not important that the DPL do things, except to make sure other people can do.
Delegate what work can be delegated. The DPL is only one person, and should not be a bottleneck. Anything that can reasonably be delegated should be delegated.
Deal with mundane management tasks. This includes spending Debian money when it makes things easier, such as by funding sprints.
Represent the project in public, or find people to do that in specific cases.
Help resolve conflicts within the project.

I do not feel it is the job of the DPL to set goals for the project, technical or otherwise, any more than any other member of the project. Such goals tend to best come from enthusiastic individual developer who want something and are willing to work on it. The DPL should enable such developers, and make sure they have what they need to do the work. My plan, if elected

Keep Debian running. Debian can run for a long time effectively on autopilot, even if the DPL vanishes, but not indefinitely. At minimum, the DPL should delegate the secretary and technical committee members, and decide on how money should be spent. I will make sure this minimum level is achieved.
While I have no technical goals to set for the project, I have an organisational one. I believe it is time for the project to form a social committee whose mandate is to step in and help resolve conflicts in their early stages, before they grow big enough that the DPL, the tech-ctte, listmasters, or the DAM needs to involved. See below for more details on this. If I am elected, I will do my best to get a social committee started, and I will assume that any vote for me is also a vote for a social committee.

Social committee (Note: It's been suggested that this is a silly name, but I haven't had time to come up with anything better. I already rejected "nanny patrol".) We are a big project now. Despite our reputation, we are a remarkably calm project, but there are still occasional conflicts, and some of them spill out into our big mailing lists. We are not very good, as a project, in handling such situations. It is not a new idea, but I think its time has come, and I propose that we form a new committee, a social committee, whose job is to help de-escalate conflict situations while they are still small conflicts, to avoid them growing into big problems, and to help resolve big conflicts if they still happen. This is something the DPL has always been doing. People write to the DPL to ask for mediation, or other help, when they can't resolve a situation by themselves. We also have the technical committee, listmasters, GRs, and the expulsion process defined by the DAM. These are mostly heavy-weight tools and by the time it's time to consider their use, it's already too late to find a good solution. Having the DPL do this alone puts too much pressure on one person. We've learnt that important tasks should generally be handled by teams rather than just one person. Thus, I would like us to have a social committee that:

is reasonably small, like the technical committee
attempts to resolve conflicts at an early stage
is delegated by the DPL to have authority to do this
doesn't necessarily have direct authority to remove people from mailing lists, IRC channels, or expel them from the project, but has authority suggest such measures to the appropriate team
may do other things, such as educate people on how to resolve conflicts constructively themselves, or deal with chronic conflicts in addition to acute flare-ups

About me I've been a Debian developer since 1996. I've been retired twice, while I spent large amounts of time on other things. I haven't been a member of any important team in Debian, but I've been around long enough that I know many people, and have a reasonable understanding of how the projects works.

11 March 2016

Steinar H. Gunderson: Agon and the Candidates tournament

The situation where Agon (the designated organizer of the Chess World Championship, and also the Candidates tournament, the prequalifier to said WC) is trying to claim exclusive rights of the broadcasting of the moves (not just the video) is turning bizarre. First of all, they have readily acknowledged they have no basis in copyright to do so; chess moves, once played, are facts and cannot be limited. They try to jump through some hoops with a New York-specific doctrine (even though the Candidates, unlike the World Championship, is played in Moscow) about hot news , but their main weapon seems to be that they simply will throw out anyone from the hall who tries to report on the moves, and then try to give them only to those that promise not to give them on. This leads to the previously unheard-of situation where you need to register and accept their terms just to get to watch the games in your browser. You have to wonder what they will be doing about the World Championship, which is broadcast unencrypted on Norwegian television (previous editions also with no geoblock). Needless to say, this wasn't practically possible to hold together. All the big sites (like Chessdom, ChessBomb and Chess24) had coverage as if nothing had happened. Move sourcing is a bit of a murky business where nobody really wants to say where they get the moves from (although it's pretty clear that for many tournaments, the tournament organizers will simply come to one or more of the big players with an URL they can poll at will, containing the games in the standard PGN format), and this was no exception ChessBomb went to the unusual move of asking their viewers to download Tor and crowdsource the moves, while Chessdom and Chess24 appeared to do no such thing. In fact, unlike Chessdom and ChessBomb, Chess24 didn't seem to say a thing about the controversy, possibly because they now found themselves on the other side of the fence from Norway Chess 2015, where they themselves had exclusive rights to the PGN in a similar controversy although it would seem from a tweet that they were perfectly okay with people just re-broadcasting from their site if they paid for a (quite expensive) premium membership, and didn't come up with any similar legal acrobatics to try to scare other sites. However, their ToS were less clear on the issue, and they didn't respond to requests for clarification at the time, so I guess all of this just continues to be on some sort of gentleman's agreement among the bigger players. (ChessBomb also provides PGNs for premium members for the tournaments they serve, but they expressly prohibit rebroadcast. They claim that for the tournaments they host, which is a small minority, they provide free PGNs for all.) Agon, predictably, sent out angry letters where they threatened to sue the sites in question, although it's not clear at all to me what exactly they would sue for. Nobody seemed to care, except one entity TWIC, which normally has live PGNs from most tournaments, announced they would not be broadcasting from the Candidats tournament. This isn't that unexpected, as TWIC (which is pretty much a one-man project anyway) mainly is about archival, where they publish weekly dumps of all top-level games played that week. This didn't affect a lot of sites, though, as TWIC's live PGNs are often not what you'd want to base a top-caliber site on (they usually lack clock information, and moves are often delayed by half a minute or so). I run a hobby chess relay/analysis site myself (mainly focusing on the games of Magnus Carlsen), though, so I've used TWIC a fair bit in the past, and if I were to cover the Candidates tournament (I don't plan to do so, given Agon's behavior, although I plan to cover the World Championship itself), I might have been hit by this. So, that was the background. The strange part started when worldchess.com, Agon's broadcasting site, promptly went down during the first round of the Candidates tournament today Agon blamed DDoS, which I'm sure is true, but it's unclear exactly how strong the DDoS was, and if they did anything at all to deal with it other than to simply wait it out. But this lead to the crazy situation where the self-declared monopolist was the only big player not broadcasting the tournament in some form. And now, in the trully bizarre move, World Chess is publishing a detailed rebuttal of Agon's arguments, explaining how it is bad for chess, not juridically sound, and also morally wrong. Yes, you read that right; Agon's broadcast site is carrying an op-ed saying Agon is wrong. You at least have to give them credit for not trying to censor their columinst when he says something they don't agree with. Oh, and if you want those PGNs? I will, at least for the time being, be pushing them out live on http://pgn.sesse.net/. I have not gone into any agreement with Agon, and they're hosted in Norway, far from any New York-specific doctrines. So feel free to relay from them, although I would of course be happy to know if you do.

Alessio Treglia: We need creativity: which one?

A constant demand for creativity is raising from every corner of the Western world, from any business sector or professional activity, by individual or communities. This term is used everywhere, even in advertising to attract the attention of consumers: as a thirsty wanderer lost in the desert sand, the need for creativity seems to be the source of an oasis of salvation. alba

Julien Ries anthropological research showed us that, already more than two million years ago, Homo Habilis looks like Symbolicus, with aesthetic sensibility, sense of symmetry and consciousness of creativity. Gilbert Durand confirms that the specific activity of man, the identity card of Homo Sapiens, is the symbolic activity, an essential part of his creativity. Then, man is creative at the moment when his first activates his imaginative feature. So we can ask ourselves, how did we miss the creativity of man, of which so much we feel the need, or at least where is it hiding now? But above all which kind of creativity are we talking about? <Read More >

Matthew Garrett: I stayed in a hotel with Android lightswitches and it was just as bad as you'd imagine

I'm in London for Kubecon right now, and the hotel I'm staying at has decided that light switches are unfashionable and replaced them with a series of Android tablets.

A tablet displaying the text UK_bathroom isn't responding. Do you want to close it?

One was embedded in the wall, but the two next to the bed had convenient looking ethernet cables plugged into the wall. So.

I managed to borrow a couple of USB ethernet adapters, set up a transparent bridge (brctl addbr br0; brctl addif br0 enp0s20f0u1; brctl addif br0 enp0s20f0u2; ifconfig br0 up) and then stuck my laptop between the tablet and the wall. tcpdump -i br0 showed traffic, and wireshark revealed that it was Modbus over TCP. Modbus is a pretty trivial protocol, and notably has no authentication whatsoever. tcpdump showed that traffic was being sent to 172.16.207.14, and pymodbus let me start controlling my lights, turning the TV on and off and even making my curtains open and close. What fun!

And then I noticed something. My room number is 714. The IP address I was communicating with was 172.16.207.14. They wouldn't, would they?

I mean yes obviously they would.

It's basically as bad as it could be - once I'd figured out the gateway, I could access the control systems on every floor and query other rooms to figure out whether the lights were on or not, which strongly implies that I could control them as well. Jesus Molina talked about doing this kind of thing a couple of years ago, so it's not some kind of one-off - instead, hotels are happily deploying systems with no meaningful security, and the outcome of sending a constant stream of "Set room lights to full" and "Open curtain" commands at 3AM seems fairly predictable.

We're doomed.

(edited: this previously claimed I could only access systems on my own floor, but it turns out that each floor is a separate broadcast domain and I just needed to set a gateway to access the others)

(further edit: I'm deliberately not naming the hotel. They were receptive to my feedback and promised to do something about the issue.)

comments

Wouter Verhelst: File not found on exec

cat hello.c
#include <stdio.h>
int main(void)  
    printf("Hello World!\n");
    return 0;

A simple, standard, C program. Compiling and running it shouldn't produce any surprises... except when it does.

./hello
bash: ./hello: No such file or directory

The first time I got that, it was quite confusing. So, strace to the rescue -- I thought. But nope:

strace ./hello
execve("./hello", ["./hello"], [/* 14 vars */]) = -1 ENOENT (No such file or directory)

(output trimmed) No luck there. What's going on? No, it's not some race condition whereby the file is being removed. A simple ls -l will show it's there, and that it's executable. So what? When I first encountered this, I was at a loss. Eventually, after searching for several hours, I figured it out and filed it under "surprising things with an easy fix". And didn't think of it anymore for a while, because once you understand what's going on, it's not that complicated. But I recently realized that it's not that obvious, and I've met several people who were at a loss when encountering this, and who didn't figure it out. So, here goes: If you tell the kernel to run some application (i.e., if you run one of the exec system calls), it will open the binary and try to figure out what kind of application it's dealing with. It may be a script with a shebang line (in which case the kernel calls the appropriate interpreter), or it may be an ELF binary, or whatever. If it is an ELF binary, the kernel checks if the architecture of the binary matches the CPU we're running on. If that's the case, it will just execute the instructions in the binary. If it's not an ELF binary, or if the architecture doesn't match, it will fall back on some other mechanism (e.g., the binfmt_misc subsystem could have some emulator set up to run binaries for the architecture in question, or may have been set up to run java on jar files, etc). Eventually, if all else fails, the kernel will return an error:

./hello: cannot execute binary file: Exec format error

"Exec format error" is the error message for ENOEXEC, the error code which the kernel returns if it determines that it cannot run the given binary. This makes it fairly obvious what's wrong, and why. Now assume the kernel is biarch -- that is, it runs on an architecture which can run binaries for two CPU ISAs; e.g., this may be the case for an x86-64 machine, where the CPU can run binaries for that architecture as well as binaries for the i386 architecture (and its 32-bit decendants) without emulation. If the kernel has the option to run 32-bit x86 binaries enabled at compile time (which most binary distributions do, these days), then running i386 ELF binaries is possible, in theory. As far as the kernel is concerned, at least. So, the kernel maps the i386 binary into memory, and jumps to the binary's entry point. And here is where it gets interesting: When the binary in question uses shared libraries, the kernel not only needs to open this binary itself, but also the runtime dynamic linker (RTDL). It is then the job of the RTDL to map the shared libraries into memory for the process to use, before jumping to its code. But what if the RTDL isn't installed? Well, then the kernel won't find it. The RTDL is just this file, so that means it will produce an ENOENT error code -- the error message for which is "No such file or directory". And there you have it. The solution is simple, and the explanation too; but still, even so, it can be a bit baffling: the system tells you "the file isn't there", without telling you which file it's missing. Since you passed it only one file, it's reasonable for you to believe the missing file is the binary you're asking the system to execute. But usually, that's not the problem: the problem is that the file containing the RTDL is not installed, which is just a result of you not having enabled multiarch. Solution:

dpkg --add-architecture <target architecture>
apt update
apt install libc6:<target architecture>

Obviously you might need to add some more libraries, too, but that's not usually a problem.

Raphaël Hertzog: Freexian s report about Debian Long Term Support, February 2016

Like each month, here comes a report about the work of paid contributors to Debian LTS. Individual reports In February, 112.50 work hours have been dispatched among 11 paid contributors. Their reports are available:

Antoine Beaupr did 8h.
Ben Hutchings did 14 hours (out of 11.25 hours allocated + 10 extra hours remaining, meaning that he still has 7.25 extra hours to do over March).
Brian May did 10 hours.
Chris Lamb did 18 hours (instead of the 11.25 hours he was allocated, he will work less in March).
Damyan Ivanov did 4 hours (out of 11.25 hours allocated, leaving 7.25 extra hours for March).
Guido G nther did 7 hours (out of 11.25 hours allocated, leaving 4.25 extra hours for March).
Markus Koschany did 11.25 hours.
Mike Gabriel did 8 hours.
Santiago Ruano Rinc n did 12 hours (out of 11h allocated + 2.50 remaining, thus keeping 1.50 extra hours for March).
Scott Kitterman did 8 hours.
Thorsten Alteholz did 11.25 hours.

Evolution of the situation The number of sponsored hours continued to decrease a little bit. It s not worrisome yet but we should try to get back to a positive slope if we want to be able to do an outstanding job for wheezy LTS. On the positive side, TOSHIBA renewed their platinum sponsorship for another 6 months at least and we have some contacts for new sponsors, though they are far from being concluded yet. We are now in transition between squeeze LTS and wheezy LTS. The paid contributors are helping the security team by fixing packages that were fixed in squeeze already but that are still outstanding in wheezy. They are also taking generic measures to prepare wheezy LTS (for example to ensure all packages work with OpenJDK 7.x since support for 6.x will be dropped in the LTS period). Thanks to our sponsors New sponsors are in bold (none this month).

Platinum sponsors:

TOSHIBA (for 5 months)

Gold sponsors:

The Positive Internet (for 21 months)
Blablacar (for 20 months)
Linode LLC (for 10 months)

Silver sponsors:

Domeneshop AS (for 20 months)
Universit Lille 3 (for 20 months)
Trollweb Solutions (for 18 months)
Nantes M tropole (for 14 months)
University of Luxembourg (for 12 months)
Dalenys (for 10 months)
Univention GmbH (for 6 months)
Universit Jean Monnet de St Etienne (for 6 months)

Bronze sponsors:

David Ayers IntarS Austria (for 21 months)
Offensive Security (for 21 months)
Seznam.cz, a.s. (for 21 months)
Evolix (for 20 months)
Freeside Internet Service (for 20 months)
MyTux (for 20 months)
Linuxhotel GmbH (for 18 months)
Intevation GmbH (for 17 months)
Daevel SARL (for 16 months)
Bitfolk LTD (for 15 months)
Megaspace Internet Services GmbH (for 15 months)
Greenbone Networks GmbH (for 14 months)
NUMLOG (for 14 months)
WinGo AG (for 13 months)
Ecole Centrale de Nantes LHEEA (for 9 months)
Sig-I/O (for 7 months)
Entr ouvert (for 5 months)
Adfinis SyGroup AG

No comment Liked this article? Click here. My blog is Flattr-enabled.

10 March 2016

Lunar: Reproducible builds: week 45 in Stretch cycle

What happened in the reproducible builds effort between February 28th and March 5th:

Toolchain fixes

Antonio Terceiro uploaded gem2deb/0.27 that forces generated gemspecs to use the date from `debian/changelog`.

Antonio Terceiro uploaded gem2deb/0.28 that forces generated gemspecs to have their contains file lists sorted.

Robert Luberda uploaded ispell/3.4.00-5 which make builds of hashes reproducible.

C dric Boutillier uploaded ruby-ronn/0.7.3-4 which will make the output locale agnostic. Original patch by Chris Lamb.

Markus Koschany uploaded spring/101.0+dfsg-1. Fixed by Alexandre Detiste.

Ximin Luo resubmitted the patch adding the `--clamp-mtime` option to Tar on Savannah's bug tracker. Lunar rebased our experimental dpkg on top of the current master branch. Changes in the test infrastructure are required before uploading a new version to our experimental repository. Reiner Herrmann rebased our custom texlive-bin against the latest uploaded version.

Packages fixed The following 77 packages have become reproducible due to changes in their build dependencies: asciidoctor, atig, fuel-astute, jekyll, libphone-ui-shr, linkchecker, maven-plugin-testing, node-iscroll, origami-pdf, plexus-digest, pry, python-avro, python-odf, rails, ruby-actionpack-xml-parser, ruby-active-model-serializers, ruby-activerecord-session-store, ruby-api-pagination, ruby-babosa, ruby-carrierwave, ruby-classifier-reborn, ruby-compass, ruby-concurrent, ruby-configurate, ruby-crack, ruby-css-parser, ruby-cucumber-rails, ruby-delorean, ruby-encryptor, ruby-fakeweb, ruby-flexmock, ruby-fog-vsphere, ruby-gemojione, ruby-git, ruby-grack, ruby-htmlentities, ruby-jekyll-feed, ruby-json-schema, ruby-listen, ruby-markerb, ruby-mathml, ruby-mini-magick, ruby-net-telnet, ruby-omniauth-azure-oauth2, ruby-omniauth-saml, ruby-org, ruby-origin, ruby-prawn, ruby-pygments.rb, ruby-raemon, ruby-rails-deprecated-sanitizer, ruby-raindrops, ruby-rbpdf, ruby-rbvmomi, ruby-recaptcha, ruby-ref, ruby-responders, ruby-rjb, ruby-rspec-rails, ruby-rspec, ruby-rufus-scheduler, ruby-sass-rails, ruby-sass, ruby-sentry-raven, ruby-sequel-pg, ruby-sequel, ruby-settingslogic, ruby-shoulda-matchers, ruby-slack-notifier, ruby-symboltable, ruby-timers, ruby-zip, ticgit, tmuxinator, vagrant, wagon, yard. The following packages became reproducible after getting fixed:

air-quality-sensor/0.1.4-1 uploaded by Benedikt Wildenhain, fixed upstream, original patch by Chris Lamb.

device3dfx/2013.08.08-4 by Guillem Jover.

fldigi/3.23.08-1 by Kamal Mostafa.

fltk1.1/1.1.10-22 by Aaron M. Ucko.

freeimage/3.17.0+ds1-2 by Ghislain Antony Vaillant.

gimagereader/3.1.2+git368fa8f-2 by Philip Rinn.

ginkgocadx/3.7.5-1 by Gert Wollny, fixed upstream.

jadetex/3.13-17 by Norbert Preining.

opensips/2.1.2-1 by Razvan Crainea.

ruby-sqlite3/1.3.11-2 uploaded by C dric Boutillier, original patch by Lunar.

runawk/1.6.0-2 uploaded by Andrew Shadura, patch by Reiner Herrmann.

systraq/20160303-1 by Joost van Baal-Ili .

Some uploads fixed some reproducibility issues, but not all of them:

auto-multiple-choice/1.2.1-4 by Georges Khaznadar.

avfs/1.0.3-1 uploaded by Michael Meskes, original patch by Chris Lamb.

console-setup/1.138 uploaded by Anton Zinoviev, original patch by Reiner Herrmann.

gromacs/5.1.2-1 by Nicholas Breen.

mrrescue/1.02c-2 by Alexandre Detiste.

usb-modeswitch-data/20160112-2 by Didier Raboud.

Patches submitted which have not made their way to the archive yet:

#816209 on elog by Reiner Herrmann: use `printf` instead of `echo` which is shell-independent.

#816214 on python-pip by Reiner Herrmann: removes timestamp from generated Python scripts.

#816230 on rows by Reiner Herrmann: tell grep to always treat the input as text.

#816232 on eficas by Reiner Herrmann: use `printf` instead of `echo` which is shell-independent.

Florent Daigniere and bancfc reported that linux-grsec was currently built with `GRKERNSEC_RANDSTRUCT` which will prevent reproducible builds with the current packaging.

tests.reproducible-builds.org pbuilder has been updated to the last version to be able to support `Build-Depends-Arch` and `Build-Conflicts-Arch`. (Mattia Rizzolo, h01ger) New package sets have been added for Subgraph OS, which is based on Debian Stretch: packages and build dependencies. (h01ger) Two new `armhf` build nodes have been added (thanks Vagrant Cascadian) and integrated in our Jenkins setup with 8 new `armhf` builder jobs. (h01ger)

strip-nondeterminism development strip-nondeterminism version 0.016-1 was released on Sunday 28th. It will now normalize the `POT-Creation-Date` field in GNU Gettext `.mo` files. (Reiner Herrmann) Several improvements to the packages metadata have also been made. (h01ger, Ben Finney)

Package reviews 185 reviews have been removed, 91 added and 33 updated in the previous week. New issue: fileorder_in_gemspec_files_list. 43 FTBFS bugs were reported by Chris Lamb, Martin Michlmayr, and gregor herrmann.

Misc. After merging the patch from Dhiru Kholia adding support for `SOURCE_DATE_EPOCH` in rpm, Florian Festi opened a discussion on the rpm-ecosystem mailing list about reproducible builds. On March 4th, Lunar gave an overview of the general reproducible builds effort at the Internet Freedom Festival in Valencia.

Antoine Beaupr : Trying out Keybase

I was privileged enough to get access to the alpha preview of Keybase.io. Last year, the project raised 11M$ in venture capital and they seem to be attacking the hard problems, so it is a serious project, worth considering for any cryptography hacker like me. This was obviously discussed elsewhere previously but since Keybase has evolved a bit since then, I figured it would be worth adding my grain of salt.

The elite alpha problem My first problem with Keybase was of course that I wasn't in the elite club of invited people. Somehow, I finally got in but the problem remains for the everyone else. In my case, it was the Keybase filesystem announcement that brought my attention back to the project, although I have yet to test that itself - yet another invite-only problem (need to email `chris@keybase.io` to get access, I guess). Still, 10GB of storage might be interesting for some people, but considering the number of hurdles or "friction", as they say, that needs to be jumped over to get access, I am not sure how much traction this will get for now. The project has been in alpha for a while now (more than a year) with no announced public beta just yet, which is way slower than similarly challenging projects with less capital. Let's encrypt, as a comparison, was publicly announced in 2014 and by the end of 2015, the public beta was opened.

Private key leakage My second major concern is that Keybase does worrisome things to private key material. If you tell it to import your regular GPG keys, it will copy them into the keybase keyring, which is already a problem: now you have two places where you store sensitive secrets. But worse, Keybase offers you to upload your private key material on the server, without clearly and explicitely exposing the potential risks that means to the user. The key, presumably, is encrypted (at least that is what I can parse from the Keybase privacy policy). Yet this is still a concern: a determined attacker (e.g. the government or another malicious entity) could force Keybase to give up the encrypted private key material and install sniffers that would reveal the passphrase that would allow decryption and impersonation of the related identity. That is a huge deal. This is what caused Hushmail's demise in 2007: they had similar promises about how they had a "secure" email (in this case) service based on OpenPGP, but since they key was stored on the server, they only had to serve a malicious applet to victims and were able to give cleartext data to the US government (12 CDs!) under a Mutual Legal Assistance Treaty with Canada. In Keybase, When I generate a new key, it asks me if I want to upload the key to their servers, and "Yes" is chosen as a default, so it is clearly a policy that is encouraged. This is presumably to enable browser-based crypto, yet this could be done by storing keys within the browser, not on the server. The mere possibility that the client can meddle with secret key material in such a way is a problem. Even if uploading secret keys to the server is optional, the fact that it is a possibility widens the attack surface on the client significantly. GPG has been working on isolating private key operations to a different process (gpg-agent) or smart cards, SSH has privilege seperation. The idea of uploading secret key material online is so beyond current practices that it should bring the project to a full stop already. My private key material is pretty sensitive. I am a Debian developer: if someone gets access to that key, they can upload arbitrary code for thousands of packages, affecting thousands if not millions of users. That is a huge deal, and I am not ready to give that power to an arbitrary third party that I know little about.

Privacy policy issues Furthermore, the Keybase privacy policy explicitely states that "Logs of this information may persist for an indefinite period", which include, and I quote:

Your computer s IP address
Your preferences and settings (time zone, language, privacy preferences, application preferences, etc.)
The URL of the site that referred you to the Service
The buttons and controls you clicked on (if any) within the Service
How long you used the Service and which parts and features you used
Session times and lengths

For a company working to improve user's privacy, this is a pretty bad policy. It exposes Keybase users to undue surveillance, on a lot of private information that is not necessary to operate the service. I haven't actually read the remaining of the policy, as I was already scared as hell and figure I would just log off for now and try not to add any extra personal information in that already growing pile. Let's mention, while I'm here, that the keybase commandline client has this odd way of working that it forks a daemon in the background all the time, even to run just keybase status. One has to wonder if this thing is included in the surveillance data above, which would make the keybase client an amazing surveillance device in itself. To stop all this, you need:

keybase logout
keybase ctl stop

Not quite obvious...

Usability issues Which brings me to a series of usability issues I have found while working with Keybase. Having worked on usability myself in the Monkeysphere project, I understand how hard those problems can be. But right now, the fact is that keybase is only a commandline client, with some web-based sugar sprinkled on top. I recognize the huge efforts that have been done to make the user experience (UX) as easy as possible through the commandline, but this is definitely not "grand public" material yet. Even as a OpenPGP hacker, Keybase can get confusing. Since it has its own separate keyring from GPG, things that used to be obvious before are suddenly new things to learn. For example, I originally generated a key for `anarcat@debian.org` in keybase, thinking it would be a separate thing from my current GPG identity. Well, that ended up creating a new key in my GPG keyring, duplicating my existing identity, with no way for me to tell this was a Keybase copy. This could have ended up on the keyservers and confuse anyone looking for my PGP key.

Revoking keys So I revoked that key, which, in itself, is not such an obvious process either. I first had to "drop" the key:

keybase pgp drop '010143572d4e438f457a7447c9758804cc7be44c1ee2b7915c3904567d0a3fb5cf590a'

To find that magic number, I had to go on the website and click through the PGP key to end up on the revoke link which told me which magic string to put. The web UI was telling me I could use

keybase
status

to get that information, but I couldn't find that in the output here:

$ keybase status
Username:      anarcat
Logged in:     no
Device:
    name:      angela
    ID:        070f6a13f2a66afbb463f49dadfd4518
    status:    active
Keybase keys:  unlocked
Session:       anarcat [salt only]
KBFS:
    status:    not running
    version:
    log:       /home/anarcat/.cache/keybase/keybase.kbfs.log
Service:
    status:    running
    version:   1.0.14-1
    log:       /home/anarcat/.cache/keybase/keybase.service.log
Platform Information:
    OS:        linux
    Runtime:   go1.5.3
    Arch:      amd64
Client:
    version:   1.0.14-1
Desktop app:
    status:    not running
    version:
    log:       /home/anarcat/.cache/keybase/Keybase.app.log
Config path:   /home/anarcat/.config/keybase/config.json
Default user:  anarcat
Other users:
command-line client:
    keybase status [pid: 14562, version: 1.0.14-1]

So keybase drop "deletes" the key from the keybase client (and presumably the server). One problem here is that we have three different verbs to mean the same thing: in some places, the website says "deleted" (in the web graph), the commandline client uses "drop" and everyone else using PGP in the world uses "revoke". So what is it? Also note that just dropping that key is not enough to eradicate that key: you still need to generate and import a revocation certificate on GPG's side as well, for that key. This makes sense, because you may have imported that key and you may not want to destroy it when you remove it from keybase, but there could be a little better guessing and UX here as well.

Password usage Then another problem is that, when you `signup` on the commandline, it asks you to choose a passphrase. It's not clear at all from the UI what that passphrase is for. Is it to protect the private key material? To login the website? Both? It turns out it is actually to login to the website and the Keybase API. It could also be the key to the private key material, but I haven't quite figured that out yet. Similarly, the "paper devices" are a little confusing to me as well. The registration process is fairly insistent that I need to write a series of secret words down on a piece of paper and put it my wallet. This is based on the idea that you can use that key to recover your account and setup new devices. However, it is not made clear at all how those credentials should be protected or used. What if I lose my wallet? What do I do then? Is it okay if i write it on the back of my laptop instead? I know it sounds like stupid questions to crypto geek, but it's definitely stuff people will do, and education, at all levels, is necessary here.

The existing web of trust Finally, I found it surprisingly counter-intuitive to sign keys with keybase. I was assuming the whole point of the thing was to expand the web of trust, but it seems their mission lies somewhere else: there is no facility to sign keys directly in keybase. You sign statements on various social sites (Github, Twitter, Reddit...) but doing traditional key exchanges seems to be off limits. My obvious use case was to sign my existing key material with my newly generated anarcat@keybase.io key, so that keybase would become another way of verifying my identity. Well, this is not possible with keybase directly - you still have to go through the regular:

gpg --sign-key DEADBEEF

Which is really unfortunate, because some craaaazy people like me insist on doing in person, actually secure key exchanges sometimes, and the tools to do this right now (mainly gpg, caff and monkeysign) really need a lot of love and help. Keybase could have helped a lot in pushing those initiatives forward and bring new ways of doing secure key exchanges, instead of only lowering the standards on cryptographic identity.

Usual proprietary startup rant I can't help but finish by noticing that the whole basic problem with Keybase is more fundamental than a few usability issues, which can all be fixed fairly easily. Keybase.IO is a web-based service that is entirely proprietary. The client is freely downloadable and free software - but the server side is not. This makes us rely on a central point of failure and the goodwill of those operators to not only keep the service running (for free?) indefinitely, but to protect us against all attackers, state-run or otherwise. This is definitely against basic free software principles and the open architecture of the internet federation, well embodied in the Franklin street statement. Online services should be decentralized and open like the OpenPGP keyservers have always been, period.

Conclusion I am curious to see where Keybase brings us, as a community. So far, I am concerned about centralization and devaluation of the web of trust as cryptographic system. I do not believe that trusting multiple corporate social networking sites bring much benefit to our security, although it does improve usability significantly and is certainly better than TOFU, so that part of the project is definitely interesting, especially since it allows leveraging classical internet protocols like DNS and HTTPS. Key exchange is a critical problem that still hasn't properly been resolved in the cryptographic community. Most efforts at building communication tools (say like Signal or Telegram) mostly ignore the problem and expect people to read up strings of numbers or rely on synchronous communications (see the Axolotl rachet for Signal's take on it). Keybase's attempts at fixing this are great, but needs much more work to actually resolve the PKI issues in a significant way. More fundamentally, the practice of storing keys on the servers should just stop: it is a definite no go for me, and a classic crypto mistake that has bitten way too many people. In my mind, it discredits the whole project. The point of OpenPGP is end to end encryption across devices: storing the keys on a server breaks that apart, and doesn't improve much on the existing HTTPS security systems already in place on the web. It is especially a problem given the new waves of attacks on cryptography from western governments, from the UK to California... Until those issues are resolved, I can't get myself to recommend Keybase to anyone just yet.

Post-scriptum Note to Keybase developers: I have considered filing issues regarding the above, but unfortunately, I was unable to filter through the gigantic github issues list for duplicates, and didn't have time to file detailed reports for everything. I apologize for bypassing those usual conventions. If you want to try out Keybase to make your own opinion, I do have 4 invites I can give away, but I will do so only if you need to test specific issues. Otherwise, I note that there is a Request For Package for the Debian package to become official, that Debian developers (or Keybase developers!) may be interested in completing.

Next.

Previous.

Search Results: "tar"

23 March 2016

22 March 2016

20 March 2016

19 March 2016

18 March 2016

17 March 2016

16 March 2016

14 March 2016

13 March 2016

12 March 2016

11 March 2016

10 March 2016

strip-nondeterminism development strip-nondeterminism version 0.016-1 was released on Sunday 28th. It will now normalize the POT-Creation-Date field in GNU Gettext .mo files. (Reiner Herrmann) Several improvements to the packages metadata have also been made. (h01ger, Ben Finney)

Package reviews 185 reviews have been removed, 91 added and 33 updated in the previous week. New issue: fileorder_in_gemspec_files_list. 43 FTBFS bugs were reported by Chris Lamb, Martin Michlmayr, and gregor herrmann.

strip-nondeterminism development strip-nondeterminism version 0.016-1 was released on Sunday 28th. It will now normalize the `POT-Creation-Date` field in GNU Gettext `.mo` files. (Reiner Herrmann) Several improvements to the packages metadata have also been made. (h01ger, Ben Finney)