Search Results: "ron"

15 February 2014

Thomas Goirand: OpenStack 2013.2.2 uploaded

This is the 2nd point release of OpenStack Havana (this is the name of the current stable release of OpenStack). It was out on Thursday (US time), and I uploaded it on Friday (Chinese time). Unfortunately, I realized that the latest python-keystoneclient didn t support the token and endpoint command line options, effectively breaking Keystone itself, and all automated endpoint registration in all core packages. So after fixing Keystone and openstack-pkg-tools, I had to re-upload a 2nd Debian release. However, after this glitch, the packages passed successfully our CI testing suite: my friends at eNovance and I run the tempest functional test suite each time there s this kind of major update, so that we can validate the packages are working as expected. And all went well after the new keystone command line options were fixed in the postinst scripts. As a side note, I fail to see consistency on deprecating these interfaces. While we still have the old glance index thing in the Glance client (this is really old), keystoneclient just broke backward compatibility, when keystone is quite recent. So, it looks to me that it wasn t a great idea to just remove token and endpoint this way (though I can understand the need of consistency). Hopefully, this will all be fixed anyway, when we move to the unified python-openstackclient command lines instead of per-project clients. I realized today that the pkg-openstack team now maintains more than 100 packages. This of course includes general purpose Python modules, which probably will move to the Python module team if it one day supports Git (life is too short: I do not wish to learn such a deprecated technology as SVN). Though this is still a lot of packages, and some more are coming: as per the OpenStack TODO list wiki page, Ironic (a replacement for nova-baremetal), TripleO (OpenStack on OpenStack), Tuskar (contols where things are deployed with TripleO) and Designate (DNS as a Service) are already packaged, and I am waiting for the Icehouse (this is the name of the next stable release) beta3 to be out to upload them to Experimental. This is scheduled for the 6th of March. I believe I am currently nearly up-to-date with the current global-requirements.txt (eg: python dependencies of OpenStack), pending a few Python module upgrades, so hopefully, packaging OpenStack Icehouse beta3 will smooth.

Keith Packard: Altos1.3.2

AltOS 1.3.2 Bug fixes and improved APRS support Bdale and I are pleased to announce the release of AltOS version 1.3.2. AltOS is the core of the software for all of the Altus Metrum products. It consists of firmware for our cc1111, STM32L151, LPC11U14 and ATtiny85 based electronics and Java-based ground station software. This is a minor release of AltOS, including bug fixes for TeleMega, TeleMetrum v2.0 and AltosUI . AltOS Firmware GPS Satellite reporting and APRS improved Firmware version 1.3.1 has a bug on TeleMega when it has data from more than 12 GPS satellites. This causes buffer overruns within the firmware. 1.3.2 limits the number of reported satellites to 12. APRS now continues to send the last known good GPS position, and reports GPS lock status and number of sats in view in the APRS comment field, along with the battery and igniter voltages. AltosUI TeleMega GPS Satellite, GPS max height and Fire Igniters AltosUI was crashing when TeleMega reported that it had data from more than 12 satellites. While the TeleMega firmware has been fixed to never do that, AltosUI also has a fix in case you fly a TeleMega board without updated firmware. GPS max height is now displayed in the flight statistics. As the u-Blox GPS chips now provide accurate altitude information, we ve added the maximum height as computed by GPS here. Fire Igniters now uses the letters A through D to label the extra TeleMega pyro channels instead of the numbers 0-3.

14 February 2014

Paul Tagliamonte: Introducing: Acid!

I ve been hacking (on and off) on a small bit of code (written in Hy) called Acid. So, all of this is *really* fluid, and I m going to change it s API and code (it s currently just one massive hack), but I think the idea has kinda jelled enough for me to chat a bit about. Acid is a DSL for writing event-driven Hy. Currently it s just using global events (that ll change soon), and it s a monumental hack (not sure how much I can do there), but it works pretty well. It s built on top of tulip / asyncio, which is an amazing new Python 3.4 standard library package for working with async code. Although asyncio is designed around network-based usecases, I ve been trying to shoehorn in an event system on top of it. It s going well so far. A basic Acid script looks something like: 
(trip
  (on :startup
    (every 1 second (emit :clock-pulse nil)))
  (on :clock-pulse (print "."))
  (emit :startup nil))
which will print a dot ( . ) every second forever. Something a bit more advanced (fetch the MBTA Red line T information once a minute forever) 
(defn get-endpoint-url [line]
  (.format "http://developer.mbta.com/lib/rthr/ 0 .json"
    (get  :red-line  "red"
          :blue-line "blue"  line)))
(trip
  ;; OK. Let's do some work with MBTA feeds.
  (on :update-feed
      ;; let's just update feeds on a cron.
      (print (.json (.get requests (get-endpoint-url event))))
      (emit :feed-updated event))
  (every 1 minute
         (emit :update-feed :red-line)))
Clearly, both of these examples are just as clear as their straight functional counterparts, but I m keen to see what I can do with temporal recursion for long-running daemons. I think my first job will be porting snitch's codebase to Acid. Let me know if you have any ideas!

12 February 2014

Mario Lang: Roughly 1500 source packages have possibly broken links in debian/control

There are currently roughly 1500 source packages in Debian which possibly (very likely actually) do have broken URLs in debian/control. While it is quite useful that we have VCS information and Homepage URLs in the Packages file these days, we also created a rather big source of bitrot. These URLs are typically paste-and-forget. Sure, people occasionally catch the fact that a homepage or VCS has moved, especially if they are active and in good contact with their upstreams. However, there are also other cases...
DUCK to the rescue! My coworker Simon Kainz has worked on a service that helps at least to track which URLs are currently broken. We've initially discussed some way to make this a part of Lintian, which is what I would have prefered. However, for good reasons, Lintian doesn't want to call out to the net by default, so these checks would likely not get run by many developers anyway. So Simon ended up creating DUCK - the Debian Url ChecKer which actually goes out to the net and verifies that all the Vcs-* and Homepage fields of debian/control are actually reachable. The frontend allows developers to search for packages they maintain, to quickly see if they have any URLs which are possibly broken. There is a slight chance of temporary network problems of course, so what DUCK does is to show the status of checks in the last few days, so that you quickly see if you are dealing with a typical false positive. First of all, thanks to Simon! I think DUCK is an excellent project for a future NM candidate. I actually already wanted to advocate him for DD, but we ended up on a website which suggested that new contributors should start by applying for DM these days, and only later go for DD. I find that actually quite strange, especially in Simons case, but well, we did not feel like argueing. Secondly, and thats also very important: whats needed to improve the overall quality of URLs in the package system is your attention! You can easily search for email addresses of maintainers or uploaders. Team members can create a bookmark entry that checks for problems in all packages maintained by the teams they are a member of. You just need to actually visit these pages from time to time. It would probably not be well received if we filed all these bugs at once :-). So we need you to care, since we don't want to generate too much noise regarding "just these broken URLs". Or are they? If your vcs fields are broken, debcheckout will not work properly. Which defeats the purpose of debcheckout. If your homepage URL is broken, packages.d.o will also have a wrong link. 1500 packages with broken URLs. Don't you think we can do better then that?

Dirk Eddelbuettel: RInside 0.2.11

A new release 0.2.11 of RInside is now on CRAN. RInside provides a set of convenience classes which facilitate embedding of R inside of C++ applications and programs, using the classes and functions provided by the Rcpp R and C++ integration package. This release, the first in fourteen months, fixes one important initialization issue created by the recent Rcpp 0.11.0 release, adds a few other changes related to that release and improves a number of small points such as new or improved examples. The NEWS extract below has more details.
Changes in RInside version 0.2.11 (2014-02-11)
  • Updated for Rcpp 0.11.0:
    • Updated initialization by assigning global environment via pointer only after R itself has been initialized with special thanks to Kevin Ushey for the fix
    • Updated DESCRIPTION with Imports: instead of Depends:
    • Added correspondiing importFrom(Rcpp, evalCpp) to NAMESPACE
    • Noted in all inst/examples/*/Makefile that Rcpp no longer requires a library argument, but left code for backwards compatibility in case 0.11.0 is not yet installed.
  • Added --vanilla --slave to default arguments for R initialization
  • Added a few more explicit #include statements in the qt example which Qt 5.1 now appears to require with thanks to Spencer Behling for the patch
  • Added new MPI example with worker functions and RInside instance, kindly contributed by Nicholas Pezolano and Martin Morgan
CRANberries also provides a short report with changes from the previous release. More information is on the RInside page. Questions, comments etc should go to the rcpp-devel mailing list off the Rcpp R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Don Armstrong: Working with Org-mode: Committing Changes Everywhere

I'm a huge fan of Org-mode, and I keep all of my org-mode files in git repositories which are under myrepos control. However, because I often make lots of changes to my agenda and notes, I hate having to manually visit each individual project and make changes to it. [And it's also annoying when I forget to commit a specific change and then have to try to get my laptop and desktop back into sync.] Luckily, myrepos can easily run a command in parallel in all of the repositories! The following "update_org_files" command will update all of my org-file containing repositories in parallel: 
#!/bin/bash
ORG_GREP='-e .org$ -e .org_archive$ -e .org_done$'
if [ "x$1" == "xdoit" ]; then
    if git status --porcelain -z   grep -z '^ M'   grep -zq $ORG_GREP; then
        git status --porcelain -z   grep -z '^ M'   grep -z $ORG_GREP   \
            sed -z 's/^ M//g'   \
            xargs -0 git commit -m'update org files'
        git push;
    fi; 
else
    emacsclient -n -e '(org-save-all-org-buffers)' >/dev/null 2>&1
    mr -d ~ -j5 run update_org_files doit;
fi;
An updated version of this lives in my git repository

10 February 2014

Mario Lang: Neurofunkcasts

I have always loved Drum and Bass. In 2013 I rediscovered my love for Darkstep and Neurofunk, and found that these genres have developed quite a lot in the recent years. Some labels like Black Sun Empire and Evol Intent produce mixes/sets on a regular basis as podcasts these days. This article aggregates some neurofunk podcasts I like a lot, most recent first. Enjoy 33 hours and 57 minutes of fun with dark and energizing beats. Thanks to BSE Contrax and Evol Intent for providing such high quality sets. You can also see the Python source for the program that was used to generate this page.

Russell Coker: Fingerprints and Authentication

Dustin Kirkland wrote an interesting post about fingerprint authentication [1]. He suggests using fingerprints for identifying users (NOT authentication) and gives an example of a married couple sharing a tablet and using fingerprints to determine who s apps are loaded. In response Tollef Fog Heen suggests using fingerprints for lightweight authentication, such as resuming a session after a toilet break [2]. I think that one of the best comments on the issue of authentication for different tasks is in XKCD comic 1200 [3]. It seems obvious that the division between administrator (who installs new device drivers etc) and user (who does everything from playing games to online banking with the same privileges) isn t working, and never could work well particularly when the user in question installs their own software. I think that one thing which is worth considering is the uses of a signature. A signature can be easily forged in many ways and they often aren t checked well. It seems that there are two broad cases of using a signature, one is to enter into legally binding serious contract such as a mortgage (where wanting to sign is the relevant issue) and the other is cases where the issue doesn t matter so much (EG signing off on a credit card purchase where the parties at risk can afford to lose money on occasion for efficient transactions). Signing is relatively easy but that s because it either doesn t matter much or because it s just a legal issue which isn t connected to authentication. The possibility of serious damage (sending life savings or incriminating pictures to criminals in another jurisdiction) being done instantly never applied to signatures. It seems to me that in many ways signatures are comparable to fingerprints and both of them aren t particularly good for authentication to a computer. In regard to Tollef s ideas about lightweight authentication I think that the first thing that would be required is direct user control over the authentication required to unlock a system. I have read about some Microsoft research into a computer monitoring the office environment to better facilitate the user s requests, an obvious extension to such research would be to have greater unlock requirements if there are more unknown people in the area or if the device is in a known unsafe location. But apart from that sort of future development it seems that having the user request a greater or lesser authentication check either at the time they lock their session or by policy would make sense. Generally users have a reasonable idea about the risk of another user trying to login with their terminal so user should be able to decide that a toilet break when at home only requires a fingerprint (enough to keep out other family members) while a toilet break at the office requires greater authentication. Mobile devices could use GPS location to determine unlock requirements, GPS can be forged, but if your attacker is willing and able to do that then you have a greater risk than most users. Some users turn off authentication on their phone because it s too inconvenient. If they had the option of using a fingerprint most of the time and a password for the times when a fingerprint can t be read then it would give an overall increase in security. Finally it should be possible to unlock only certain applications. Recent versions of Android support widgets on the lock screen so you can perform basic tasks such as checking the weather forecast without unlocking your phone. But it should be possible to have different authentication requirements for various applications. Using a fingerprint scan to allow playing games or reading email in the mailing list folder would be more than adequate security. But reading the important email and using SMS probably needs greater authentication. This takes us back to the XKCD cartoon.
Related posts:
  1. SASL Authentication and Debian/Wheezy After upgrading a mail server to Debian/Unstable (which will soon...
  2. biometrics and passwords In a comment on my post more about securing an...

9 February 2014

Neil Williams: Debian ARMMP in LAVA using iMX53

The home lab is my LAVA development environment and I ve recently got two iMX53 Quick Start Boards working with a typical LAVA master image based on a Linaro build of Linux 3.1.0 for mx5 with a Ubuntu Oneiric Ocelot 11.10 rootfs: 
 3.1.0-1002-linaro-lt-mx5 (buildd@hubbard) (gcc version 4.6.1 (Ubuntu/Linaro 4.6.1-9ubuntu3) ) #13-Ubuntu PREEMPT Fri Dec 16 01:21:07 UTC 2011
As part of my Debian work, it was clearly time to look at a current, Debian, kernel and rootfs and as I m developing and testing on Debian unstable, this would necessarily mean testing the Debian ARMMP (multi-platform) kernel which replaces the mx5 build used in Wheezy. 
Linux version 3.12-1-armmp (debian-kernel@lists.debian.org) (gcc version 4.8.2 (Debian 4.8.2-10) ) #1 SMP Debian 3.12.9-1 (2014-02-01)
I will be scripting the creation of a suitable image for these tests and there are other changes planned in LAVA to make it easier to build suitable images, but it is useful to document just how I worked out how to build the first image. Manual build steps First, I ve already discovered that the u-boot on the iMX53 doesn t like ext4, so the first step was to prepare an ext3 filesystem for the image. With a SATA drive attached, it was also much better to use that than the SD card, at least for generating the image. I m also doing this natively, so I am working inside the booted master image. This is fine as the master is designed to manipulate test images, so the only package I needed to install on the LAVA master image was debootstrap. I had an empty SATA drive to play with for these tests, so first prepare an ext3 filesystem: 
# mkfs.ext3 /dev/sda1
# mkdir /mnt/sata
# mount /dev/sda1 /mnt/sata
# mkdir /mnt/sata/chroots/
Start the debootstrap: 
# apt-get update
# apt-get install debootstrap
# debootstrap --arch=armhf --include=linux-image-armmp \
 --verbose unstable \
 /mnt/sata/chroots/unstable-armhf http://ftp.uk.debian.org/debian
Various actions in this chroot will need proc, so mount it here: 
# chroot /mnt/sata/chroots/unstable-armhf
# mount proc -t proc /proc
# mount devpts -t devpts /dev/pts
# exit
You may also have to edit the apt sources the LAVA master image doesn t have editors installed, so either use echo or download an edited file. I m using: 
deb http://ftp.uk.debian.org/debian sid main
flash-kernel needs changes For the initial tests, I ve got to get this image to boot directly from u-boot, so flash-kernel is going to be needed inside the chroot and to get the iMX53 to work with the ARMMP kernel and Device Tree, flash-kernel will need an update which will mean a patch: 
# chroot /mnt/sata/chroots/unstable-armhf
# apt-get update
# apt-get install patch flash-kernel
# cp /usr/share/flash-kernel/db/all.db /home
# cd /home
# patch -p2
The patch itself goes through a couple of iterations. Initially, it is enough to use: 
 Machine: Freescale MX53 LOCO Board
-Kernel-Flavors: mx5
+Kernel-Flavors: armmp
+DTB-Id: imx53-qsb.dtb
+DTB-Append: yes
Later, once the device has booted with the ARMMP kernel, the Machine Id can be updated to distinguish it from the mx5 flavour (from /proc/cpuinfo) and use the model name from the Device Tree (/proc/device-tree/model): 
diff --git a/db/all.db b/db/all.db
index fab3407..41f6c78 100644
--- a/db/all.db
+++ b/db/all.db
@@ -62,6 +62,18 @@ U-Boot-Initrd-Address: 0x00800000
 Required-Packages: u-boot-tools
 Bootloader-Sets-Incorrect-Root: yes
 
+Machine: Freescale i.MX53 Quick Start Board
+Kernel-Flavors: armmp
+DTB-Id: imx53-qsb.dtb
+DTB-Append-From: 3.12
+Boot-DTB-Path: /boot/dtb
+U-Boot-Kernel-Address: 0x70008000
+U-Boot-Initrd-Address: 0x0
+Boot-Kernel-Path: /boot/uImage
+Boot-Initrd-Path: /boot/uInitrd
+Required-Packages: u-boot-tools
+Bootloader-Sets-Incorrect-Root: no
+
 Machine: Freescale MX53 LOCO Board
 Kernel-Flavors: mx5
 U-Boot-Kernel-Address: 0x70008000
(I will be filing this patch in a bug report against flash-kernel soon.) With that patched, update and run flash-kernel: 
# mv all.db /usr/share/flash-kernel/db/all.db
# flash-kernel
flash-kernel: installing version 3.12-1-armmp
Generating kernel u-boot image... done.
Taking backup of uImage.
Installing new uImage.
Generating initramfs u-boot image... done.
Taking backup of uInitrd.
Installing new uInitrd.
Installing new dtb.
# exit
LAVA overlays This will be a LAVA test image and it needs an overlay if you want a vanilla image, set up a passwd inside the chroot instead. 
# cd /mnt/sata/chroots/unstable-armhf/
# wget --no-check-certificate https://launchpad.net/~linaro-maintainers/+archive/overlay/+files/linaro-overlay-minimal_1112.2_all.deb
# wget --no-check-certificate https://launchpad.net/~linaro-maintainers/+archive/overlay/+files/linaro-overlay_1112.2_all.deb
# chroot /mnt/sata/chroots/unstable-armhf/
# dpkg -i linaro-overlay-minimal_1112.2_all.deb linaro-overlay_1112.2_all.deb
# rm linaro-overlay-minimal_1112.2_all.deb linaro-overlay_1112.2_all.deb
# exit
Changes to allow the chroot to boot Now the chroot needs setting up as a boot image: 
# chroot /mnt/sata/chroots/unstable-armhf/
# echo T0:23:respawn:/sbin/getty -L ttymxc0 115200 vt102 >> ./etc/inittab
# echo auto lo eth0 > ./etc/network/interfaces.d/mx53loco
# echo iface lo inet loopback >> ./etc/network/interfaces.d/mx53loco
# echo iface eth0 inet dhcp >> ./etc/network/interfaces.d/mx53loco
# apt-get clean
# umount /dev/pts
# umount /proc
# exit
Partitioning as a LAVA test image This would be enough as a single partition test image but, currently, LAVA expects a much more hard-wired image. Depending on the history of the device and the need for LAVA to be able to put fresh kernel builds together with a known rootfs, LAVA has used a separate /boot and / partition in the test image for nearly all boards. Standard LAVA test images for many boards (like the iMX53) also have a small unallocated space at the start of the SD card, so until I can get LAVA upstream to handle test images of arbitrary design, I m adapting the image to suit the expectations inside LAVA. Yes, I know but it s better to get something working before spending time fixing things to make it work better. It will be fixed, in time. So I needed to separate out the /boot contents from the rest of the rootfs whilst keeping the chroot itself in a state which I can easily update and use to create other images. 
# cd /mnt/sata/chroots/unstable-armhf/boot/
# tar -czf ../../boot.tar.gz ./*
# cd ..
# tar -czf ../root.tar.gz ./*
Now the test image file and its partitions: 
# dd if=/dev/zero of=/mnt/sata/images/empty/debian.img bs=1M count=1024
# cp /mnt/sata/images/empty/debian.img /mnt/sata/images/debian-unstable-armhf-armmp.img
# losetup /dev/loop0 /mnt/sata/images/debian-unstable-armhf-armmp.img
# parted /dev/loop0 -s unit mb mktable msdos
# parted /dev/loop0 -s unit mb mkpart primary 1 10
# parted /dev/loop0 -s unit mb mkpart primary 11 110
# parted /dev/loop0 -s unit mb mkpart primary 111 1024
I did make the mistake of using kpartx at this stage but there are many areas of confusion when translating the kpartx output to offsets for mount when parted is easier: 
# parted /dev/loop0 unit B -s print
Model:  (file)
Disk /dev/loop0: 1073741824B
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Number  Start       End          Size        Type     File system  Flags
 1      1048576B    10485759B    9437184B    primary
 2      10485760B   110100479B   99614720B   primary
 3      110100480B  1024458751B  914358272B  primary
Use the Start numbers and use losetup to create the loop devices for each
partition 
# losetup -o 10485760 /dev/loop1 /dev/loop0
# losetup -o 110100480 /dev/loop2 /dev/loop0
# mkfs.vfat /dev/loop1
# mkfs.ext3 /dev/loop2
Now clean up the loop mountpoints: 
# losetup -d /dev/loop2
# losetup -d /dev/loop1
# losetup -d /dev/loop0
# losetup -a
losetup -a should return nothing. If it doesn t, investigate the contents of /dev/mapper and use dmsetup remove until losetup -a does report empty. Otherwise the subsequent stages will fail. Now deploy the /boot contents into the empty image: 
# mount -oloop,offset=10485760 /mnt/sata/images/debian-unstable-armhf-armmp.img /mnt/boot/
# pushd /mnt/boot/
# tar -xzf /mnt/sata/chroots/boot.tar.gz
# popd
# sync
# umount /mnt/boot/
and the / contents (removing the duplicate ./boot contents) 
# mount -oloop,offset=110100480 /mnt/sata/images/debian-unstable-armhf-armmp.img /mnt/root/
# pushd /mnt/root
# tar -xzf /mnt/sata/chroots/root.tar.gz
# rm ./boot/*
# popd
# sync
# umount /mnt/root
Check the image 
# mount -oloop,offset=10485760 debian-unstable-armhf-armmp.img /mnt/boot
# ls /mnt/boot
config-3.12-1-armmp  initrd.img-3.12-1-armmp  System.map-3.12-1-armmp  uImage  uInitrd  vmlinuz-3.12-1-armmp
# umount /mnt/boot
# mount -oloop,offset=110100480 debian-unstable-armhf-armmp.img /mnt/root
# ls /mnt/root
bin  boot  dev  etc  home  initrd.img  lib  lost+found  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var  vmlinuz
# ls /mnt/root/bin/auto-serial-console
/mnt/root/bin/auto-serial-console
# umount /mnt/root
# md5sum debian-unstable-armhf-armmp.img
Downloads Yes, I have put that file online, if you are interested. Do read the readme first though. Getting the image off the device 
# ip a
# python -m SimpleHTTPServer 80 2>/dev/null
then wget (just http://, IP address / and the filename), md5sum , finish with Ctrl-C. Depending on setup, it may be quicker to transfer the uncompressed
file over a LAN than to let the device compress it. Your main machine
will do the compression much faster, even with a larger download. (It also helps to not compress the image on the device, you can
test the mount offsets more easily and do check that the image
can be mounted with the offsets from parted.) Results 
# cat /proc/cpuinfo
processor	: 0
model name	: ARMv7 Processor rev 5 (v7l)
Features	: swp half thumb fastmult vfp edsp thumbee neon vfpv3 tls vfpd32 
CPU implementer	: 0x41
CPU architecture: 7
CPU variant	: 0x2
CPU part	: 0xc08
CPU revision	: 5
Hardware	: Freescale i.MX53 (Device Tree Support)
Revision	: 0000
Serial		: 0000000000000000

# uname -a
Linux imx53-02 3.12-1-armmp #1 SMP Debian 3.12.9-1 (2014-02-01) armv7l GNU/Linux

# lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux unstable (sid)
Release:	unstable
Codename:	sid
Next! Yes, there is a lot to do from here on. Then there is the whole issue of actually making this multi-platform. After all, it is far from ideal that a multi-platform kernel package has to have platform-specific operations using flash-kernel at each update. So Grub on ARM is going to be on the list of things to investigate .

8 February 2014

Sam Hartman: IETF Takes on Challenge of Internet Monitoring

At IETF 88, we held a plenary discussion of how we could harden the Internet against ongoing monitoring and survalence. There were no significant surprises in what people said about monitoring. So, we had an opportunity to focus on what the IETF as a standards organization responsible for technical standards that drive the Internet can do about this problem. I think we made amazing progress. The IETF works by consensus. We discuss issues, and see if there s some position that can gain a rough consensus of the participants in the discussion. After a couple of hours of discussion, Russ Housley asked several consensus questions. The sense of the room is that we should treat these incidents of monitoring as an attack and include them in the threats we try and counter with security features in our protocols. The room believes that encryption, even without authentication has value in fighting these attacks. There was support for the idea of end-to-end encryption is valuable even when there are middle boxes. IETF decisions made in meetings are confirmed on public mailing lists, so the sense of the room is not final. Also, note that I did not capture the exact wording of the questions that were posed. This is huge. There is very strong organizational agreement that we re going to take work in this space as seriously. Now that we ve decided pervasive monitoring is an attack, anyone can ask how a proposed protocol (or change to a protocol) counters that attack. If it doesn t handle the attack and there is a way to address the attack, then we will be in a stronger position arguing the threat could be addressed. In addition, the commitment to encryption providing value without authentication will be useful in providing privacy and minimizing fingerprinting by passive attackers. The IETF is only one part of the solution. We can work on the standards that describe how systems interact. However, implementations, policy makers, operators and users will also play a role in securing the Internet against pervasive attacks.

Enrico Zini: original-sin

Original sin I feel that I was somewhat born innocent, an animal with sound, primal instincts. But if I remained that way, I wouldn't be able to function in a complex society, so I got education. Education taught me what is expected from me in order to be accepted by my peers[1]. Education taught me more than that: it also taught me to enjoy what a complex society can give me: art, science, history, philosophy, adding depth of meaning and correlations to my perceptions and memories. Education wasn't perfect, though. Some of my educators obsessed about some of the expectations, and gave me rules to follow that aren't really needed to interact with a society. Sit down for hours in silence without complaining. Don't talk back to figures of authority, even when they are abusing me. Don't ever feel that my efforts in a task have been enough, because there is always something more that I could have done. Do what people expect me to do, regardless of what I wish to do. Do what I need to do, not what I want to do. So I grew up with a set of arbitrary expectations that weren't needed to function in a society, and weren't in any way meeting any of my needs, yet I still felt them as a part of me, putting every effort into meeting them that I would put in making myself happy. I like to call this set of learned arbitrary, unneeded expectations "neurosis". I like this as an interpretation of the "original sin" myth: in order to go from an innocent animal to a member of a complex society, I acquired a set of neuroses that make me behave in a meaningless way. Or, rephrased going along with the myth, that rob me of my innocence. In some environments like BDSM, many have a name for an unnegotiated practice forced on a person: they call it "abuse". It made sense to accept that I have been abused many times while being educated, because my educators also had their own baggage of parasite expectations, of neuroses. And since I don't currently know how I can ever be sure that I freed myself from all my neuroses, I feel like I should accept that it is possible that I do and will abuse others; by accepting it as a possibility, I hope at least to be able to realise as soon as possible that I am doing it, and try to stop. I have now realised that my life is far simpler and more rewarding when expectations are negotiated between all the parties involved. I have recently spent a substantial amount of my energy in recognising and renegotiating many of the expectations and the neuroses that I had learnt while growing up, and, who knows, perhaps this effort will continue throughout my life. So there you go, an agnostic atheist freeing himself of his original sin, and regaining his paradise lost one step at a time, alive, on earth. And enjoying every moment of it. [1] I suspect that is why people educated in a high class society tend to be accepted more easily by a high class society: they conform to the right set of expectations. But I digress.

Vasudev Kamath: Friendica instance on my VPS is down

I started running Friendica instance on my VPS. With help of Jonas Smedegaard I managed to run Friendica in a uWSGI container. The site was running at samsargika.copyninja.info and is no longer accessible. Since VPS itself was running Debian Wheezy I couldn't run uWSGI with PHP support on it. (support for PHP in uWSGI landed after Wheezy). But Jonas was kind enough for me to provide a backported version. Recently Wheezy got a security update for PHP and thats where all the problem started. The backported uwsgi-plugin-php was not recompiled to use security updated PHP and I couldn't upgrade the things. After few days I noticed first freeze on my VPS and I had to reboot the VPS to get it online again. The fact I noticed was uWSGI being killed due to a OOM in syslog but I didn't explore much and consulted Jonas to get a updated uWSGI. But that didn't happen as Jonas himself is facing some problem with uWSGI builds. While again checking with aptitude for upgrade I accidentally confirmed removal of uwsgi-plugin-php for getting security updates :-/. But nothing happened to my running service as upgrade of libs in Debian doesn't cause the restart of all services which are using that lib (desired effect is restart of service but I don't know the side affects involved). Second freeze happened yesterday and on the reboot uwsgi-plugin-php was missing there by taking my Friendica instance down. More closer investigation showed the same OOM but this time I noticed each OOM occurred just after cronjob was running poller.php a script which is actually causing all federation in Friendica. So it was clear there is something wrong either in poller.php or in my setup which was making it eat memory and freeze my VPS. But I also found some stupidity I did during configuration which Jonas also pointed me out.
  1. Installing cronjob inside crontab rather than cron.d
  2. Installing poller.php crontab for root user :-/
I basically violated the basic rule by running a unsafe script as root user, good that script didn't do some crazy stuff. So even though my instance went down I learnt my lessons
  1. Don't ever ever ever run a unsafe script as root that too through cron
  2. Sandbox the unsafe script so it can be killed on time rather than it taking the whole system down.
  3. PHP is not really secure, if it was secure there wouldn't be security updates and atleast my site would be still running :-D
So I now need to wait till Jonas get me new shiny backported uWSGI linked against new PHP on Wheezy and till that time I need to explore on how I can sandbox the poller.php script.

Sam Hartman: Debian: Init Interfaces Our Users *and* Free Software

Recently, I ve been watching the Debian Schadenfreude related to init systems. For those who do not know, Debian is trying to decide whether Systemd or Upstart will be used to start software on Debian. There are two other options, although I think Systemd and Upstart are the main contenders. Currently the Technical Committee is considering the issue, although there s some very real chance that the entire project will get dragged through this swamp with the general resolution process. This is one of those discussions that proves that Debian truly is a community: if we didn t have something really strong holding us together we d never put up with something this painful. Between the accusations that our users now know the persecution European pagans faced at the hands of Christians as the Systemd priests drive all before them, a heated discussion of how the Debian voting system interacts with this issue, two failed calls for votes, a discussion of conflicts of interests, and a last-minute discussion of whether the matter had been appropriately brought before the Technical committee (and if so, under what authority), there s certainly schadenfreude to go around if you re into that sort of thing. However, through all this, the Technical Committee has been doing great work understanding the issues; it has been a pleasure to watch their hard work from the side lines. I d like to focus on one key question they ve found: how tightly can other software depend on the init system. Each init system offers some nice features. Upstart has an event triggering model. Systemd manages login sessions and at least in my opinion has a really nice service file format. Can I take advantage of these in my software. If I do, then users of my software might need to use a particular init system. Ian Jackson argues that we should give our users the choice of what init system to run. He reminds us that Debian is a community that supports diversity and diverse goals. We support multiple desktop systems, web browsers, etc. Wherever we can, we support people working on their goals and give developers the opportunity to make it work. When developers succeed, we give our users the opportunity to choose from among the options that have been developed. I think of Ian s argument as an appeal to part of our Social Contract. Clause 4 of the social contract begins:
Our priorities are our users and free software.
We will be guided by the needs of our users and the free software community. We will place their interests first in our priorities. We will support the needs of our users for operation in many different kinds of computing environments.
Ian is right that our users will be served by choice in init systems. However, this trade off needs to be balanced against the needs of the free software community. Diversity is an important goal but it should not come at the price of stagnation and innovation. If I want to avoid using init scripts because they don t provide restart on failure, because they are hard to write correctly, and because they don t provide access to modern security isolation techniques, I should be able to do that. If Systemd service files provide a superior solution, I should be able to work toward using them. If the desire for init system diversity shuts down my ability to find like-minded people who want to take advantage of improvements in init systems and work towards that goal, then we ve significantly damaged Debian as a forum for technical cooperation. Early in my work as a Debian Developer, Anthony Towns taught me an interesting principle. Those who do the work get significant influence over how and what gets done. Ian s right that we need to enable people to work towards choice in init systems. Those like-minded people need to be given a chance to find each other and pursue their goal. However the cost of success should be on the shoulders of those who value choice in init systems. It should not come at the cost of preventing people from depending on improvements in init systems. The best proposal I ve seen for balancing this is to enumerate stable interfaces that people want to use. Things like the logind interface or my favorite the service file interface. It needs to be possible to make another implementation of the interface. The interface needs to be stable enough that a dedicated team could have a chance of catching up with an implementation. At some point during the release cycle such interfaces would need to be frozen. However, I don t think it is reasonable to mandate that there are multiple implementations of the interface, only that there could be. The point is to give people a chance to work towards diversity in init systems, not to force it down peoples throats kicking and screaming until they leave the project or ignore our processes. Steve Langasek and Colin Watson seem to be working towards this. It s possible there s another approach besides interfaces. My main concern is the same as Ian s: maintain Debian as a forum for people to pursue their goals and work together. I suspect we see the conflict in these goals differently. I hope that we as a project can explore this conflict and figure out where we have common ground. I commit to exploring how we can work towards init choice in my frameworks; I ask those who prioritize init choice to commit to explore how we can take advantage of new features in their framework.

John Goerzen: Migrated from Hetzner to OVH hosting

Since August 2011, my sites such as complete.org have been running on a Xen-backed virtual private server (VPS) at Hetzner Online, based in Germany. I had what they called their VQ19 package, which included 2GB RAM, 80GB HDD, 100Mb NIC and 4TB transfer. Unlike many other VPS hosts, I never had performance problems. However, I did sometimes have hardware problems with the host, and it could take hours to resolve. Their tech support only works business hours German time, which was also a problem. Meanwhile, OVH, a large European hosting company, recently opened a datacenter in Canada. Although they no longer offer their value-line Kimsufi dedicated servers there starting at $11.50/mo they do offer their midrange SoYouStart servers there. $50/mo gets a person a 4-core 3.2GHz Xeon server with 32GB RAM, 2x2TB SATA HDD, 200Mbps bandwidth. Not bad at all! The Kimsufi options are still good for lower-end needs as well. I signed up for one of the SoYouStart servers. I ve been pleased with my choice to migrate, and at the possibilities that having hardware like that at my disposal open up, but it is not without its downside. The primary downside is lack of any kind of KVM console. If the server doesn t boot, I can t see the Grub error message (or whatever) behind it. They do provide hardware support and automatic technician dispatching when the server isn t pingable, but they state they have no KVM access at all. They support many OS flavors, and have a premade image for them, but there is no using a custom ISO to install; if you want ZFS on Linux, for instance, you can t very easily build it into root. My server was promised within 72 hours, but delivered much quicker: within about 1. I had two times they said they had to replace a motherboard within the first day; once they did it in 30 minutes, and the other took them 2.5 hours for some reason. They do have phone support, which answers almost immediately, but the people there are not the people actually in the datacenter. It was frustrating with a server down for hours and nobody really commenting on what was going on. The server performs quite well, and after the initial issues, I ve been happy. I was initially planning an all-ZFS installation. SoYouStart does offer a rescue environment, but it doesn t support ZFS, so I figured I better stick with an ext4 root at least. The default Debian install uses RAID1 on md-raid, with a 20GB root partition and the rest of the 2TB drive in /home, and then a swap partition on each drive (mysteriously NOT in the RAID!) So I broke the mirror on /home and converted those into the two legs of a mirrored vdev for a zpool. I run all of the real work inside KVM VMs, so that should minimize the number of times I have to do anything to the root filesystem that could cause trouble. SoYouStart includes 100GB of space on a separate FTP server for backup purposes. I have scripts that upload nightly tarballs of the root filesystem, plus full zfs send streams of everything else. Every hour, it uploads an incremental zfs send stream as well. This all works quite nicely; even if the machine is a complete loss, I d never lose more than an hour s work, and could restore it completely from a rescue environment. Very nice! I ll write more in a few days about the ZFS setup I m using, and some KVM discoveries as well.

7 February 2014

Mario Lang: Solara: Fun with accessible RPGs on iOS

Solara is a fun turn-based fantasy strategy game for iOS which works perfectly well with VoiceOver. AppleVis has a review and all the details like a link into the AppStore. You build your castle level by level, and train your heroes to be stronger in battle. While your heroes are at battle, you can not control any real-time aspect of a battle. The combination of your heroes and their individual strengths (levels) is essential to determine if you can win a battle. So it sort of resembles rolling dice. At the beginning of a battle, you can choose which of your various heroes you'd like to send. You normally have four slots to fill, in earlier levels of Solara even less (to ease you into game play). Fighting is done in two different parts of the game. You need to use your heroes to solve quests, which will be rewarded by experience points and gold, which in turn will increase your playing level. From level 10 on, you will also be able to play in the arena against other player of solara. These arena fights are rewarded with arena points which in turn determine your place in an arena turnament. These turnaments usually last a few days. At the end, the top places are rewarded with gold. The way solara is structured makes it a perfect fit for VoiceOver. There are no real-time gesture problems to solve. It is basically an interactive text adventure with a fantasy theme and a weighted skills mechanism for determining the outcome of a fight. And luckily, the authors of solara realized this and did everything necessary to make this game really nicely playable with VoiceOver. My personal comment: I initially felt its implemented nicely, but playing it would be a waste of time. Now, a few months later, I am at level 39, and have seen hints that there are 50 levels in total. While I am not really hooked, I guess I will finish it eventually :-)

6 February 2014

Andrew Pollock: [life] Day 11, Short day, BJJ again and Science Friday prep

Today was a short day. Sarah is going down to Canberra for the weekend with Zoe, and wanted to pick her up after her Brazilian Jiu Jitsu class. We got away a bit late on the bike and got to the class maybe 5 minutes late, but it was just us anyway, so it was all good. I know now it's best to make sure we're out the door no later than 9am. The class was just as good as last time, and Sarah arrived halfway through to watch, so she got to see it for herself. I biked home the scenic way again, but made a slight wrong turn somewhere, so I popped out further up Wynnum Road than I wanted to be, and so had to backtrack a bit to cross it. After I got home, I decided to pop out to Jaycar again and get some more bits and pieces and do some grocery shopping. I'd made some conductive play dough on Wednesday afternoon while Megan was here, and the girls just used it as standard play dough. Today I made some non-conductive play dough to go with it, and tested it all out. At Jaycar I bought a battery pack and some banana clips, so I soldered everything together and it works nicely. We can just jam the banana clips into the appropriate bits of play dough and make LEDs light up. I bought a bunch of LEDs from Jaycar as well. So we should be all set for next Friday. After that I spent the rest of the afternoon tinkering around with my electronics project. There's nothing like learning by trial and error.

Johannes Schauer: botch updates

My last update about ongoing development of botch, the bootstrap/build ordering tool chain, was three months ago with the announcement of bootstrap.debian.net. Since then a number of things happened, so I thought an update was due.

New graphs for port metrics By default, a dependency graph is created by arbitrarily choosing an installation set for binary package installation or source package compilation. Installation set vertices and source vertices are connected according to this arbitrary selection. Niels Thykier approached me at Debconf13 about the possibility of using this graph to create a metric which would be able to tell for each source package, how many other source packages would become uncompilable or how many binary packages would become uninstallable, if that source package was removed from the archive. This could help deciding about the importance of packages. More about this can be found at the thread on debian-devel. For botch, this meant that two new graph graphs can now be generated. Instead of picking an arbitrary installation set for compiling a source package or installing a binary package, botch can now create a minimum graph which is created by letting dose3 calculate strong dependencies and a maximum graph by using the dependency closure.

Build profile syntax in dpkg With dpkg 1.17.2 we now have experimental build profile support in unstable. The syntax which ended up being added was: 
Build-Depends: large (>= 1.0), small <!profile.stage1>
But until packages with that syntax can hit the archive, a few more tools need to understand the syntax. The patch we have for sbuild is very simple because sbuild relies on libdpkg for dependency parsing. We have a patch for apt too, but we have to rebase it for the current apt version and have to adapt it so that it works exactly like the functionality dpkg implements. But before we can do that we have to decide how to handle mixed positive and negative qualifiers or whether to remove this feature altogether because it causes too much confusion. The relevant thread on debian-dpkg starts here.

Update to latest dose3 git Botch heavily depends on libdose3 and unfortunately requires features which are only available in the current git HEAD. The latest version packaged in Debian is 3.1.3 from October 2012. Unfortunately the current dose3 git HEAD also relies on unreleased features from libcudf. On top of that, the GraphML output of the latest ocamlgraph version (1.8.3) is also broken and only fixed in the git. For now everything is set up as git submodules but this is the major blocker preventing any packaging of botch as a Debian package. Hopefully new releases will be done soon for all involved components.

Writing and reading GraphML Botch is a collection of several utilities which are connected together in a shell script. The advantage of this is, that one does not need to understand or hack the OCaml code to use botch for different purposes. In theory it also allows to insert 3rd party tools into a pipe to further modify the data. Until recently this ability was seriously hampered by the fact that many botch tools communicated with each other through marshaled OCaml binary files which prevent everything which is not written in OCaml from modifying them. The data that was passed around like this were the dependency graphs and I initially implemented it like that because I didnt want to write a GraphML parser. I now ended up writing an xmlm based GraphML parser so as of now, botch only reads and writes ASCII text files in XML (for the graphs) and in rfc822 packages format (for Packages and Sources files) which can both easily be modified by 3rd party tools. The ./tools directory contains many Python scripts using the networkx module to modify GraphML and the apt_pkg module to modify rfc822 files.

Splitting of tools To further increase the ability to modify program execution without having to know OCaml, I split up some big tools into multiple smaller ones. Some of the smaller tools are now even written in Python which is probably much more hackable for the general crowd. I converted those tools to Python which did not need any dose3 functionality and which were simple enough so that writing them didnt take much time. I could convert more tools but that might introduce bugs and takes time which I currently dont have much of (who does?).

Gzip instead of bz2 Since around January 14, snapshot.debian.org doesnt offer bzip2 compressed Packages and Sources files anymore but uses xz instead. This is awesome for must purposes but unfortunately I had to discover that there exist no OCaml bindings for libxz. Thus, botch is now using gzip instead of bz2 until either myself or anybody else finds some time to write a libxz OCaml binding.

Self hosting Fedora Paul Wise made me aware of Harald Hoyer's attempts to bootstrap Fedora. I reproduced his steps for the Debian dependency graph and it turns out that they are a little bit bigger. I'm exchanging emails with Harald Hoyer because it might not be too hard to use botch for rpm based distributions as well because dose3 supports rpm. The article also made me aware of the tred tool which is part of graphviz and allows to calculate the transitive reduction of a graph. This can help making horrible situations much better.

Dose3 bugs I planned to generate such simplified graphs for the neighborhood of each source package on bootstrap.debian.net but then binutils stopped building binutils-gold and instead provided binutils-gold while libc6-dev breaks binutils-gold (<< 2.20.1-11). This unfortunately triggered a dose3 bug and thus bootstrap.debian.net will not generate any new results until this is fixed in dose3. Another dose3 bug affects packages which Conflicts/Replaces/Provides:bar while bar is fully virtual and are Multi-Arch:same. Binaries of different architecture with this property can currently not be co-installed with dose3. Unfortunately linux-libc-dev has this property and thus botch cannot be used to analyze cross builds until that bug is fixed in dose3. I hope I get some free time soon to be able to look at these dose3 issues myself.

More documentation Since I started to like the current set of tools and how they work together I ended up writing over 2600 words of documentation in the past few days. You can start setting up and running botch by reading the first steps and get more detailed information by reading about the 28 tools that botch makes use of as of now. All existing articles, thesis and talks are linked from the wiki home.

Tim Retout: FOSDEM 2014

I attended FOSDEM this year. As always, it was very busy, and the Brussels transport system was as confusing as ever. This time it was nice to accidentally bump into so many people I know from years past. Lunar's talk on reproducible builds of Debian packages was interesting - being able to independently verify that a particular binary package was built from a particular source package is quite attractive. Also Mailpile declared an alpha release. The bit in the talk that I didn't know already was the description of exactly how the search function works. Seeing how they integrate GPG into the contacts/compose features brought home how lacking most (all?) other mail clients are when it comes to usable encryption. On Sunday afternoon I managed to grab a seat in the Go devroom, to hear about crazy things like how YouTube are putting a daemon called Vitess in front of MySQL to abstract away sharding (at the cost of some transaction guarantees). You would have thought Google would already have a scalable data store of some sort? Other bits I remember: Michael Meeks talking about GPU-enabling spreadsheet formulae calculations. And hearing Wietse Venema talk about Postfix was pretty awesome.

Andrew Pollock: [life] Day 10, Playgroup again

I was bracing myself for a rough night, but Zoe slept through until almost 6am. Today was playgroup again, so we just took a slow start to the day and headed out on the bike a bit after 9am. There was a mix of new faces and people from last week. I packed a lunchbox this time instead of relying on the snack collection in my "Dad Bag" backpack. We worked on trying to approach other girls to get them to play. Unfortunately they're all that little bit younger and more shy, so we haven't had a huge success. Zoe's been pretty happy just playing with the various toys and playing with me. Parachute time was a big hit again. I think there were about 25 adults there (I was the only Dad this week), with many with multiple children. The youngest baby I saw was a 7 week old. Zoe had been asking me what a honeydew was, so after Playgroup we dropped by the Hawthorne Garage and grabbed half a honeydew and a couple of tomatoes and I made some fritters for lunch with leftover corned beef from dinner the other night. Zoe wasn't too impressed with them. I think she didn't like the diced tomato. She declared that the honeydew was just like rockmelon. We read a story in the hammock and she went down pretty easily for nap after that. I'd just started doing the dishes when my mate Steve gave me a call. He was in the area, and he dropped in. That worked out well, because I'd been meaning to pick his brain about my little electronics project. We geeked it up for a bit. Zoe's mini lab coat arrived in the mail, and when she finally woke up after 3pm, I asked her if she wanted to go out to Bunnings to get a pair of kids-sized safety glasses, but she said she wanted to do something "fun" instead. When I asked her what she wanted to do, she initially wanted to go to Wet and Wild, but that's an all day outing and we're going next Wednesday, so she settled on going to Colmslie pool again instead. So we quickly got organised and jumped on the bike and biked over to the pool. We started out in the kid's play area, which had a couple of small slides and some fountains and whatnot, and then we gravitated over to the 50 metre pool, where Zoe practised her diving for a bit and generally quite confidently navigated the 1.5m deep end of the pool. It was good, because at least I could just stand up instead of tread water while she practised diving in off the blocks. We had a post-swim ice cream and biked home, and then I whipped up dinner and it was bathtime and bedtime. I'm going to have to start tracking my weekly kilometres with Zoe on the bike, because I think it'll be a reasonable distance.

5 February 2014

Thomas Koch: shared infrastructure coop

I'm working in a very small web agency with 4 employees, one of them part time and our boss who doesn't do programming. It shouldn't come as a surprise, that our development infrastructure is not perfect. We have many ideas and dreams how we could improve it, but not the time.
Now we have two obvious choices: Either we just do nothing or we buy services from specialized vendors like github, atlassian, travis-ci, heroku, google and others.
Doing nothing does not work for me. But just buying all this stuff doesn't please me either. We'd depend on proprietary software, lock-in effects or one-size-fits-all offerings.
Another option would be to find other small web shops like us, form a cooperative and share essential services. There are thousands of web shops in the same situation like us and we all need the same things: As I said, all of the above is available as commercial offerings. But I'd prefer the following to be satisfied:
Does something like that already exists? There already is the German cooperative hostsharing which is kind of similar but does provide mainly hosting, not services. But I'll ask them next after writing this blog post.
Is your company interested in joining such an effort? Does it sound silly?




Next.

Previous.