Search Results: "tar"

11 February 2014

Andrew Pollock: [life] Day 14, Kindergarten and a make up swim class

Zoe woke up pretty early today, at around 5:30am. I also officially became a Xoogler at some point overnight. I used this transition as the opportunity to move from my Nexus 4 to my Nexus 5, so I could have a clean start on a phone that had never had work credentials on it. I started that process before breakfast, and mostly had a usable phone by the time we biked to Kindergarten. Zoe was a bit more like her usual clingy self at drop off today, so I spent a bit longer getting her settled. I got home and pretty much spent the rest of the day sorting out my phone. I can say many good things about Android, but seamless device migration isn't one of them. I'll probably factory reset my Nexus 4 tomorrow. I biked back to Kindergarten to pick up Zoe, and sure enough, she was fast asleep again. It turned out that she'd gone down about 30 minutes later today, so she was deeply asleep, and didn't take too kindly to being woken up. She was very sad, and it took forever just to leave the Kindergarten. We were well and truly the last ones out. Then she saw Megan having a trial tennis lesson at the tennis courts on the way to the bike, and wanted to do that too, and we burned some more time, while I tried to talk her out of that. Zoe had a make up swim class today to cover the one she'd missed on Saturday when she went down to Canberra with Sarah. That was at 3:15pm. We were a tad late for it. Weekday swim classes are much quieter than the weekend ones, so I used the opportunity to talk to one of the swim school staff about progressing Zoe to the next level. They conducted an assessment on the spot, and graduated her! She's moving from "Platypus" to "Octopus", starting next week. The class time options were pretty crappy though, something like 11:45am on Saturday, or a late afternoon weekday. We've opted for a weekday class after Kindergarten for the rest of this quarter and will try and get an earlier Saturday class next quarter. There's a reasonable gap between the end of Kindergarten and the start of swim class, so I'll have to figure out how best to use that time. I might make our library visit in between, even though it means a bit of back and forth. The upside is Saturday mornings are now wide open. We had a celebratory ice cream when we got home from swim class, and went out for a babyccino after dinner. I think it's important to celebrate achievements. I managed to get Zoe to bed slightly early tonight in light of her reduced nap time, but she just woke up briefly at 10:30pm. She seems to be particularly sensitive to mosquitoes, and has a bite on her forehead, right at the hairline, and it was giving her some grief. I gave her some antihistamine and she's gone back to bed fairly easily. Fingers crossed she sleeps okay for the rest of the night.

10 February 2014

Matthew Garrett: IBM's remote firmware configuration protocol

I spent last week looking into the firmware configuration protocol used on current IBM system X servers. IBM provide a tool called ASU for configuring firmware settings, either in-band (ie, running on the machine you want to reconfigure) or out of band (ie, running on a remote computer and communicating with the baseboard management controller - IMM in IBM-speak). I'm not a fan of using vendor binaries for this kind of thing. They tend to be large (ASU is a 20MB executable) and difficult to script, so I'm gradually reimplementing them in Python. Doing that was fairly straightforward for Dell and Cisco, both of whom document their configuration protocol. IBM, on the other hand, don't. My first plan was to just look at the wire protocol, but it turns out that it's using IPMI 2.0 to communicate and that means that the traffic is encrypted. So, obviously, time to spend a while in gdb.

The most important lesson I learned last time I did this was "Check whether the vendor tool has a debug option". ASU didn't mention one in its help text and there was no sign of a getopt string, so this took a little longer - but nm helpfully showed a function called DebugLog and setting a breakpoint on that in gdb showed it being called a bunch of the time, so woo. Stepping through the function revealed that it was checking the value of a variable, and looking for other references to that variable in objdump revealed a -l argument. Setting that to 255 gave me rather a lot of output. Nearby was a reference to --showsptraffic, and passing that as well gave me output like:

BMC Sent: 2e 90 4d 4f 00 06 63 6f 6e 66 69 67 2e 65 66 69
BMC Recv: 00 4d 4f 00 21 9e 00 00

which was extremely promising. 2e is IPMI-speak for an OEM specific command, which seemed pretty plausible. 4d 4f 00 is 20301 in decimal, which is the IANA enterprise number for IBM's system X division (this is required by the IPMI spec, it wasn't an inspired piece of deduction on my behalf). 63 6f 6e 66 69 67 2e 65 66 69 is ASCII for config.xml. That left 90 and 06 to figure out. 90 appeared in all of the traces, so it appears to be the command to indicate that the remaining data is destined for the IMM. The prior debug output indicated that we were in the QuerySize function, so 06 presumably means query the size. And this is supported by the response - 00 (a status code), 4d 4f 00 (the IANA enterprise number again, to indicate that the responding device is speaking the same protocol as you) and 21 9e 00 00 - or, in rational endianness, 40481, an entirely plausible size.

Once I'd got that far the rest started falling into place fairly quickly. 01 rather than 06 indicated a file open request, returning a four byte file handle of some description. 02 was read, 03 was write and 05 was close. Hacking this together with pyghmi meant I could open a file and read it. Success!

Well, kind of. I was getting back a large blob of binary. The debug trace showed calls to an EfiDecompress function, so on a whim I tried just decompressing it using the standard UEFI compression format. Shockingly, it worked and now I had a 345K XML blob and presumably many more problems than I'd previously had. Parsing the XML was fairly straightforward, and now I could retrieve the full set of config options, along with the default, current and possible values for them.

Making changes was pretty much just the reverse of this. A small XML blob containing the new values is compressed and written to asu_update.efi. One of the elements is a random identifier, which will then appear in another file called config_log along with a status. Just keep re-reading this until the value changes to CM_DONE and we're good.

The in-band configuration appears to be identical, but rather than sending commands over the wire they're send through the system's IPMI controller directly. Yes, this does mean that it's sending compressed XML through a single io port. Yes, I'd rather be drinking.

I've documented the protocol here and hope to be able to release this code before too long - right now I'm trying to nail down the interface a little, but it's broadly working.

comment count unavailable comments

Mario Lang: Neurofunkcasts

I have always loved Drum and Bass. In 2013 I rediscovered my love for Darkstep and Neurofunk, and found that these genres have developed quite a lot in the recent years. Some labels like Black Sun Empire and Evol Intent produce mixes/sets on a regular basis as podcasts these days. This article aggregates some neurofunk podcasts I like a lot, most recent first. Enjoy 33 hours and 57 minutes of fun with dark and energizing beats. Thanks to BSE Contrax and Evol Intent for providing such high quality sets. You can also see the Python source for the program that was used to generate this page.

Andrew Pollock: [life] Day 13, early start, Kindergarten and messing around with a hot glue gun

I had an early start today. I set my alarm for 5:30am, and finally got going a bit after 6am, and knocked out a 10km run. It was my first 10km in about a month. It wasn't pretty, but I was looking for accomplishment, not achievement. Sarah's doing a work training course that had an early start, so she asked me if she could drop Zoe to me early and have me take care of Kindergarten drop off for her. Zoe arrived at around 8am, and I put some sunscreen on her and we jumped on the bike. I had one of the smoothest drop offs I've had since we've been back in Australia. Megan was already there, and after we got settled, the two girls started building with some large wooden blocks, and Zoe pretty much gave me a hug and a kiss and sent me on my way. She didn't require any handing off to a teacher, wasn't clingy or anything. It was wonderful to see. Today was supposed to be my Debian day, but I've got something like 3 tax returns to do for two different countries, so I got stuck into collating records instead. After lunch I allowed myself to get sidetracked with trying to figure out the Google Drive API, because I'm trying to directly scan documents to Google Drive. Then it was time to do the Kindergarten pick up. Both girls were fast asleep when I got there. I tried rousing Zoe without much success, so I went to try and wake up Megan. She was also pretty heavily sleeping, so I left her with one of the teachers to try and wake her up and returned to Zoe. She took a lot of work to wake up, but woke up happy. I was expecting a couple of packages today, and the first one was being delivered by Australia Post as we got home, so that was pretty good timing. The second one was delivered while we were home, so that worked out pretty well. For this afternoon's activity with the girls, I thought we'd make peg dolls. I've been stockpiling craft materials for almost as long as I've been back in Australia. I bought the big IKEA EXPEDIT 5 x 5 shelf, and have the bottom two rows set up as Zoe-accessible craft materials, reminiscent of the "bin wall" that the TechShop had. I've been getting stuff from Riot!, but I've also found that there's lots of good stuff to be had for cheap from Overflow. So today's activity I think came from Riot! It was your bog standard wooden clothes pin, but the kit included a conical wooden "sleeve" that I guess was supposed to be a "dress". I thought I'd take things to the next level by hot-gluing a styrofoam ball on the top of the peg, and then hot-gluing eyes and a pom-pom nose on. The girls decided on their own to glue orange cellophane for hair, and got me to draw on eyebrows and a mouth. I had a couple of sheets of felt, so I cut holes in the middle of them and used the conical wooden sleeve to hold the felt dress in place. Pipe cleaners made the arms and they were finished. The girls seemed to enjoy making and playing with them. It was the first time I've used the hot glue gun, and I wasn't sure how it was going to go. I explained to the girls that the tip was hot and to stay away from it, and let them use it under close supervision. Unfortunately Zoe learned the hard way that the glue is also very hot after it's been applied to the surface that it's going to be used on. Hot glue was definitely the way to go for a quick project where we didn't want to be sitting around waiting for PVA glue to dry. After that, the girls watched a DVD for a little while and then Megan's Dad arrived to pick her up. I started dinner a bit late, I should have started it while the girls were watching TV, so we didn't sit down to eat until later than usual. Bedtime went pretty smoothly, but also ran a bit late. Here's hoping Zoe sleeps well tonight.

9 February 2014

Erich Schubert: Minimizing usage of debian-multimedia packages

How to reduce your usage of debian-multimedia packages: As you might be aware, the "deb-multimedia" packages have seen their share of chaos. Originally, they were named "debian-multimedia", but it was then decided that they should better be called "deb-multimedia" as they are not an official part of Debian. The old domain was then grabbed by cybersquatters when it expired. While a number of packages remain indistributable for Debian due to legal issues - such as decrypting DVDs - and thus "DMO" remains useful to many desktop users, please note that for many of the packages, a reasonable version exists within Debian "main". So here is a way to prevent automatically upgrading to DMO versions of packages that also exist in Debian main: We will use a configuration option known as apt pinning. We will modify the priority of DMO package to below 100, which means they will not be automatically upgraded to; but they can be installed when e.g. no version of this package exists within Debian main. I.e. the packages can be easily installed, but it will prefer using the official versions. For this we need to create a file I named /etc/apt/preferences.d/unprefer-dmo with the contents: 
Package: *
Pin: release o=Unofficial Multimedia Packages
Pin-Priority: 123
As long as the DMO archive doesn't rename itself, this pin will work; it will continue to work if you use a mirror of the archive, as it is not tied to the URL. It will not downgrade packages for you. If you want to downgrade, this will be a trickier process. You can use aptitude for this, and start with a filter (l key, as in "limit") of ?narrow(~i,~Vdmo). This will only show packages installed where the version number contains dmo. You can now patrol this list, enter the detail view of each package, and check the version list at the end for a non-dmo version. I cannot claim this will be an easy process. You'll probably have a lot of conflicts to clear up, due to different libavcodec* API versions. If I recall correctly, I opted to uninstall some dmo packages such as ffmpeg that I do not really need. In other cases, the naming is slightly different: Debian main has handbrake, while dmo has handbrake-gtk. A simple approach is to consider uninstalling all of these packages. Then reinstall as needed; since installing Debian packages is trivial, it does not hurt to deinstall something, does it? When reinstalling, Debian packages will be preferred over DMO packages. I prefer to use DFSG-free software wherever possible. And as long as I can watch my favorite series (Tatort, started in 1970 and airing episode 900 this month) and the occasional DVD movie, I have all I need. An even more powerful aptitude filter to review installed non-Debian software is ?narrow(~i,!~ODebian), or equivalently ?narrow(?installed,?not(?origin(Debian))). This will list all package versions, which cannot be installed from Debian sources. In particular, this includes versions that are no longer available (they may have been removed because of security issues!), software that has been installed manually from .deb files, or any other non-Debian source such as Google or DMO. (You should check the output of aptitude policy that no source claims to be o=Debian that isn't Debian though). This filter is a good health check for your system. Debian packages receive a good amount of attention with respect to packaging quality, maintainability, security and all of these aspects. Third party packages usually failed the Debian quality check in at least one respect. For example, Sage isn't in Debian yet; mostly because it has so many dependencies, that making a reliable and upgradeable installation is a pain. Similarly, there is no Hadoop in Debian yet. If you look at Hadoop packaging efforts such as Apache Bigtop, they do not live up to Debian quality yet. In particular, the packages have high redundancy, and re-include various .jar copies instead of sharing access to an independently packaged dependency. If a security issue arises with any of these jars, all the Hadoop packages will need to be rebuilt. As you can see, it is usually not because of Debian being too strict or too conservative about free software when software is not packaged. More often than not, it's because the software in question itself currently does not live up to Debian packaging quality requirements. And of course sometimes because there is too little community backing the software in question, that would improve the packaging. If you want your software to be included in Debian, try to make it packaging friendly. Don't force-include copies of other software, for example. Allow the use of system-installed libraries where possible, and provide upgrade paths and backwards compatibility. Make dependencies optional, so that your software can be pacakged even if a dependency is not yet available - and does not have to be removed if the dependency becomes a security risk. Maybe learn how to package yourself, too. This will make it easier for you to push new features and bug fixes to your userbase: instead of requiring your users to manually upgrade, make your software packaging friendly. Then your users will be auto-upgraded by their operating system to the latest version. Update: If you are using APT::Default-Release or other pins, these may override above pin. You may need to use apt-cache policy to find out if the pins are in effect, and rewrite your default-release pin into e.g. 
Package: *
Pin: release a=testing,o=Debian
Pin-Priority: 912
for debugging, use a different priority for each pin, so you can see which one is in effect. Notice the o=Debian in this pin, which makes it apply only to Debian repositories.

Neil Williams: Debian ARMMP in LAVA using iMX53

The home lab is my LAVA development environment and I ve recently got two iMX53 Quick Start Boards working with a typical LAVA master image based on a Linaro build of Linux 3.1.0 for mx5 with a Ubuntu Oneiric Ocelot 11.10 rootfs: 
 3.1.0-1002-linaro-lt-mx5 (buildd@hubbard) (gcc version 4.6.1 (Ubuntu/Linaro 4.6.1-9ubuntu3) ) #13-Ubuntu PREEMPT Fri Dec 16 01:21:07 UTC 2011
As part of my Debian work, it was clearly time to look at a current, Debian, kernel and rootfs and as I m developing and testing on Debian unstable, this would necessarily mean testing the Debian ARMMP (multi-platform) kernel which replaces the mx5 build used in Wheezy. 
Linux version 3.12-1-armmp (debian-kernel@lists.debian.org) (gcc version 4.8.2 (Debian 4.8.2-10) ) #1 SMP Debian 3.12.9-1 (2014-02-01)
I will be scripting the creation of a suitable image for these tests and there are other changes planned in LAVA to make it easier to build suitable images, but it is useful to document just how I worked out how to build the first image. Manual build steps First, I ve already discovered that the u-boot on the iMX53 doesn t like ext4, so the first step was to prepare an ext3 filesystem for the image. With a SATA drive attached, it was also much better to use that than the SD card, at least for generating the image. I m also doing this natively, so I am working inside the booted master image. This is fine as the master is designed to manipulate test images, so the only package I needed to install on the LAVA master image was debootstrap. I had an empty SATA drive to play with for these tests, so first prepare an ext3 filesystem: 
# mkfs.ext3 /dev/sda1
# mkdir /mnt/sata
# mount /dev/sda1 /mnt/sata
# mkdir /mnt/sata/chroots/
Start the debootstrap: 
# apt-get update
# apt-get install debootstrap
# debootstrap --arch=armhf --include=linux-image-armmp \
 --verbose unstable \
 /mnt/sata/chroots/unstable-armhf http://ftp.uk.debian.org/debian
Various actions in this chroot will need proc, so mount it here: 
# chroot /mnt/sata/chroots/unstable-armhf
# mount proc -t proc /proc
# mount devpts -t devpts /dev/pts
# exit
You may also have to edit the apt sources the LAVA master image doesn t have editors installed, so either use echo or download an edited file. I m using: 
deb http://ftp.uk.debian.org/debian sid main
flash-kernel needs changes For the initial tests, I ve got to get this image to boot directly from u-boot, so flash-kernel is going to be needed inside the chroot and to get the iMX53 to work with the ARMMP kernel and Device Tree, flash-kernel will need an update which will mean a patch: 
# chroot /mnt/sata/chroots/unstable-armhf
# apt-get update
# apt-get install patch flash-kernel
# cp /usr/share/flash-kernel/db/all.db /home
# cd /home
# patch -p2
The patch itself goes through a couple of iterations. Initially, it is enough to use: 
 Machine: Freescale MX53 LOCO Board
-Kernel-Flavors: mx5
+Kernel-Flavors: armmp
+DTB-Id: imx53-qsb.dtb
+DTB-Append: yes
Later, once the device has booted with the ARMMP kernel, the Machine Id can be updated to distinguish it from the mx5 flavour (from /proc/cpuinfo) and use the model name from the Device Tree (/proc/device-tree/model): 
diff --git a/db/all.db b/db/all.db
index fab3407..41f6c78 100644
--- a/db/all.db
+++ b/db/all.db
@@ -62,6 +62,18 @@ U-Boot-Initrd-Address: 0x00800000
 Required-Packages: u-boot-tools
 Bootloader-Sets-Incorrect-Root: yes
 
+Machine: Freescale i.MX53 Quick Start Board
+Kernel-Flavors: armmp
+DTB-Id: imx53-qsb.dtb
+DTB-Append-From: 3.12
+Boot-DTB-Path: /boot/dtb
+U-Boot-Kernel-Address: 0x70008000
+U-Boot-Initrd-Address: 0x0
+Boot-Kernel-Path: /boot/uImage
+Boot-Initrd-Path: /boot/uInitrd
+Required-Packages: u-boot-tools
+Bootloader-Sets-Incorrect-Root: no
+
 Machine: Freescale MX53 LOCO Board
 Kernel-Flavors: mx5
 U-Boot-Kernel-Address: 0x70008000
(I will be filing this patch in a bug report against flash-kernel soon.) With that patched, update and run flash-kernel: 
# mv all.db /usr/share/flash-kernel/db/all.db
# flash-kernel
flash-kernel: installing version 3.12-1-armmp
Generating kernel u-boot image... done.
Taking backup of uImage.
Installing new uImage.
Generating initramfs u-boot image... done.
Taking backup of uInitrd.
Installing new uInitrd.
Installing new dtb.
# exit
LAVA overlays This will be a LAVA test image and it needs an overlay if you want a vanilla image, set up a passwd inside the chroot instead. 
# cd /mnt/sata/chroots/unstable-armhf/
# wget --no-check-certificate https://launchpad.net/~linaro-maintainers/+archive/overlay/+files/linaro-overlay-minimal_1112.2_all.deb
# wget --no-check-certificate https://launchpad.net/~linaro-maintainers/+archive/overlay/+files/linaro-overlay_1112.2_all.deb
# chroot /mnt/sata/chroots/unstable-armhf/
# dpkg -i linaro-overlay-minimal_1112.2_all.deb linaro-overlay_1112.2_all.deb
# rm linaro-overlay-minimal_1112.2_all.deb linaro-overlay_1112.2_all.deb
# exit
Changes to allow the chroot to boot Now the chroot needs setting up as a boot image: 
# chroot /mnt/sata/chroots/unstable-armhf/
# echo T0:23:respawn:/sbin/getty -L ttymxc0 115200 vt102 >> ./etc/inittab
# echo auto lo eth0 > ./etc/network/interfaces.d/mx53loco
# echo iface lo inet loopback >> ./etc/network/interfaces.d/mx53loco
# echo iface eth0 inet dhcp >> ./etc/network/interfaces.d/mx53loco
# apt-get clean
# umount /dev/pts
# umount /proc
# exit
Partitioning as a LAVA test image This would be enough as a single partition test image but, currently, LAVA expects a much more hard-wired image. Depending on the history of the device and the need for LAVA to be able to put fresh kernel builds together with a known rootfs, LAVA has used a separate /boot and / partition in the test image for nearly all boards. Standard LAVA test images for many boards (like the iMX53) also have a small unallocated space at the start of the SD card, so until I can get LAVA upstream to handle test images of arbitrary design, I m adapting the image to suit the expectations inside LAVA. Yes, I know but it s better to get something working before spending time fixing things to make it work better. It will be fixed, in time. So I needed to separate out the /boot contents from the rest of the rootfs whilst keeping the chroot itself in a state which I can easily update and use to create other images. 
# cd /mnt/sata/chroots/unstable-armhf/boot/
# tar -czf ../../boot.tar.gz ./*
# cd ..
# tar -czf ../root.tar.gz ./*
Now the test image file and its partitions: 
# dd if=/dev/zero of=/mnt/sata/images/empty/debian.img bs=1M count=1024
# cp /mnt/sata/images/empty/debian.img /mnt/sata/images/debian-unstable-armhf-armmp.img
# losetup /dev/loop0 /mnt/sata/images/debian-unstable-armhf-armmp.img
# parted /dev/loop0 -s unit mb mktable msdos
# parted /dev/loop0 -s unit mb mkpart primary 1 10
# parted /dev/loop0 -s unit mb mkpart primary 11 110
# parted /dev/loop0 -s unit mb mkpart primary 111 1024
I did make the mistake of using kpartx at this stage but there are many areas of confusion when translating the kpartx output to offsets for mount when parted is easier: 
# parted /dev/loop0 unit B -s print
Model:  (file)
Disk /dev/loop0: 1073741824B
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Number  Start       End          Size        Type     File system  Flags
 1      1048576B    10485759B    9437184B    primary
 2      10485760B   110100479B   99614720B   primary
 3      110100480B  1024458751B  914358272B  primary
Use the Start numbers and use losetup to create the loop devices for each
partition 
# losetup -o 10485760 /dev/loop1 /dev/loop0
# losetup -o 110100480 /dev/loop2 /dev/loop0
# mkfs.vfat /dev/loop1
# mkfs.ext3 /dev/loop2
Now clean up the loop mountpoints: 
# losetup -d /dev/loop2
# losetup -d /dev/loop1
# losetup -d /dev/loop0
# losetup -a
losetup -a should return nothing. If it doesn t, investigate the contents of /dev/mapper and use dmsetup remove until losetup -a does report empty. Otherwise the subsequent stages will fail. Now deploy the /boot contents into the empty image: 
# mount -oloop,offset=10485760 /mnt/sata/images/debian-unstable-armhf-armmp.img /mnt/boot/
# pushd /mnt/boot/
# tar -xzf /mnt/sata/chroots/boot.tar.gz
# popd
# sync
# umount /mnt/boot/
and the / contents (removing the duplicate ./boot contents) 
# mount -oloop,offset=110100480 /mnt/sata/images/debian-unstable-armhf-armmp.img /mnt/root/
# pushd /mnt/root
# tar -xzf /mnt/sata/chroots/root.tar.gz
# rm ./boot/*
# popd
# sync
# umount /mnt/root
Check the image 
# mount -oloop,offset=10485760 debian-unstable-armhf-armmp.img /mnt/boot
# ls /mnt/boot
config-3.12-1-armmp  initrd.img-3.12-1-armmp  System.map-3.12-1-armmp  uImage  uInitrd  vmlinuz-3.12-1-armmp
# umount /mnt/boot
# mount -oloop,offset=110100480 debian-unstable-armhf-armmp.img /mnt/root
# ls /mnt/root
bin  boot  dev  etc  home  initrd.img  lib  lost+found  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var  vmlinuz
# ls /mnt/root/bin/auto-serial-console
/mnt/root/bin/auto-serial-console
# umount /mnt/root
# md5sum debian-unstable-armhf-armmp.img
Downloads Yes, I have put that file online, if you are interested. Do read the readme first though. Getting the image off the device 
# ip a
# python -m SimpleHTTPServer 80 2>/dev/null
then wget (just http://, IP address / and the filename), md5sum , finish with Ctrl-C. Depending on setup, it may be quicker to transfer the uncompressed
file over a LAN than to let the device compress it. Your main machine
will do the compression much faster, even with a larger download. (It also helps to not compress the image on the device, you can
test the mount offsets more easily and do check that the image
can be mounted with the offsets from parted.) Results 
# cat /proc/cpuinfo
processor	: 0
model name	: ARMv7 Processor rev 5 (v7l)
Features	: swp half thumb fastmult vfp edsp thumbee neon vfpv3 tls vfpd32 
CPU implementer	: 0x41
CPU architecture: 7
CPU variant	: 0x2
CPU part	: 0xc08
CPU revision	: 5
Hardware	: Freescale i.MX53 (Device Tree Support)
Revision	: 0000
Serial		: 0000000000000000

# uname -a
Linux imx53-02 3.12-1-armmp #1 SMP Debian 3.12.9-1 (2014-02-01) armv7l GNU/Linux

# lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux unstable (sid)
Release:	unstable
Codename:	sid
Next! Yes, there is a lot to do from here on. Then there is the whole issue of actually making this multi-platform. After all, it is far from ideal that a multi-platform kernel package has to have platform-specific operations using flash-kernel at each update. So Grub on ARM is going to be on the list of things to investigate .

Thomas Goirand: Removal of XCP / Xen-API from Jessie and Sid

This is sad, but no choice, due to the lack of upstream support: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=738322 I just hope xenserver-core will work soon in Debian, so we can fix this unfortunate situation.

Jan Dittberner: New server, new blog, new home for portfolio.debian.net

New server The Debian Member Portfolio Service moved to a new server. This is the first service that I installed using Salt states which worked pretty well. Behind the scenes it moved from Apache httpd + mod_wsgi to nginx and uwsgi and from Debian Squeeze versions of the Python modules to the versions in Wheezy.
New blog I started this new blog using the Nikola static site and blog generator. I really like writing this way (with my favourite editor, ReSTructured Text and git) instead of using some awkward browser based GUI. I chose Nikola over other alternatives because it's Python based. I hope that I'll write a bit more often than I did in the past.

8 February 2014

Paul Tagliamonte: Upcoming Talks

Hello, World! I ve got two upcoming talks - one at LibrePlanet 2014 about OpenPGP, and one at PyCon US 2014 (in Canada) on Hy. Slides will be posted as I give the talks. Hope to see folks there!

Vasudev Kamath: Friendica instance on my VPS is down

I started running Friendica instance on my VPS. With help of Jonas Smedegaard I managed to run Friendica in a uWSGI container. The site was running at samsargika.copyninja.info and is no longer accessible. Since VPS itself was running Debian Wheezy I couldn't run uWSGI with PHP support on it. (support for PHP in uWSGI landed after Wheezy). But Jonas was kind enough for me to provide a backported version. Recently Wheezy got a security update for PHP and thats where all the problem started. The backported uwsgi-plugin-php was not recompiled to use security updated PHP and I couldn't upgrade the things. After few days I noticed first freeze on my VPS and I had to reboot the VPS to get it online again. The fact I noticed was uWSGI being killed due to a OOM in syslog but I didn't explore much and consulted Jonas to get a updated uWSGI. But that didn't happen as Jonas himself is facing some problem with uWSGI builds. While again checking with aptitude for upgrade I accidentally confirmed removal of uwsgi-plugin-php for getting security updates :-/. But nothing happened to my running service as upgrade of libs in Debian doesn't cause the restart of all services which are using that lib (desired effect is restart of service but I don't know the side affects involved). Second freeze happened yesterday and on the reboot uwsgi-plugin-php was missing there by taking my Friendica instance down. More closer investigation showed the same OOM but this time I noticed each OOM occurred just after cronjob was running poller.php a script which is actually causing all federation in Friendica. So it was clear there is something wrong either in poller.php or in my setup which was making it eat memory and freeze my VPS. But I also found some stupidity I did during configuration which Jonas also pointed me out.
  1. Installing cronjob inside crontab rather than cron.d
  2. Installing poller.php crontab for root user :-/
I basically violated the basic rule by running a unsafe script as root user, good that script didn't do some crazy stuff. So even though my instance went down I learnt my lessons
  1. Don't ever ever ever run a unsafe script as root that too through cron
  2. Sandbox the unsafe script so it can be killed on time rather than it taking the whole system down.
  3. PHP is not really secure, if it was secure there wouldn't be security updates and atleast my site would be still running :-D
So I now need to wait till Jonas get me new shiny backported uWSGI linked against new PHP on Wheezy and till that time I need to explore on how I can sandbox the poller.php script.

Mario Lang: boost::python and boost::variant

I have unsuccessfully tried to find a solution for the following problem on the internet several times. Now that I have come at least closer to a usable approach, I thought I'd document what I have found so that others trying to achieve a similar thing can use this as a starting point. Boost.Python offers a very nice and flexible way to interface C++ data types with Python. With just a few lines of code, and the proper linker flags, you get a Python importable shared object from your C++ compiler. This can be very productive. However, there is one aspect of C++ data types that I couldn't figure out how to interface with Python, which are C++ discriminated unions, or more specifically, heterogeneous containers. While Python has no problems with containers containing objects of different types, C++ does not make this very easy by default. Usually the problem is solved with a container of pointers to a base class, and various subclasses with virtual functions. However, this approach is not always practical, especially if the different types of objects in an heterogeneous container dont have many things in common. This is where discriminated unions come to the rescue. They basically behave like a normal union in C, but have an additional field which indicates the type of object currently stored in the union. Boost.Variant does exactly that, with a nice visitor interface added on top of it.
Heterogeneous containers in C++ If we put the boost::variant<> template inside a STL container like std::vector<>, the result is a heterogeneous container. For the purpose of illustration, lets implement such a container. The example below is deliberately simple. In reality, the various types allowed in your variant will probably have more fields then just one. 
#include <boost/variant.hpp>
#include <vector>
struct a   int x;  ;
struct b   std::string y;  ;
typedef boost::variant<a, b> variant;
typedef std::vector<variant> vector;
To ease creation of these two types of objects, we are going to write a few factory functions. We are going to wrap them in Python later on. 
variant make_variant()   return variant();  
vector make_vector()   return vector a(), b(), a() ;
Boost.Python Now lets create a Python module which exports the above functionality to Python. 
#include <boost/python/class.hpp>
#include <boost/python/def.hpp>
#include <boost/python/implicit.hpp>
#include <boost/python/init.hpp>
#include <boost/python/module.hpp>
#include <boost/python/object.hpp>
#include <boost/python/suite/indexing/vector_indexing_suite.hpp>
vector_indexing_suite apparently needs operator== defined on the value_type of the container. In our case, this is our boost::variant<a, b> type. Luckily, boost::variant<> already provides operator==. However, that operator== relies on operator== being defined for the underlying types. Since equality comparison is probably useful for other things as well, lets just create operator== for our two classes a and b. 
bool operator==(a const &lhs, a const &rhs)   return lhs.x == rhs.x;  
bool operator==(b const &lhs, b const &rhs)   return lhs.y == rhs.y;
Convert a boost::variant<> to PyObject * Boost.Python needs a way to convert our discriminated union to a Python object. This code relies on Python class definitions being present for all underlying variant types. We will define them later. 
struct variant_to_object : boost::static_visitor<PyObject *>  
  static result_type convert(variant const &v)  
    return apply_visitor(variant_to_object(), v);
   
  template<typename T>
  result_type operator()(T const &t) const  
    return boost::python::incref(boost::python::object(t).ptr());
   
 ;
And finally, lets create our Python module. 
BOOST_PYTHON_MODULE(bpv)  
  using namespace boost::python;
  class_<a>("a", init<a>()).def(init<>()).def_readwrite("x", &a::x);
  class_<b>("b", init<b>()).def(init<>()).def_readwrite("y", &b::y);
  to_python_converter<variant, variant_to_object>();
  implicitly_convertible<a, variant>();
  implicitly_convertible<b, variant>();
  def("make_variant", make_variant);
  class_<vector>("vector").def(vector_indexing_suite<vector, true>());
  def("make_vector", make_vector);
Compiling Lets create a shared object for Python. 
$ g++ -std=c++11 -fPIC -shared $(python-config --includes) -o bpv.so file.cpp -lboost_python
Running We can load the module into Python and see what it does. 
>>> import bpv
>>> variant=bpv.make_variant()
>>> variant
<bpv.a object at 0x7f06bb2130c0>
>>> variant.x
0
>>> variant.x=2
>>> variant.x
2
Nice. We can access the underlying type, and even modify it. Lets see how our heterogeneous container wrapping code behaves. 
>>> vector=bpv.make_vector()
>>> vector
<bpv.vector object at 0x7f20693289d0>
>>> len(vector)
3
>>> list(vector)
[<bpv.a object at 0x7f20693190c0>, <bpv.b object at 0x7f20693193d0>, <bpv.a object at 0x7f2069319440>]
So far, so good. This will at least make it possible to convert heterogeneous containers from C++ to Python, which was my initial goal. Unfortunately, contained objects are not treated as references. Whenever retrieved, we get a copy. So in-place modification does not work. 
>>> vector[0].x
0
>>> vector[0].x=2
>>> vector[0].x
0
However, we can override an existing element with a modified copy. 
>>> e0=vector[0]
>>> type(e0)
<class 'bpv.a'>
>>> e0.x = 2
>>> vector[0] = e0
>>> vector[0].x
2
And we can also use the append and extend methods of Python containers. 
>>> len(vector)
3
>>> vector.extend(vector)
>>> vector.append(bpv.a())
>>> len(vector)
7
>>> len(filter(lambda x: type(x)==bpv.b, vector))
2
>>> len(filter(lambda x: type(x)==bpv.a, vector))
5
>>> map(lambda x: x.x, filter(lambda x: type(x)==bpv.a, vector))
[2, 0, 2, 0, 0]
All that is missing for a perfect world is reference semantics for container elements. If anyone has a hint on how to achieve this, please let me know.

Sam Hartman: Debian: Init Interfaces Our Users *and* Free Software

Recently, I ve been watching the Debian Schadenfreude related to init systems. For those who do not know, Debian is trying to decide whether Systemd or Upstart will be used to start software on Debian. There are two other options, although I think Systemd and Upstart are the main contenders. Currently the Technical Committee is considering the issue, although there s some very real chance that the entire project will get dragged through this swamp with the general resolution process. This is one of those discussions that proves that Debian truly is a community: if we didn t have something really strong holding us together we d never put up with something this painful. Between the accusations that our users now know the persecution European pagans faced at the hands of Christians as the Systemd priests drive all before them, a heated discussion of how the Debian voting system interacts with this issue, two failed calls for votes, a discussion of conflicts of interests, and a last-minute discussion of whether the matter had been appropriately brought before the Technical committee (and if so, under what authority), there s certainly schadenfreude to go around if you re into that sort of thing. However, through all this, the Technical Committee has been doing great work understanding the issues; it has been a pleasure to watch their hard work from the side lines. I d like to focus on one key question they ve found: how tightly can other software depend on the init system. Each init system offers some nice features. Upstart has an event triggering model. Systemd manages login sessions and at least in my opinion has a really nice service file format. Can I take advantage of these in my software. If I do, then users of my software might need to use a particular init system. Ian Jackson argues that we should give our users the choice of what init system to run. He reminds us that Debian is a community that supports diversity and diverse goals. We support multiple desktop systems, web browsers, etc. Wherever we can, we support people working on their goals and give developers the opportunity to make it work. When developers succeed, we give our users the opportunity to choose from among the options that have been developed. I think of Ian s argument as an appeal to part of our Social Contract. Clause 4 of the social contract begins:
Our priorities are our users and free software.
We will be guided by the needs of our users and the free software community. We will place their interests first in our priorities. We will support the needs of our users for operation in many different kinds of computing environments.
Ian is right that our users will be served by choice in init systems. However, this trade off needs to be balanced against the needs of the free software community. Diversity is an important goal but it should not come at the price of stagnation and innovation. If I want to avoid using init scripts because they don t provide restart on failure, because they are hard to write correctly, and because they don t provide access to modern security isolation techniques, I should be able to do that. If Systemd service files provide a superior solution, I should be able to work toward using them. If the desire for init system diversity shuts down my ability to find like-minded people who want to take advantage of improvements in init systems and work towards that goal, then we ve significantly damaged Debian as a forum for technical cooperation. Early in my work as a Debian Developer, Anthony Towns taught me an interesting principle. Those who do the work get significant influence over how and what gets done. Ian s right that we need to enable people to work towards choice in init systems. Those like-minded people need to be given a chance to find each other and pursue their goal. However the cost of success should be on the shoulders of those who value choice in init systems. It should not come at the cost of preventing people from depending on improvements in init systems. The best proposal I ve seen for balancing this is to enumerate stable interfaces that people want to use. Things like the logind interface or my favorite the service file interface. It needs to be possible to make another implementation of the interface. The interface needs to be stable enough that a dedicated team could have a chance of catching up with an implementation. At some point during the release cycle such interfaces would need to be frozen. However, I don t think it is reasonable to mandate that there are multiple implementations of the interface, only that there could be. The point is to give people a chance to work towards diversity in init systems, not to force it down peoples throats kicking and screaming until they leave the project or ignore our processes. Steve Langasek and Colin Watson seem to be working towards this. It s possible there s another approach besides interfaces. My main concern is the same as Ian s: maintain Debian as a forum for people to pursue their goals and work together. I suspect we see the conflict in these goals differently. I hope that we as a project can explore this conflict and figure out where we have common ground. I commit to exploring how we can work towards init choice in my frameworks; I ask those who prioritize init choice to commit to explore how we can take advantage of new features in their framework.

John Goerzen: Migrated from Hetzner to OVH hosting

Since August 2011, my sites such as complete.org have been running on a Xen-backed virtual private server (VPS) at Hetzner Online, based in Germany. I had what they called their VQ19 package, which included 2GB RAM, 80GB HDD, 100Mb NIC and 4TB transfer. Unlike many other VPS hosts, I never had performance problems. However, I did sometimes have hardware problems with the host, and it could take hours to resolve. Their tech support only works business hours German time, which was also a problem. Meanwhile, OVH, a large European hosting company, recently opened a datacenter in Canada. Although they no longer offer their value-line Kimsufi dedicated servers there starting at $11.50/mo they do offer their midrange SoYouStart servers there. $50/mo gets a person a 4-core 3.2GHz Xeon server with 32GB RAM, 2x2TB SATA HDD, 200Mbps bandwidth. Not bad at all! The Kimsufi options are still good for lower-end needs as well. I signed up for one of the SoYouStart servers. I ve been pleased with my choice to migrate, and at the possibilities that having hardware like that at my disposal open up, but it is not without its downside. The primary downside is lack of any kind of KVM console. If the server doesn t boot, I can t see the Grub error message (or whatever) behind it. They do provide hardware support and automatic technician dispatching when the server isn t pingable, but they state they have no KVM access at all. They support many OS flavors, and have a premade image for them, but there is no using a custom ISO to install; if you want ZFS on Linux, for instance, you can t very easily build it into root. My server was promised within 72 hours, but delivered much quicker: within about 1. I had two times they said they had to replace a motherboard within the first day; once they did it in 30 minutes, and the other took them 2.5 hours for some reason. They do have phone support, which answers almost immediately, but the people there are not the people actually in the datacenter. It was frustrating with a server down for hours and nobody really commenting on what was going on. The server performs quite well, and after the initial issues, I ve been happy. I was initially planning an all-ZFS installation. SoYouStart does offer a rescue environment, but it doesn t support ZFS, so I figured I better stick with an ext4 root at least. The default Debian install uses RAID1 on md-raid, with a 20GB root partition and the rest of the 2TB drive in /home, and then a swap partition on each drive (mysteriously NOT in the RAID!) So I broke the mirror on /home and converted those into the two legs of a mirrored vdev for a zpool. I run all of the real work inside KVM VMs, so that should minimize the number of times I have to do anything to the root filesystem that could cause trouble. SoYouStart includes 100GB of space on a separate FTP server for backup purposes. I have scripts that upload nightly tarballs of the root filesystem, plus full zfs send streams of everything else. Every hour, it uploads an incremental zfs send stream as well. This all works quite nicely; even if the machine is a complete loss, I d never lose more than an hour s work, and could restore it completely from a rescue environment. Very nice! I ll write more in a few days about the ZFS setup I m using, and some KVM discoveries as well.

7 February 2014

Jo Shields: Dear Fake Debian Developers, shoo.

Another post about the Valve/Collabora free games thing. This time, the bad bit people trying to scam free games from us. Before I start, I want to make one thing clear there are people who have requested keys who don t meet the criteria, but are honest and legitimate in their requests. This blogspam is not about them at all. If you re in that category, you re not being complained about. So. Some numbers. At time of writing, I ve assigned keys to 279 Debian Developers or Debian Maintainers almost 25% of the total eligible pool of about 1200. I ve denied 22 requests. Of these 10 were polite requests from people who didn t meet the conditions stated (e.g. Ubuntu developers rather than Debian). These folks weren t at all a problem for us, and I explained politely that they didn t meet the terms we had agreed at the time with Valve. No problem at all with those folks. Next, we have the chancers, 7 of them, who pretended to be eligible when they clearly weren t. For example, two people sent me signed requests pointing to their entry on the Debian New Maintainers page when challenged over the key not being in the keyring. The NM page showed that they had registered as non-uploading Debian Contributors a couple of hours previously. A few just claimed I am a DD, here is my signature when they weren t DDs at all. Those requests were also binned.
Papers, Please screenshot - denied entry application

DENIED

And then we move onto the final category. These people tried identity theft, but did a terrible job of it. There were 5 people in this category: 
From: Xxxxxxxx Xxxxxx <xxxxxxxx.xxxxxx@ieee.org>
Subject: free subscription to Debian Developer
8217 A205 5E57 043B 2883 054E 7F55 BB12 A40F 862E
This is not a signature, it s a fingerprint. Amusingly, it s not the fingerprint for the person who sent my mail, but that of Neil McGovern a co-worker at Collabora. Neil assured me he knew how to sign a mail properly, so I shitcanned that entry. 
From: "Xxxxx, Xxxxxxxxx" <x.xxxxx@bbw-bremen.de>
Subject: Incoming!
Hey dude,
I want to have the redemption code you are offering for the Valve Games
mQGiBEVhrscRBAD4M5+qxhZUD67PIz0JeoJ0vB0hsLE6QPV144PLjLZOzHbl4H3N
...snip...
Lz8An1TEmmq7fltTpQ+Y1oWhnE8WhVeQAKCzh3MBoNd4AIGHcVDzv0N0k+bKZQ=3D=3D
=3Du/4R
Wat? Learn to GPG! 
From: Xxxxxx-Xxxx Le Xxxxxxx Xxxx <xx.xxxxxxxxx@gmail.com>
Subject: pass steam
Hey me voila
Merci beaucoup
valve
2069 1DFC C2C9 8C47 9529 84EE 0001 8C22 381A 7594
Like the first one, a fingerprint. This one is for S bastien Villemot. Don t scammers know how to GPG sign? 
From: "Xxxxxxxxx Xxxxxxx" <xxxxxxxx@web.de>
Subject: thanks /DD/Steam gifts us finally something back
0x6864730DF095E5E4
Yet again, a fingerprint. This one is for Marco Nenciarini. I found this request particularly offensive due to the subject line the haughty tone from an identity thief struck me as astonishingly impertinent. Still, when will scammers learn to GPG? 
From: Sven Hoexter <svenhoexter@gmail.com>
Subject: Valve produced games
I'm would like to get the valve produced games
My keyring: 0xA6DC24D9DA2493D1 Sven Hoexter <hoexter> sig:6
Easily the best scam effort, since this is the only one which both a) registered an email address under the name of a DD, and b) used a fingerprint which actually corresponds to that human. Sadly for the scammer, I m a suspicious kind of person, so my instinct was to verify the claim via IRC. 
31-01-2014 16:52:48 > directhex: Hoaxter, have you started using gmail without updating your GPG key? (note: significantly more likely is someone trying to steal your identity a little to steal valve keys from collabora)
31-01-2014 16:54:51 < Hoaxter!~sh@duckpond6.stormbind.net: directhex: I do not use any Google services and did not change my key
So yeah. Nice try, scammer. I m not listing, in all of this, the mails which Neil received from people who didn t actually read his mail to d-d-a. I m also not listing a story which I ve only heard second ha actually no, this one is too good not to share. Someone went onto db.debian.org, did a search for every DD in France, and emailed every Jabber JID (since they look like email addresses) asking them to forward unwanted keys. All in all, the number of evildoers is quite low, relative to the number of legitimate claims 12 baddies to 279 legitimate keys issued. But still, this is why the whole key issuing thing has been taking me so long and why I have the convoluted signature-based validation system in place. Enjoy your keys, all 279 of you (or more by the time some of you read this). The offer has no explicit expiry on it Valve will keep issuing keys as long as there is reason to, and Collabora will continue to administer their allocation as long as they remain our clients. It s a joint gift to the community thousands of dollars worth of games from Valve, and a significant amount of my time to administer them from Collabora.

6 February 2014

Neil Williams: iMX53 in the home lab

The iMX53 Quick Start Board has been a mainstay of the Debian armhf port and has been part of LAVA until recently. It s a natural choice for seeing how LAVA and Debian can work together more closely. So, in the first steps to getting my home lab running, I set about restoring two of these boards to LAVA support. I will be looking into each of the possible boot methods with these boards, but the way to start is with a LAVA master image on an SD card. With the pause in iMX53 development within LAVA, this raises a couple of issues: LAVA master image on iMX53 The LAVA master image is more recent and can fix these problems. See lava-image-creation preparing a master image
$ git clone git://git.linaro.org/lava/lava-master-image-scripts.git
$ cd lava-master-image-scripts
Use patch -p1 to apply this patch (apologies for the line wrapping in the blog): 
diff --git a/lava-create-master b/lava-create-master
index 2e1d0e8..a1e6131 100755
--- a/lava-create-master
+++ b/lava-create-master
@@ -366,7 +366,7 @@ make_master()  
         verbose " * Building vanilla image with linaro-media-create..."
         if ! linaro-media-create \
             --dev $LMC_DEV \
-            --rootfs ext4 \
+            --rootfs ext3 \
             --image-file $CACHE_DIR/pristine-images/$dev-vanilla.img \
             --image-size 1G \
             --align-boot-part \
$ sudo ./lava-create-master imx53 Without the patch applied, I simply removed the SD card, mounted the third partition on my main machine and created a tarball of the contents: 
$ sudo mount /dev/mmcblk0p3 /mnt/
$ pushd /mnt
$ sudo tar -czf /tmp/imx53-rootfs.tgz ./*
$ popd
$ sudo umount /mnt/
Reformat p3 to ext3 (or ext2 if you prefer) 
$ sudo mkfs.ext3 /dev/mmcblk0p3
Put the rootfs back onto the replacement ext3 partition. 
$ sudo mount /dev/mmcblk0p3 /mnt/
$ pushd /mnt
$ sudo tar -xzvf /tmp/imx53-rootfs.tgz
$ popd
$ sudo sync
$ sudo umount /mnt/
The bootscript is buggy, so run: 
> run loaduimage
> run mmcboot
Then fix bootscript or you won t be able to reboot: Simplest way may be to set u-boot to ignore boot.scr 
> setenv script
> saveenv
> reset
Device configuration in LAVA I found screen to be a bit awkward as a serial connection, (use Ctrl-A : quit) to quit. I tried autologin for the lucid image but that was not needed for the master image which uses the linaro overlay. The PDU commands are for my own setup. The daemon (the machine running the pdu daemon scripts), hostname (the hostname or IP of the PDU itself) and the port numbers will change between labs. Again, the blog has changed the line endings. 
device_type = mx53loco
hostname = imx53-01
connection_command = telnet hobbes 4002
#login_prompt = lucid-desktop login:
#password_prompt = Password:
#username = lucid
#password = lucid
hard_reset_command = /usr/bin/pduclient --daemon localhost --hostname pdu --command reboot --port 05
power_off_cmd = /usr/bin/pduclient --daemon localhost --hostname pdu --command off --port 05
power_on_cmd = /usr/bin/pduclient --daemon localhost --hostname pdu --command on --port 05
testboot_offset = 3
Critical testboot_offset is currently undocumented. Without it, you will see: 
 ** Partition 4 not valid on device 0 **
You can trace this back in a LAVA test log to: 
  setenv bootcmd "fatload mmc 0:4 0x70000000 uImage; fatload mmc 0:4 0x72000000 uInitrd; fatload mmc 0:4 0x71ff0000 board.dtb; bootm 0x70000000 0x72000000 0x71ff0000"
The correct line needs to use mmc 0:5 in each case.
This is what testboot_offset modifies the default is 2. 
  setenv bootcmd "fatload mmc 0:4 0x70000000 uImage; fatload mmc 0:5 0x72000000 uInitrd; fatload mmc 0:5 0x71ff0000 board.dtb; bootm 0x70000000 0x72000000 0x71ff0000"
Hacking session in LAVA A hacking session in LAVA is a way of getting an SSH login directly into a deployed test image, as root. It s a test image, so look around, trash stuff, build stuff, reboot the board When you are done, use stop-hacking and the test completes, leaving the master image unchanged, ready for the next rootfs to go onto the test image you were using. This will be improved in later blog entries but as a starter and so that you can play with the SATA drive with a recent installation, this is the JSON for a LAVA job which gives you an SSH session inside the LAVA test image on an iMX53. The test image comes from Linaro releases 12.01. I haven t set up a proxy yet, so to save time, I downloaded it once and used a local file:// URL in the testjob. Note the rootfstype parameter without this, LAVA will helpfully format the partition onto which the rootfs is installed to ext4, with predictable results. (This bug is currently being fixed in LAVA so that LAVA uses the same filesystem as the downloaded image. All part of making LAVA less presumptive.) The smoke-tests-basic testdef is gratuitous and can be removed but this particular run won t be quick it took over an hour to get into the hacking session, but future topics here will improve that so the extra few minutes for the smoke-tests is not a big win. Change the value for GATEWAY to something sane for your network, change the value for PUB_KEY to your SSH public key. Change the image location to something which works for your setup. 
 
 "health_check": false,
 "job_name": "iMX53-hacking",
 "logging_level": "DEBUG",
 "device_type": "mx53loco",
 "timeout": 900,
 "actions": [
 
 "command": "deploy_linaro_image",
 "parameters": 
 "image": "file:///home/linaro/lava/mx53loco-ubuntu-desktop.img.gz",
 "rootfstype": "ext3"
 
 ,
 
 "command": "lava_test_shell",
 "parameters": 
 "testdef_repos": [
 
 "git-repo": "git://git.linaro.org/qa/test-definitions.git",
 "testdef": "ubuntu/smoke-tests-basic.yaml"
 ,
 
 "git-repo": "http://git.linaro.org/git/lava-team/hacking-session.git",
 "parameters": 
 "GATEWAY": "10.15.0.1",
 "PUB_KEY": "...."
 ,
 "testdef": "hacking-session-debian.yaml"
 
 ],
 "timeout": 18000
 
 ,
 
 "command": "submit_results_on_host",
 "parameters": 
 "server": "http://sylvester.codehelp/RPC2/",
 "stream": "/anonymous/codehelp/"
 
 
 ]
(Use http://jsonlint.com to reformat.) I was part of the way through running debootstrap on the SATA drive when ser2net timed out because I hadn t changed the values at that time. Configuring set2net for LAVA ser2net can run on any machine capable of providing the serial connections, including a dedicated device. Decide on a port range to use on the machine running ser2net. If the device has more than one network interface, ensure that ser2net only offers connections on the relevant interface by prefixing the port with the IP address or hostname. telnet is generally the easiest ser2net state to use with LAVA. 
#  ::::
#     TCP port
#            Name   or  number of the TCP/IP port to accept con-
#            nections from for this device.  A port number may
#            be of the form [host,]port, such as 127.0.0.1,2000
#            or localhost,2000.  If this is specified, it will
#            only bind to the IP address specified. Otherwise
#            it will bind to all the ports on the machine.
# 10.15.0.1,4000:telnet:600:/dev/ttyUSB0:115200 8DATABITS NONE 1STOPBIT banner
Clear the timeout or long running jobs will fail. 
10.15.0.1,4000:telnet:0:/dev/ttyUSB0:115200 8DATABITS NONE 1STOPBIT banner
10.15.0.1,4001:telnet:0:/dev/ttyUSB1:115200 8DATABITS NONE 1STOPBIT banner
I ve tried relatively carefully to preserve the instructions here from my notes. However, if you do try this yourself and there are problems, let me know and I ll update. I m also planning on putting all of the config files, patches, JSON examples and other code into a git repo fairly soon to make things easier.

Johannes Schauer: botch updates

My last update about ongoing development of botch, the bootstrap/build ordering tool chain, was three months ago with the announcement of bootstrap.debian.net. Since then a number of things happened, so I thought an update was due.

New graphs for port metrics By default, a dependency graph is created by arbitrarily choosing an installation set for binary package installation or source package compilation. Installation set vertices and source vertices are connected according to this arbitrary selection. Niels Thykier approached me at Debconf13 about the possibility of using this graph to create a metric which would be able to tell for each source package, how many other source packages would become uncompilable or how many binary packages would become uninstallable, if that source package was removed from the archive. This could help deciding about the importance of packages. More about this can be found at the thread on debian-devel. For botch, this meant that two new graph graphs can now be generated. Instead of picking an arbitrary installation set for compiling a source package or installing a binary package, botch can now create a minimum graph which is created by letting dose3 calculate strong dependencies and a maximum graph by using the dependency closure.

Build profile syntax in dpkg With dpkg 1.17.2 we now have experimental build profile support in unstable. The syntax which ended up being added was: 
Build-Depends: large (>= 1.0), small <!profile.stage1>
But until packages with that syntax can hit the archive, a few more tools need to understand the syntax. The patch we have for sbuild is very simple because sbuild relies on libdpkg for dependency parsing. We have a patch for apt too, but we have to rebase it for the current apt version and have to adapt it so that it works exactly like the functionality dpkg implements. But before we can do that we have to decide how to handle mixed positive and negative qualifiers or whether to remove this feature altogether because it causes too much confusion. The relevant thread on debian-dpkg starts here.

Update to latest dose3 git Botch heavily depends on libdose3 and unfortunately requires features which are only available in the current git HEAD. The latest version packaged in Debian is 3.1.3 from October 2012. Unfortunately the current dose3 git HEAD also relies on unreleased features from libcudf. On top of that, the GraphML output of the latest ocamlgraph version (1.8.3) is also broken and only fixed in the git. For now everything is set up as git submodules but this is the major blocker preventing any packaging of botch as a Debian package. Hopefully new releases will be done soon for all involved components.

Writing and reading GraphML Botch is a collection of several utilities which are connected together in a shell script. The advantage of this is, that one does not need to understand or hack the OCaml code to use botch for different purposes. In theory it also allows to insert 3rd party tools into a pipe to further modify the data. Until recently this ability was seriously hampered by the fact that many botch tools communicated with each other through marshaled OCaml binary files which prevent everything which is not written in OCaml from modifying them. The data that was passed around like this were the dependency graphs and I initially implemented it like that because I didnt want to write a GraphML parser. I now ended up writing an xmlm based GraphML parser so as of now, botch only reads and writes ASCII text files in XML (for the graphs) and in rfc822 packages format (for Packages and Sources files) which can both easily be modified by 3rd party tools. The ./tools directory contains many Python scripts using the networkx module to modify GraphML and the apt_pkg module to modify rfc822 files.

Splitting of tools To further increase the ability to modify program execution without having to know OCaml, I split up some big tools into multiple smaller ones. Some of the smaller tools are now even written in Python which is probably much more hackable for the general crowd. I converted those tools to Python which did not need any dose3 functionality and which were simple enough so that writing them didnt take much time. I could convert more tools but that might introduce bugs and takes time which I currently dont have much of (who does?).

Gzip instead of bz2 Since around January 14, snapshot.debian.org doesnt offer bzip2 compressed Packages and Sources files anymore but uses xz instead. This is awesome for must purposes but unfortunately I had to discover that there exist no OCaml bindings for libxz. Thus, botch is now using gzip instead of bz2 until either myself or anybody else finds some time to write a libxz OCaml binding.

Self hosting Fedora Paul Wise made me aware of Harald Hoyer's attempts to bootstrap Fedora. I reproduced his steps for the Debian dependency graph and it turns out that they are a little bit bigger. I'm exchanging emails with Harald Hoyer because it might not be too hard to use botch for rpm based distributions as well because dose3 supports rpm. The article also made me aware of the tred tool which is part of graphviz and allows to calculate the transitive reduction of a graph. This can help making horrible situations much better.

Dose3 bugs I planned to generate such simplified graphs for the neighborhood of each source package on bootstrap.debian.net but then binutils stopped building binutils-gold and instead provided binutils-gold while libc6-dev breaks binutils-gold (<< 2.20.1-11). This unfortunately triggered a dose3 bug and thus bootstrap.debian.net will not generate any new results until this is fixed in dose3. Another dose3 bug affects packages which Conflicts/Replaces/Provides:bar while bar is fully virtual and are Multi-Arch:same. Binaries of different architecture with this property can currently not be co-installed with dose3. Unfortunately linux-libc-dev has this property and thus botch cannot be used to analyze cross builds until that bug is fixed in dose3. I hope I get some free time soon to be able to look at these dose3 issues myself.

More documentation Since I started to like the current set of tools and how they work together I ended up writing over 2600 words of documentation in the past few days. You can start setting up and running botch by reading the first steps and get more detailed information by reading about the 28 tools that botch makes use of as of now. All existing articles, thesis and talks are linked from the wiki home.

Steve Kemp: External Comments, updated

The simple external-comments code is now complete enough for me to stop poking it on a daily basis: All in all I'm pretty happy with the way it works, and the server-code. The client-side Javascript is less good, but I'm probably done poking that too. In an ideal world the client-side code should be a jQuery plugin, but I've not worked out how to make a static method (the JSONP callback) be a member of a jQuery plugin-object. So without that I have to re-pass the options around too many places, rather than making them a member of "this". Meh, pull requests welcome for adding new storage back-ends (redis and sqlite are supported by default), and similarly for cleanups. Links:

Andrew Pollock: [life] Day 10, Playgroup again

I was bracing myself for a rough night, but Zoe slept through until almost 6am. Today was playgroup again, so we just took a slow start to the day and headed out on the bike a bit after 9am. There was a mix of new faces and people from last week. I packed a lunchbox this time instead of relying on the snack collection in my "Dad Bag" backpack. We worked on trying to approach other girls to get them to play. Unfortunately they're all that little bit younger and more shy, so we haven't had a huge success. Zoe's been pretty happy just playing with the various toys and playing with me. Parachute time was a big hit again. I think there were about 25 adults there (I was the only Dad this week), with many with multiple children. The youngest baby I saw was a 7 week old. Zoe had been asking me what a honeydew was, so after Playgroup we dropped by the Hawthorne Garage and grabbed half a honeydew and a couple of tomatoes and I made some fritters for lunch with leftover corned beef from dinner the other night. Zoe wasn't too impressed with them. I think she didn't like the diced tomato. She declared that the honeydew was just like rockmelon. We read a story in the hammock and she went down pretty easily for nap after that. I'd just started doing the dishes when my mate Steve gave me a call. He was in the area, and he dropped in. That worked out well, because I'd been meaning to pick his brain about my little electronics project. We geeked it up for a bit. Zoe's mini lab coat arrived in the mail, and when she finally woke up after 3pm, I asked her if she wanted to go out to Bunnings to get a pair of kids-sized safety glasses, but she said she wanted to do something "fun" instead. When I asked her what she wanted to do, she initially wanted to go to Wet and Wild, but that's an all day outing and we're going next Wednesday, so she settled on going to Colmslie pool again instead. So we quickly got organised and jumped on the bike and biked over to the pool. We started out in the kid's play area, which had a couple of small slides and some fountains and whatnot, and then we gravitated over to the 50 metre pool, where Zoe practised her diving for a bit and generally quite confidently navigated the 1.5m deep end of the pool. It was good, because at least I could just stand up instead of tread water while she practised diving in off the blocks. We had a post-swim ice cream and biked home, and then I whipped up dinner and it was bathtime and bedtime. I'm going to have to start tracking my weekly kilometres with Zoe on the bike, because I think it'll be a reasonable distance.

5 February 2014

Daniel Pocock: Debian SIP without reading the manual

Most people don't have time to read manuals. The Debian SIP service has a reasonably thorough user guide with screenshots but some people may just want to have this level of detail if they really have to troubleshoot a problem. WebRTC is the fastest way to start Just set up your db.debian.org RTC password and use it to log in to the https://rtc.debian.org site. It all runs in the browser, no plugin or configuration required. Firewall/NAT trouble The current range of WebRTC browsers seem to work well for consumer and small office networks with NAT routers or mobile/cell networks. Corporate networks with an outright block on UDP will be supported soon. Supported browsers The Chromium 31 package on wheezy appears to work fine and was used in some of the demos at DebConf13 For Iceweasel/Firefox users, it is not possible to use the version 17 package from wheezy at all. At the time of writing, Iceweasel 24 is in jessie and sid while version 27 is available upstream. Mozilla gradually introduced WebRTC support on both desktop and Android between versions 20 to 24 and it is still evolving. If you don't want to use Chromium and don't want to upgrade your desktop/laptop beyond wheezy, consider trying Firefox on an Android device. Your friends can call you too There is no excuse for them to pester you with proprietary solutions any more. They can dial your debian.org SIP address from https://freephonebox.net. You can even link to it from your web site or blog by adding the dial parameter to the URL, like https://freephonebox.net?dial=pocock%40debian.org Please complain Bug reports and questions are always welcome. A good place for questions or complaints where the cause of the bug is not obvious is the Free-RTC mailing list.

Thomas Koch: shared infrastructure coop

I'm working in a very small web agency with 4 employees, one of them part time and our boss who doesn't do programming. It shouldn't come as a surprise, that our development infrastructure is not perfect. We have many ideas and dreams how we could improve it, but not the time.
Now we have two obvious choices: Either we just do nothing or we buy services from specialized vendors like github, atlassian, travis-ci, heroku, google and others.
Doing nothing does not work for me. But just buying all this stuff doesn't please me either. We'd depend on proprietary software, lock-in effects or one-size-fits-all offerings.
Another option would be to find other small web shops like us, form a cooperative and share essential services. There are thousands of web shops in the same situation like us and we all need the same things: As I said, all of the above is available as commercial offerings. But I'd prefer the following to be satisfied:
Does something like that already exists? There already is the German cooperative hostsharing which is kind of similar but does provide mainly hosting, not services. But I'll ask them next after writing this blog post.
Is your company interested in joining such an effort? Does it sound silly?




Next.

Previous.