Search Results: "xam"

27 June 2011

Raphaël Hertzog: Deciphering one of dpkg s weirdest errors: short read on buffer copy

As a Debian/Ubuntu user, you re likely to be exposed at some point to an error reported by dpkg. In a series of articles, I ll explain some of the errors that you might encounter. Some error messages can be confusing at times. Most of the error strings do not appear very often and developers thus tend to use very terse description of the underlying problem. In other cases the architecture of the software makes it difficult to pin-point the real problem because the part that displays the error is several layers above the one that generated the initial error. This is for example the case with this error of dpkg:
Unpacking replacement xulrunner-1.9.2 ...
dpkg-deb (subprocess): data: internal gzip read error: '<fd:0>: too many length or distance symbols'
dpkg-deb: error: subprocess <decompress> returned error exit status 2
dpkg: error processing /var/cache/apt/archives/xulrunner-1.9.2_1.9.2.17+build3+nobinonly-0ubuntu1_amd64.deb (--unpack):
 short read on buffer copy for backend dpkg-deb during  ./usr/lib/xulrunner-1.9.2.17/components/libdbusservice.so'
First, the decompression layer discovers something unexpected in the data read in the .deb file and dpkg-deb outputs the error message coming from zlib ( too many length or distance symbols ). This causes the premature end of dpkg-deb --fsys-tarfile that dpkg had executed to extract the .data.tar archive from the deb file. In turn, dpkg informs us that dpkg-deb did not send all the data that were announced (and hence the short read in the error message) and that were meant to be part of the file /usr/lib/xulrunner-1.9.2.17/components/libdbusservice.so . That s all nice but it doesn t help you much in general. What you must understand from the above is that the .deb file is corrupted (sometimes just truncated). In theory it should not happen since APT verifies the checksums of files when they are downloaded. But computers are not infallible and even if the downloaded data was good, it can have been corrupted when stored on disk (for example cheap SSD disks are known to not last very well). Try removing the file (usually with apt-get clean since it s stored in APT s cache) and let APT download it again. Chances are that it will work on the second try. Otherwise consider doing a memory and HDD check as something is probably broken in your computer.
Join my free newsletter and learn more tips for users.

4 comments Liked this article? Click here. My blog is Flattr-enabled.

25 June 2011

Philipp Kern: Porting a library to gtk3: change soname

Last week I tried switching a library to Gtk3. The needed changes to the code are available through --with-gtk3. However this is generally not enough. Even if your symbol list doesn't change, the ABI changes implicitly. The library in question had a .symbols file, but that's not enough because the resulting GUI application will bail out at runtime if symbols of both Gtk2 and Gtk3 are found in the same address space. That's mostly because C symbols don't contain any signatures with return types and parameters.

So if your library upstream did not change the soname for the Gtk3 build, please encourage them to do so. Also keep in mind that this most likely means new pkg-config files specific to the Gtk3 build, too. At least if you want your reverse-depends to be able to build against either Gtk2 or Gtk3 in a predictable way.

An example is this change to gtk-vnc, which uses gtk-vnc-2.0 as the new API/pkg-config name for the Gtk3 build, gtk-vnc-1.0 remains the old Gtk2 one. The soname changes from libgtk-vnc-1.0.so.0 to libgtk-vnc-2.0.so.0.

(Thanks to Michael Biebl and Julien Cristau for pointing out the obvious to me.)

Vincent Bernat: Network lab: site to site VPN

The goal of this lab is to setup a site-to-site IPsec VPN. This lab is similar to my first lab using UML. The major differences are:

The lab VPN lab

The big picture The main point of this architecture is that you get two redundant networks for each site. The external ones are used when hosts need Internet access. The internal ones are used when they want to communicate with each other. Because we don t want hosts from the first site to communicate with hosts from the second site through the external network, we extend the internal network with the help of a site-to-site IPsec VPN. Each set of networks is using two layer-3 networks on top of two layer-2 networks. OSPF is used to provide redundancy and aggregation. For example, R1 has two routes through V1 to access R2 which is on the other site:
# ip route show 192.168.96.0/19
192.168.96.0/19  proto zebra  metric 10000 
        nexthop via 192.168.1.100  dev eth0 weight 1
        nexthop via 192.168.2.100  dev eth1 weight 1

Setting up the lab First, grab the source of the lab:
$ git clone https://vincentbernat@github.com/vincentbernat/network-lab.git
$ cd network-lab/lab-s2s-vpn
You may try to run ./setup directly. It may be safer to build an image of Debian inside inside a directory and use it as a base for the lab.
$ sudo debootstrap sid ../sid-chroot http://ftp.fr.debian.org/debian/
[...]
$ sudo chroot ../sid-chroot /bin/bash
# apt-get install iproute zsh rsyslog less makedev tcpdump
# apt-get install bird quagga racoon
# exit
$ ROOT=../sid-chroot ./setup
You still need a working UML kernel. On Ubuntu, the one provided in user-mode-linux does not contain the aufs.ko module. I did not find a simple workaround. You can just grab the package from Debian and install it on your Ubuntu with dpkg -i.

Use of BIRD With the previous lab, Quagga was used as an OSPF routing daemon. We still use it on R1, R2, E1 and E2. However, Quagga is not able to run multiple instances of OSPF. BIRD can. Therefore, we use it on V1 and V2 because we need two OSPF instances: one for the external network and one for the internal network. Since we also need ECMP, we need at least BIRD 1.3.0. BIRD allows the definition of multiple routing tables. Those tables are independant of the kernel routing tables. Each protocol will maintain its own set of routes and will allow to export (or import) routes from one of the routing table maintained by BIRD. In our setup, we use four protocols and only one routing table (the default one). BIRD routing tables The ospf EXTERNAL protocol is one of the two OSPF instances we use on V1. We import all the routes learned through the OSPF process into the internal BIRD routing table. We don t export any route from the BIRD routing table to this OSPF instance. Here is the configuration of this protocol:
protocol ospf EXTERNAL  
   ecmp yes;
   import all;
   export none;
   area 0.0.0.0  
      networks  
         203.0.113.0/26;
         203.0.113.64/26;
         203.0.113.150/32;
       ;
      interface "eth2";
      interface "eth3";
      interface "lo"  
        stub yes;
       ;
    ;
 
203.0.113.150/32 is defined on the loopback interface of V1. This is the IP of the local VPN endpoint. The ospf INTERNAL protocol is the second OSPF instance. We import all routes from this protocol to the BIRD routing table. We also want to advertise the route to the remote site. This route is defined in the static STATIC protocol and imported into BIRD routing table.
protocol static STATIC  
   import all;
   export none;
   route 192.168.96.0/19 via "lo";
 
Therefore, ospf INTERNAL protocol export all routes originating from the static STATIC protocol. This is done using filters:
protocol ospf INTERNAL  
   ecmp yes;
   import all;
   export filter  
     if proto = "STATIC" then accept;
     reject;
    ;
   area 0.0.0.0  
      networks  
         192.168.1.0/24;
         192.168.2.0/24;
       ;
      interface "eth0";
      interface "eth1";
    ;
 
The kernel protocol is managing one kernel routing table. By default, it handles the main table. We don t import any route from the kernel. However, we copy routes that come from other protocols, except the static route because we already have a default route and the static route is some kind of blackhole, we don t want to use it for real routing.
protocol kernel  
   persist;
   import none;
   export filter  
     if proto = "EXTERNAL" then accept;
     if proto = "INTERNAL" then accept;
     reject;
    ;
 
The filtering mechanism built inside BIRD is pretty powerful.

Testing After starting the lab, you need to wait a few seconds. You should then be able to ping R2 from R1:
R1# ping -c3 -I 192.168.15.1 192.168.115.1
PING 192.168.115.1 (192.168.115.1) from 192.168.15.1 : 56(84) bytes of data.
Warning: time of day goes back (-5435us), taking countermeasures.
64 bytes from 192.168.115.1: icmp_req=1 ttl=62 time=0.921 ms
64 bytes from 192.168.115.1: icmp_req=2 ttl=62 time=0.569 ms
64 bytes from 192.168.115.1: icmp_req=3 ttl=62 time=0.698 ms
--- 192.168.115.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2009ms
rtt min/avg/max/mdev = 0.569/0.729/0.921/0.147 ms
On V1, you can check BIRD internal routing table using birdc:
> show route 
0.0.0.0/0          multipath [EXTERNAL 19:43] * E2 (150/10/1) [0.0.0.11]
        via 203.0.113.1 on eth2 weight 1
        via 203.0.113.65 on eth3 weight 1
192.168.96.0/19    dev lo [STATIC 19:42] * (200)
192.168.1.0/24     dev eth0 [INTERNAL 19:42] * I (150/10) [0.0.0.100]
192.168.2.0/24     dev eth1 [INTERNAL 19:42] * I (150/10) [0.0.0.100]
192.168.15.0/24    multipath [INTERNAL 19:43] * I (150/20) [0.0.0.10]
        via 192.168.1.10 on eth0 weight 1
        via 192.168.2.10 on eth1 weight 1
203.0.113.0/26     dev eth2 [EXTERNAL 19:42] * I (150/10) [0.0.0.100]
203.0.113.64/26    dev eth3 [EXTERNAL 19:42] * I (150/10) [0.0.0.100]
203.0.113.150/32   dev lo [EXTERNAL 19:42] * I (150/0) [0.0.0.100]
The kernel routing table is similar except it does not contain the static route.

Adding more redundancy The first goal of this lab was to test multiple OSPF instances and ECMP with BIRD. It could be completed:
  • Adding a redundant VPN is easy: this is just a matter of duplicating the setup of the existing VPN.
  • Each router should be redundant by getting a sibling with the same configuration. In the case of R1 and R2, a VIP could be shared between the router and its sibling with VRRP to allow hosts behind to have a single gateway and not use a routing protocol.
  • Because of the use of static routes instead of BGP, there is nothing to detect a VPN failure (which may be problematic if we setup a redundant VPN). Some VPN are able to invalidate a route when the associated VPN is down. This is not our case. Moreover, there may be some duplication in routing. This is not the case here because each site has its own internal network.

Vincent Bernat: Network lab: site to site VPN

The goal of this lab is to setup a site-to-site IPsec VPN. This lab is similar to my first lab using UML. The major differences are:

The lab VPN lab

The big picture The main point of this architecture is that you get two redundant networks for each site. The external ones are used when hosts need Internet access. The internal ones are used when they want to communicate with each other. Because we don t want hosts from the first site to communicate with hosts from the second site through the external network, we extend the internal network with the help of a site-to-site IPsec VPN. Each set of networks is using two layer-3 networks on top of two layer-2 networks. OSPF is used to provide redundancy and aggregation. For example, R1 has two routes through V1 to access R2 which is on the other site:
# ip route show 192.168.96.0/19
192.168.96.0/19  proto zebra  metric 10000 
        nexthop via 192.168.1.100  dev eth0 weight 1
        nexthop via 192.168.2.100  dev eth1 weight 1

Setting up the lab First, grab the source of the lab:
$ git clone https://vincentbernat@github.com/vincentbernat/network-lab.git
$ cd network-lab/lab-s2s-vpn
You may try to run ./setup directly. It may be safer to build an image of Debian inside inside a directory and use it as a base for the lab.
$ sudo debootstrap sid ../sid-chroot http://ftp.fr.debian.org/debian/
[...]
$ sudo chroot ../sid-chroot /bin/bash
# apt-get install iproute zsh rsyslog less makedev tcpdump
# apt-get install bird quagga racoon
# exit
$ ROOT=../sid-chroot ./setup
You still need a working UML kernel. On Ubuntu, the one provided in user-mode-linux does not contain the aufs.ko module. I did not find a simple workaround. You can just grab the package from Debian and install it on your Ubuntu with dpkg -i.

Use of BIRD With the previous lab, Quagga was used as an OSPF routing daemon. We still use it on R1, R2, E1 and E2. However, Quagga is not able to run multiple instances of OSPF. BIRD can. Therefore, we use it on V1 and V2 because we need two OSPF instances: one for the external network and one for the internal network. Since we also need ECMP, we need at least BIRD 1.3.0. BIRD allows the definition of multiple routing tables. Those tables are independant of the kernel routing tables. Each protocol will maintain its own set of routes and will allow to export (or import) routes from one of the routing table maintained by BIRD. In our setup, we use four protocols and only one routing table (the default one). BIRD routing tables The ospf EXTERNAL protocol is one of the two OSPF instances we use on V1. We import all the routes learned through the OSPF process into the internal BIRD routing table. We don t export any route from the BIRD routing table to this OSPF instance. Here is the configuration of this protocol:
protocol ospf EXTERNAL  
   ecmp yes;
   import all;
   export none;
   area 0.0.0.0  
      networks  
         203.0.113.0/26;
         203.0.113.64/26;
         203.0.113.150/32;
       ;
      interface "eth2";
      interface "eth3";
      interface "lo"  
        stub yes;
       ;
    ;
 
203.0.113.150/32 is defined on the loopback interface of V1. This is the IP of the local VPN endpoint. The ospf INTERNAL protocol is the second OSPF instance. We import all routes from this protocol to the BIRD routing table. We also want to advertise the route to the remote site. This route is defined in the static STATIC protocol and imported into BIRD routing table.
protocol static STATIC  
   import all;
   export none;
   route 192.168.96.0/19 via "lo";
 
Therefore, ospf INTERNAL protocol export all routes originating from the static STATIC protocol. This is done using filters:
protocol ospf INTERNAL  
   ecmp yes;
   import all;
   export filter  
     if proto = "STATIC" then accept;
     reject;
    ;
   area 0.0.0.0  
      networks  
         192.168.1.0/24;
         192.168.2.0/24;
       ;
      interface "eth0";
      interface "eth1";
    ;
 
The kernel protocol is managing one kernel routing table. By default, it handles the main table. We don t import any route from the kernel. However, we copy routes that come from other protocols, except the static route because we already have a default route and the static route is some kind of blackhole, we don t want to use it for real routing.
protocol kernel  
   persist;
   import none;
   export filter  
     if proto = "EXTERNAL" then accept;
     if proto = "INTERNAL" then accept;
     reject;
    ;
 
The filtering mechanism built inside BIRD is pretty powerful.

Testing After starting the lab, you need to wait a few seconds. You should then be able to ping R2 from R1:
R1# ping -c3 -I 192.168.15.1 192.168.115.1
PING 192.168.115.1 (192.168.115.1) from 192.168.15.1 : 56(84) bytes of data.
Warning: time of day goes back (-5435us), taking countermeasures.
64 bytes from 192.168.115.1: icmp_req=1 ttl=62 time=0.921 ms
64 bytes from 192.168.115.1: icmp_req=2 ttl=62 time=0.569 ms
64 bytes from 192.168.115.1: icmp_req=3 ttl=62 time=0.698 ms
--- 192.168.115.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2009ms
rtt min/avg/max/mdev = 0.569/0.729/0.921/0.147 ms
On V1, you can check BIRD internal routing table using birdc:
> show route 
0.0.0.0/0          multipath [EXTERNAL 19:43] * E2 (150/10/1) [0.0.0.11]
        via 203.0.113.1 on eth2 weight 1
        via 203.0.113.65 on eth3 weight 1
192.168.96.0/19    dev lo [STATIC 19:42] * (200)
192.168.1.0/24     dev eth0 [INTERNAL 19:42] * I (150/10) [0.0.0.100]
192.168.2.0/24     dev eth1 [INTERNAL 19:42] * I (150/10) [0.0.0.100]
192.168.15.0/24    multipath [INTERNAL 19:43] * I (150/20) [0.0.0.10]
        via 192.168.1.10 on eth0 weight 1
        via 192.168.2.10 on eth1 weight 1
203.0.113.0/26     dev eth2 [EXTERNAL 19:42] * I (150/10) [0.0.0.100]
203.0.113.64/26    dev eth3 [EXTERNAL 19:42] * I (150/10) [0.0.0.100]
203.0.113.150/32   dev lo [EXTERNAL 19:42] * I (150/0) [0.0.0.100]
The kernel routing table is similar except it does not contain the static route.

Adding more redundancy The first goal of this lab was to test multiple OSPF instances and ECMP with BIRD. It could be completed:
  • Adding a redundant VPN is easy: this is just a matter of duplicating the setup of the existing VPN.
  • Each router should be redundant by getting a sibling with the same configuration. In the case of R1 and R2, a VIP could be shared between the router and its sibling with VRRP to allow hosts behind to have a single gateway and not use a routing protocol.
  • Because of the use of static routes instead of BGP, there is nothing to detect a VPN failure (which may be problematic if we setup a redundant VPN). Some VPN are able to invalidate a route when the associated VPN is down. This is not our case. Moreover, there may be some duplication in routing. This is not the case here because each site has its own internal network.

Andrew Pollock: [tech] On trying to find the resource limits of a running process on an old kernel

I had cause to try and get a core dump from a segfaulting process at work the other day, and I wanted to figure out if fiddling with /etc/security/limits.conf was going to do the trick (I didn't) or if I had to modify the initscript to include a ulimit -c unlimited call. Of course, on a modern system (>= 2.6.24), one would just take a look at /proc/PID/limits and get on with life, but unfortunately the system in question was running 2.6.18, so more fiddling around was required. I'd found something once before that told me how to do it with GDB, but all I could find this time around was a rather over-complicated Knol article, which made a bunch of assumptions (well mainly that the binary in question wasn't stripped). So with some help from the Knol article, I muddled through it. Disclaimer: I don't profess to be an expert on system internals like this, so if this eats your first-born, don't come crying to me. Firstly, you need to know that it's the getrlimit(2) system call that you want to be using, and then you need to figure out the number for the resource limit you want to retrieve. The man page for getrlimit(), tells you it's defined in /usr/include/sys/resource.h, but I've found that the actual useful bits end up being in /usr/include/bits/resource.h I wanted the resource limit for the maximum core dump size, which is RLIMIT_CORE and has a value of 4. Next, you need to know that the getrlimit() system call takes an integer and a pointer to an rlimit structure as arguments. We've just figured out the value for the integer, but we're also going to need to pass a pointer as the second parameter. A pointer to enough memory to hold an rlimit structure. Fortunately, the rlimit structure is pretty simple:
struct rlimit  
    rlim_t rlim_cur;  /* Soft limit */
    rlim_t rlim_max;  /* Hard limit (ceiling for rlim_cur) */
 ;
After a bit of grepping around in /usr/include, I determined that an rlim_t is essentially an unsigned long int, so we need to allocate a pointer big enough to hold two of them. Note that if we had an unstripped binary, we could have saved a lot of faffing around by just going
print sizeof(struct rlimit)
in GDB (assuming that the binary has the getrlimit symbol in it) The sure-fire way of figuring out how much memory we need for this pointer is to go
print sizeof(unsigned long int)
in GDB, and then double that (since we want two of them). On my system an unsigned long int is four bytes, so I'm going to want to allocate enough memory for 8 bytes. Now it's time to attach GDB to the offending process and see what the resource limit currently is. (gdb -p PID)
(gdb) print malloc(8)
$1 = 152186904
(gdb) print getrlimit(4, $1)
$2 = 0
(gdb) x/2xw $1
0x9123018:	0x00000000	0x7fffffff
(gdb) quit
So in this particular case, we've retrieved the soft and hard limits of the RLIMIT_CORE resource limit, and you can see that the soft limit is zero, and the hard limit is unlimited. Note that the getrlimit() function returns an integer as its return code, which is what the $2 = 0 is above. Now it's just a case of altering the the resource limits via the preferred mechanism, restarting the process and then repeating this GDB examination of the process to check they were changed successfully. Circumstances where this can all fall down would appear to be ones where the getrlimit symbol isn't present in the binary, or the binary is compiled as a position-independent executable. I'd think that in the latter case, the system is probably modern enough to be running a kernel that supports directly examining the resource limits via the /proc filesystem.

24 June 2011

Rapha&#235;l Hertzog: People behind Debian: Sam Hartman, Kerberos package maintainer

Sam Hartman is a Debian developer since 2000. He has never taken any sort of official role within Debian (that is besides package maintainer), yet I know him for his very thoughtful contributions to discussions both on mailing lists and IRL during Debconf. Until I met him at Debconf, I didn t know that he was blind, and the first reaction was to be impressed because it must be some tremendous effort to read the volume of information that Debian generates on mailing lists. In truth he s at ease with his computer much like I am although he uses it in a completely different way. Read on to learn more, my questions are in bold, the rest is by Sam. Raphael: Who are you? Sam: I m Sam Hartman. I m a 35-year-old software engineer. I am a principal consultant and co-owner of a small consulting company, called Painless Security. I started using Debian in the mid 1990 s around the time of the bo release. I ended up deciding to join the project as a developer in 2000. Raphael: You re blind and yet you re using your computer as effectively as I am. Can you explain us how you setup your computer ? Sam: I gave a talk at Debconf9 on how my computer is set up; you can watch the video for full details. My main laptop runs Debian. I use the gnome-orca package as my primary screen reader. It speaks the Gnome desktop. It does a relatively good job of speaking Iceweasel/Firefox and Libreoffice. While it does speak gnome-terminal, it s not really good enough at speaking terminal programs that I am comfortable using it. So, I run Emacs with the Emacspeak package. Within that, I run the Emacs terminal emulator, and within that, I tend to run Screen. For added fun, I often run additional instances of Emacs within the inner screens. Raphael: Are there important problems in Debian in terms of accessibility to blind people? KDE documentation talks a lot about accessibility, but at least for blind users, the code completely fails to deliver. That means there are a lot of good packages a blind user cannot use. The non-free Adobe Flash player has some accessibility, but it could be a lot better. The free alternatives have none. The free PDF readers have basically no accessibility. You can use pdftotext, but you cannot actually read a PDF in a graphical application. It s way too easy for a misbehaving program to lock up the entire accessibility infrastructure. Gnash is a big culprit here: if my Iceweasel starts Gnash, there s a good chance that either Iceweasel or the entire desktop will appear to hang from an accessibility standpoint. Other programs, including gksu tend to fail in this way. Some of the more dynamic website features like pop up menus or selection lists are really difficult to find and click without causing them to disappear. This gets better and worse over time as the accessibility support in Iceweasel changes and as websites change. Raphael: What s your biggest achievement within Debian? Sam: I decided to be a Debian developer because I thought that in 2000, Debian support for using enterprise security and infrastructure software was lacking. Back then, any software that included crypto functionality was segregated into a special non-us archive. Some of the software was missing; I started by packaging MIT Kerberos for Debian. Other software had security or enterprise features disabled in the packages. At the height of working on this, I was maintaining krb5, some SASL modules, PAM, some PAM modules,OpenAFS, a version of and Ssh with Kerberos support. I was also involved in the effort to resolve the legal issues so that we could move security software into the main archive. I think this work has been a huge success. In fact, it s been such a success that other people are now doing most of the work. I still maintain the krb5 package. When I started I felt like I was pushing against the flow trying to get people to add patches, sometimes even having to fork a package. However, now, I can maintain just one component and there are enough others who shared my original goal that the work continues. These days, I m working on something that s an evolution of this enterprise security work. I m packaging Project Moonshot for Debian and Ubuntu. Project Moonshot is a response to the wide variety of identity management systems such as OpenID, Oauth, SAML, and the like. Moonshot works to create an identity management approach that works well both for the web and for client-server and other applications. The project is a lot of fun, but the role of Debian in the project is also interesting. One of the things we want to show with Moonshot is how our technology can be effective when it is integrated throughout a platform. To really show the potential it needs to work with as many applications in a given platform as possible. The open and collaborative nature of the Debian community makes it possible to introduce a new technology and have that technology evaluated based on its merits. We don t need huge business relationships or marketing deals to integrate our technology: we need some combination of doing the work ourselves and showing others the benefits of working with us. For someone trying to do innovative work, the Debian model is powerful. Raphael: If you could spend all your time on Debian, what would you work on? Sam: I d really love to work with the embedded Debian folks. The vision of a single source base that could be the stack from devices from small embedded devices all the way up to high-end servers is very appealing. Doing that with Debian involves a number of challenges. However with the right people working on meeting these challenges full-time, I think we could offer something promising. I d also love to have the time to contribute to project infrastructure: working on the release team, helping ftpmaster, that sort of thing. However I don t. I m just happy that so much of my consulting practice involves working on open-source software. Raphael: What s the biggest problem of Debian? Sam: There s something really not right about how we transition libraries from unstable to testing. Every time I get involved in a library transition I m shocked at how complicated it is and how disruptive it is both to testing and unstable. We need to look at technology and processes to break up the dependency snarles. For example we don t have good archive tools for keeping old versions of libraries around to ease transitions. If you haven t thought about this issue you ll probably say that I m being overly picky and this can t be the major problem for Debian. However, if you think about how much this impacts our ability to introduce things into unstable around the times of the freeze or about how much it slows the release process, you may begin to appreciate how big of an issue this is. Raphael: Is there someone in Debian that you admire for their contributions? Sam: There are a number of people who have been role models for me over the years. Anthony Towns really helped me understand a lot of what drives free-software projects and what needs to be true for positive motivation. Joey Hess showed us all that sometimes, social problems do have technical solutions. If the tools are so good that doing the right thing is far easier than any other course of action, quality improves.
Thank you to Sam for the time spent answering my questions. I hope you enjoyed reading his answers as I did. Subscribe to my newsletter to get my monthly summary of the Debian/Ubuntu news and to not miss further interviews. You can also follow along on Identi.ca, Twitter and Facebook.

5 comments Liked this article? Click here. My blog is Flattr-enabled.

23 June 2011

Evgeni Golov: monitoring Adaptec RAID controllers in Debian

So I got my hands on that new machine, somewhere, somehow, and I had to make myself comfortable on it. The biggest difference in that machine and the ones I used before is the RAID controller, an Adaptec 5405, instead of the 3ware ones I used to have before. That means getting used to new tools, messages and all these things which I ll try to document here.Step 1: does vendor deliver?Well, yes, mostly. There is (mainline) kernel support for the controller, Squeeze s kernel will boot up just fine and I just throw my junk into /dev/sda and it will handle the whole raidy thingy itself. But what is when I want to check the health of the disks, to add a spare or modify the caching? Ask the mighty vendor and it will answer Yes, you can! .So I downloaded that asm_debian_x86_x64_v6_50_18570.tgz, unpacked and faced two debs, one i368 and one amd64. Yes, you could have saved half of the bandwidth if someone would have asked for the arch before, but who cares for some additional 60MiB today? So let s see what it does. Oh no, wait, there is a README ASM_IUG_for_Debian_and_Ubuntu.txt:
1. Make sure to be in root level access.
2. Install the ASM storman .deb package. For example, if the package is storman_6.50-18570_i386.deb,
   then install it using the following command dpkg -i storman_6.50-18570_i386.deb
3. Adaptec Storage Manager will get installed at /usr/StorMan
4. Add the line /usr/StorMan to /etc/ld.so.conf. Then run "ldconfig". This is needed for the arcconf
   utility to find the path of the libstdc++.so.5 library
Well, 1 is obvious, so is 2.
For 3: hum, /usr/StorMan? Ever heard of FHS? Don t think so.
For 4: WTF?! I prefer to get libstdc++5 from my local Debian mirror instead.But at least they provide a .deb, right? Uhm, no, it s an RPM converted via alien :( And don t dare to look at preinst, postinst and postrm (it extracts an own copy of Sun JRE 1.6u16, chmods around wildly on install and does rm -rf something on uninstall, did I see that some days ago on the webs?). Let s get rid of that package as soon as possible, I decided, and just keep arcconf (which will be described later), esp because StorMan itself seems to include some call-home functionality which I d like not to have:
cp -a /usr/StorMan/arcconf /usr/local/sbin/
apt-get remove --purge storman
Step 2: does community deliver?You can, by the way, get (better) packages at http://hwraid.le-vert.net/wiki/DebianPackages from Adam C cile (Le_Vert). I didn t test those much, but I currently use the arcconf package from there, which seems to be exactly what I need: proper dependencies and just the arcconf binary, no Java sh*t.Step 3: monitor the controllerAs this post is about monitoring, not ranting, let s continue. As mentioned before, I only have arcconf installed, which is just a cli to do stuff with the controller. If I understood the docs right, the actual StorMan is able to send notification mails when a disk dies or the controller gets eaten by aliens, but as I don t have StorMan, I have to do things myself.Adam has a aacraid-status package in his repository, which includes some minimalistic shell-daemon to monitor the output of arcconf. It didn t perfectly fit my needs (and had some hickups with my disks which refuse to report a vendor, whyever), so I decided to write something myself (slightly based on what Adam has done in aacraid-status, though).The result is on GitHub: https://github.com/evgeni/aacraidAs you can see the code contains (at the time of writing) exactly ZERO comments and there is no README either, but it s actually quite usable already.I am now running aacraidd and get happy mails when something breaks :) [running = running from /etc/rc.local, no init script yet :)]Step 4: monitor the disks via smartmontoolsThere is one little problem I have with arcconf: it s closed source and does magic to the controller.With the old 3ware controller I was able to monitor the disks with smartmontools/smartd using something like this:
/dev/twa0 -d 3ware,0 -a -s L/../../2/01 -m root
/dev/twa0 -d 3ware,1 -a -s L/../../2/03
This is possible with Adaptect controllers too (not through /dev/twaX obviously). You have to have the sg module loaded, which will result in /dev/sg[012] on my machine, with sg0 being the controller itself and sg1/sg2 the both disks attached to it. Having sg loaded, one can now use smartd with:
/dev/sg1 -d sat -a -s L/../../7/01 -m root
/dev/sg2 -d sat -a -s L/../../7/02 -m root
(-d sat is the important option here!)Thanks to Thomas Krenn Wiki: Smartmontools mit Adaptec RAID Controller for the basic info and ostlogd.spenneberg.net: Adaptec-Raid-Controller S.M.A.R.T.-Healthstatus monitoren for the pointer to the sg module!Step 5: relaxYupp, 4 steps are enough, go relax until the next disk dies (you will be doing something REALLY important that moment according to Murphy).

22 June 2011

Martin Zobel-Helas: How YOU can help Debian!

I recently had been ask how persons usually not involved in Debian's development process can help Debian. This is a question that pops up quite often, so I thought I should write down a bit of that.

Help to make Debian a better OS
If you are using Debian, and you want something of your OS changed, open a bug report. This varies from wishlist bugs if you want to have an enhancement of a package over normal bugs for stuff that you think is a real bug up to serious or grave bug, if you found a security bug. Send a mail to submit@bugs.debian.org or use the tool reportbug and describe the problem you have found. Find more information here. Be as verbose as possible when explaining your problem. This will make it easier for the package maintainer to help you and to understand the problem. If you are not sure which package to report the bug against, report it against unknown. The bug will be taken care of, there are guys redirecting those bug reports to the appropriate package!
You could also help to verify bug reports. There are dozens of packages around, that have hundred of open bug reports. It will help the Debian package maintainer if you can tell him a "me too", esp for complex problems, or if you found out how to reproduce a bug.

Help by spreading the word
If you are using Debian, speak about it! If you have problems with Debian, speak about it! If you like Debian, speak about it. Read the debian-user mailing list (or a localized one) and jump in if users have the same problem you had, and help them.
All sort of publicity will help Debian. If there is a small exhibition near to your living, speak there about Debian, and how you are using it. Speak also to Debian, we can help you to announce your presence at that exhibition and provide you with information material in various languages. Good contact point for that is events@debian.org, or one of the debian-events-* mailing lists on lists.debian.org. If you need help, ask for it. Also, you can help Debian manning an exhibition. If you see events in your area, offer to help and don't be shy. Other way to help is working with the publicity team and prepare press announcements, the Debian Project News (DPN), contact journalists or press media if interesting things happen in Debian.

Help Debian to organize stuff
There are many ways to help Debian organize itself. For example the annual Debian Conference DebConf is a big organisation monster, and you don't need to be developer to help with that. Sometimes it's as easy as helping in the video team taping the conference, help at the front desk with registration, sorting badges or speaking to the caterer about needed foods. We also have miniDebConf or so called Debian Bug Squashing parties from time to time. Your company could provide office rooms, you could provide crash space for developers to sleep or even by sponsoring some beverages or food. Also helping around exhibition is a good idea. If your company is willing to print some flyers or posters this can help us.

Help by translating or writing documentation
Debian's website and all of the software Debian delivers should be available in all languages around the world! Good starting point for that is http://www.debian.org/international/ and the Debian Internationalization Mailing list. Also writing or extending documentation is a job everyone can do. If you are using a piece of software heavily and miss documentation, speak to the Debian package maintainer (you can find our at http://packages.debian.org/$yourpackage) and start submitting bugs with documentation.

Help by donating
There are actually many ways to help by donating (not only by money). Surely Debian will accept money donations via one of it's official representation (ffis, SPI, debian.ch, ...). On the other hand donating can be as simple as allowing your employees to work some specified time on the week on Debian! Or you donate machine hardware (probably not your old ones that you used five years and which are not under warranty now any more, sorry...), bandwith or colocation in your datacenter. Speak to the hardware donations team if you want to know the current needs.

Conclusion
I only wrote down a very few areas where you can help Debian, and there are plenty more! Don't hesitate to jump in to help. If you don't understand stuff: ask! But be prepared that you will be pointed to URLs where the stuff you ask for is documented. Helping Debian sometimes starts with reading tons of documentation (and i am sure you will find errors in that documentation to fix!), but after a while it makes a lot of fun to work for and with Debian. Find your own area to work on within Debian, and don't think you can't help. Even graphic designers, lawyers or clerks can help Debian!

I started using Debian around 15 years ago and became Debian Developer around 6 years ago. Within the last six years I had been in various positions inside Debian (listmaster team member, volatile team member, release team member and Stable Release Manager, Debian Sysadmin Team member) and got to those just by jumping in where help needed.


Wouter Verhelst: Nothing Else Matters

The office is located not too far from a church (dutch) which contains a fully-functional carillon. For one thing, this means that during day hours, once every fifteen minutes the bells play a little bit of music; for another, it also means that once every so often, someone sits behind the carillon's keyboard, and plays some live music. This is often done in the afternoon between 14:00 and 15:00 on weekdays, when I'm at work. I've grown to like this. Usually the music is classic carillon music; but quite often they add in something unexpected. Today, for example, I was surprised to hear a carillon version of Nothing Else Matters, which sounded surprisingly nice. I've been thinking off and on to place a microphone on the roof of the office, so that I could stream the carillon concerts to the Interwebz. It hasn't happened yet, mainly because it would require me to get organized a bit more than I currently am, but perhaps I might actually go through with it at some undeterminate point in the future... oh well.

20 June 2011

Craig Small: psmisc 22.14 Released

The upstream and debian packages for psmisc version 22.14 were released tonight. This version has a lot of bugfixes. I ve also taken some time to clean up some of the minor compile warnings I used to get which were mainly variables set but not used. Two bugs that I ll concentrate on getting fixed for the next version are all about the command line. The first is that the fuser -s flag changes the return code for items found. It should be consistent so it doesnt matter if fuser displays values or not, 0 means found something and 1 means didn t find it. I can see why in the code it might be different, because the code follows different paths with the s option, but not exactly where the problem is. The second bug is the -M flag for fuser only works if it is before the mount points so -M -m /foo works but -m /foo -M does not. Unlike the -s problem, I can see where it is failing. Currently fuser scans the options one by one. If it finds a -m option, it checks at that point if the -M option is set. The solution is to store all potential targets and only after exhausting the entire command line do we add the potential targets to our real targets. It means scanning the command line, or a subset of it, twice but the performance penalty is tiny. Finally, I released there was a rather neat Makefile command make distcheck . This target makes a tar.g archive, extracts it, builds then cleans. It is a very good program for trapping things like a file exists in your local directory but is not included in the tar.gz (lists.h was missing until I ran this). For some make distcheck is something they re known for years, but if you have an autotools package you develop, have a look at it. With all the talk about testing and test driven releases, I was going to put some test cases in. The problem is there doesn t seem to be an easy way to do testing. I might have to just hack up some scripts but surely there is a better way. That would trap, for example that fuser -s returns 1 but fuser returns 0. I looked at check, but it doesn t work for single-file programs like psmisc is mainly made of.

Rudy Godoy: Compute Clusters Integration for Debian Development and Building Report 2

Hello, this is the 2nd report regarding my project. I'll offer a brief
summary of how are we doing and then the details, so you don't have to
read everything at all.
I have to start saying that over the last two weeks I made a lot of
progress on the project. The most important part: Debian ARM image
integration with Eucalyptus is now cleared. The work to be done for the
next two weeks is pretty much fully coding to achieve what we've committed:
supporting Debian ARM images for Eucalyptus.
What was done so far
--------------------
- I've have identified and mapped what needs to be modified over the
  eucalyptus source for supporting ARM. I'm puting the details on the
  next section. I think this is the most crucial part that is now
  sucessfully achieved.
- I started a Project Log[1] on the wiki, so anyone can check and
  eventyally use the information I've already gathered. The log is being
  updated constantly. You can also check what I've done and what I'm
  working on.
- We got feeback[2] from Eucalyptus developers and packaging
  team regarding developers and packaging team regarding the ultimate
  source for debian/ packaging bits. This was quite important because
  in the past I've managed to build Eucalyptus from source using the debian/
  included in the upstream tarball. However the people in charge
  told us that they were using a different one, which indeed was
  different and resulted in different output than I expected. I'm
  still working on fixing the bits to get a clean build.
- I've joined the pkg-eucalyptus team and I've been given commit rights
  to the repo[2].
1- http://wiki.debian.org/SummerOfCode2011/BuildWithEucalyptus/ProjectLog
2- http://lists.alioth.debian.org/pipermail/pkg-eucalyptus-maintainers/2011-June/000289.html
What I'll be working over the next two weeks
--------------------------------------------
- Heavily code on the different bits that I've identified and need to be
  changed. They are:
  C code:
  - node/handlers_kvm. c,h  - handles nc creation. nc's are node
  clusters for Eucalyptus, the code is in charge of node creation. It
  prepares the "machine" that is able to run a compatible image.
  - util/data. c,h  - handles instance creation. The code over there
  implements the functions node/handlers_* use, like instance creation
  and libvirt's API parameter passing. Since libvirt's parameters for
  ARM vary from x86's ones, we need to support that.
  Perl code:
  - tools/gen_kvm_libvirt_xml - wrapper tool for creating KVM/QEMU
  libvirt's XML Domain definition. This needs to be heavily
  changed/updated - redesigned, to support ARM architecture.
- Depending on the previous task progress: Modify the Debian packaging
  bits for Eucalyptus and push them to the pkg-eucalyptus's team
  repository. This is not crucial for my project's success but I'd like
  to work with Debian official builds. I've yet to commit the
  Eucalyptus team version to the debian/ dir. I expect to have fixed
  the bits to get a successful and clean build, then commit.
Other plans
-----------
- I expect a slow down on the work pace over the second week due end of
  semester exams on the week from July 3 to July 9. After that week I'm
  on vacations and will be fully dedicated to the project.
- Code changes to Eucalyptus source will be pushed upstream. I would
  not like to keep a separate repo for this. I'll talk about this with
  Steffen. Changes to Debian bits will be pushed to the appropiate repos.
Other bits
----------
When the project started I noted it was too vague for some people and
it's results/output were unclear. This concerned me. However, while the
project started and we got lot of feedback things got much clear and I'd
like to take the opportunity to explain it a bit.
Currently, developers targeting ARM devices are struggling[1] with having
a development environment. There are missing bits that sometimes are not
in place and result on hard time for them. Also, due mobile devices
growth and vendors using Linux-based[2] operating systems it's a nice
opportunity for Debian to keep it's well known fame of: supporting every
arch over the sun. How this project fits, well by offering the ability
for developers to run ARM-based Debian images over x86/x86_64 CPU's
would impact the way they could be more productive and for Debian to be
more adopted as development environment.
I can say that I'm very enthusiast of this opportunity and I'm
certain that the outcome will result in more adoption. Even the Ubuntu
people is now pushing more and more efforts for ARM official support,
due the facts I've stated.
1- http://lists.debian.org/debian-arm/2011/06/msg00020.html
2- http://events.linuxfoundation.org/events/android-builders-summit/back

19 June 2011

Mesutcan Kurt: Python Multi-Build for Python Extensions Packaging - Report 2

Hello,

This is second report of my GSoC project for Debian. Since my last
report I have done most of things about my project's design. Me and my
mentor are thought about projects API, design etc. this two weeks. Since
preparing API is the most important task of this project two week is not
a long time, and I don't say that API design is over. According to your
comments or requests I can change the API.

If you want to see how the tool's API look like or tool's source code you
can clone project's git repository at [1]. In repository, under template/
directory you can see that.

Under the directory you will see one file and one directory. Only file is
the tool's main file, called py3build. It is a python3 script. It will
build extensions for Python3.

Firstly It will parse arguments. Arguments are:

--system args: (extension will be build for which build system)
--test: (for only test the extension)
--no-clean: (don't try to call clean method)
--configure-args args: (extra configure args)
--build-args args: (extra build args)
--test-args args: (extra test args)
--quiet: (quiet mode)
--verbose: (verbose mode)
--python-version args: (build for that version)
--build-directory args: (build in given directory)

If there is no given Build System then It will try to get Build System
list. This list will be generated from plugins/ directory which you can
find under my git repository's template/ directory.

I want to describe what will BuildSystem's look like.

Under plugins/ directory you will see __init__.py file. It will contain
base BuildSystem class, main Exception class, and other method specific
Exception classes. (f.e. ConfigureError) Main Exception class will be
derived from Exception class and other Exception classes will be derived
from this class, called Error.

Base BuildSystem will be derived from all other Build System classes. It
has __init__ method which gives build_dir, configure_args, build_args,
test_args for parameter. It will set them to self. parameter_name . So,
if they will be needed from another method, can be retrieved from
self. parameter_name , no need to pass every method.

One of the important BuildSystem class method is detect method. This
method will detect the extension's build system. If given directory has
files for building extensions for this build system (or other magical
detecting ways) it will return true.

After that with configure method we can configure extension for the build
system. This method and other methods in build system classes' will also take version parameter. This parameter will
determine that this extension will be build for which Python version. If there is
any error occurs then it will raise ConfigureError exception.

Clean method will cleans (f.e. removing build files) given directory for
that build system. If there is any error occurs then it will raise
CleanError.

Build method will build extension for the build system. It will use
self.version to determine which version it will be built for. If there is
any error, then raises BuildError.

Last method is called test. This method will test the extension for
self.version Python version. If there is any error occurs then raises
TestError.

Every build system plugin should have one class, same name with file. The
class should extend BuildSystem class and override this class' methods
for working correctly.

I also wrote an example plugin, called example. For show how it will be done (It is under
plugin/ directory under project's git repository).

py3build's get_build_system method will retrieve a BuildSystem list. This
method will be used by determine_build_system method. That method will
determine which build system is usable for this extension. After that
this method also returns a list. List is in order to higher probablity to detect.

Main method will invoke clean_for_build_system with a BuildSystem object
argument for clean for that BuildSystem, configure_for_build_system with
a BuildSystem object argument for configure for that BuildSystem,
make_for_build_system with a BuildSystem object argument for build for
that BuildSystem and test_for_build_system a BuildSystem object argument
for test for that BuildSystem. The call sequence is like that.

Every method will return True if everything is OK.

We are still working on API and this one isn't completed, and
discussing about various problems.

[1] git://git.debian.org/~mesutcan-guest/py3multibuild.git

18 June 2011

Lars Wirzenius: Chunkwise Debian package downloads

Note: This is an idea I had, not something I'm working on. If it interests you, please work on it. I may come back to this some day, but not anytime soon, but I thought I'd share it anyway. When downloading updates for a Debian package, often much of the data doesn't actually change, or at least not much. Because of this, the debdelta service is a wonderful idea (and Debian should adopt it officially). However, a different way of achieving that might be like this: When downloading updates, apt would look up the chunks ids and checksums, and see if it already has such chunks installed locally. If so, it can reuse those chunks. (Note that apt can look for chunks in the installed files, and does not need to cache things separately.) Now, I have not done the research to verify that this is actually worthwhile. Someone should experiment with different sizes of chunks to see if enough bandwidth could be saved to make this worthwhile, in an upgrade from one Debian release to another (lenny to squeeze, for example). Any takers?

Guillaume Mazoyer: Status report Jigsaw num. 2 for GSoC 2011

code.pngMy internship is now finished, this means that I will be able to work on my GSoC project a lot more than the previous weeks. I hope to make some good progress in the next days.

This last 2 weeks, I was still studying how Jigsaw works. Even if there is no progress on the Jigsaw packaging I have done some packages that will help later.

I ve spend a lot of time trying to package JT Harness and JTReg.
I ve successfully built JT Harness using CDBS but I ve finally rewritten the package using only debhelper and javahelper. Javahelper was really useful to build JAR files, move them to the right directory, put the Javadoc in the right place as well as the examples. The Debian directory of JT Harness can be found on the Java team SVN.

svn co svn+ssh://$ SVN_USER @svn.debian.org/svn/pkg-java/trunk/libjtharness-java

The source package of jtharness build 2 binary packages: libjtharness-java and libjtharness-java-doc. You can see that by looking at the control file in the debian directory. There are some interesting files in this package, let s take a look at them.

First of all, some of the source files needed to be patched for the package to build. So to accomplish this, there is a couple of patches in debian/patches : To generate these patches, I ve used the edit-patch command which is a really awesome tool. This command uses a temporary shell where we can edit files we want to patch. When the job is done, CTRL + D terminates the temporary shell and the patch is generated.

There are 4 interesting files which are used to build the binary packages and put the files in the right directories. These files are: Of course rules is the most important one. This file tells us how packages are built. A good thing is that rules is quite simple in the jtharness case.

#!/usr/bin/make -f
JAVA_HOME = /usr/lib/jvm/default-java

override_dh_auto_build:
ant -f build/build.xml

%:
dh $@ --with javahelper


We just tell where Java is and to use javahelper ( with javahelper). Overriding dh_auto_build is interesting because since the JT Harness uses Ant as build system we need to give to Ant where the build.xml file is. By default Ant is searching it in the current directory but in jtharness the file is in the build directory. Since the Ant build file is correctly written, the build generates JAR files, Javadoc and examples code. And libjtharness-java* files are here to put the generated things in the right places.
The job of libjtharness-java.jlibs is to tell where to find the JAR libraries. In this case, the intesresting JAR files were javatest.jar and jt-junit.jar . So in the .jlibs file we just give where those files are after being built.
The job of libjtharness-java-doc.javadoc is to tell where the Javadoc is and where to put it. So there is only one line that says the generated Javadoc is in jar-build/javadoc and put it in /usr/share/doc/libjtharness-java/api .
And finally, libjtharness-java-doc.install is used to move the examples. The syntax is same as libjtharness-java-doc.javadoc .

The JT Harness package can be found in Sid now thanks to Sylvestre Ledru who uploaded it for me. So the ITP bug that I created is now closed.

Building the package for jtreg is quite similar. In fact, it probably easier but I m still facing a problem. jtharness is just a library, jtreg does have main classes and generate some binaries. The difficulty here is to generate the man pages for the 2 generated binaries: jtdiff and jtreg. These 2 files are shell scripts which just use jtreg.jar with the correct parameters.

I tried several tools to generate the man pages of each script. Sadly, I can t find a way that does not causes any problems.
My first choice was to use help2man. This tool looked like to be perfect for the job. But I couldn t find how to launch the scripts correctly. The scripts needs jtreg.jar but they also need javatest.jar from the libjtharness-java package. The classpath cannot be set correctly due how the scripts are written. So to use help2man I ll probably have to patch and completely rewrite jtdiff.sh and jtreg.sh in the sources.
My second choice was to use txt2man. During the build of jtreg there is a usage.txt file which is generated and which can be used as a man page. txt2man can take this text file and output a man page based on the text. I thought it was working but when building the package, lintian threw some warnings about syntax errors in the man pages.

W: jtreg: manpage-has-bad-whatis-entry usr/share/man/man1/jtdiff.1.gz
W: jtreg: manpage-has-errors-from-man usr/share/man/man1/jtdiff.1.gz 292: warning [p 4, 3.3i]: cannot adjust line


Finally, few days ago I decided to try rst2man. This tool looks cool and easy to use. But the text that I have to convert into a man page is not a reStructured text so it cannot without patching the usage.txt file for example. I was going to do it but there is no such file in the sources of jtreg. Actually this file is generated during the build but the text had to come from somewhere. And it was from two properties files in the source code.
I m still looking for a good solution to generate the man pages. If someone has any idea just let me know ;-)

The debian packaging of jtreg is not in the Java team SVN yet because of the man pages problem. I opened an ITP bug so people know that someone is working on jtreg.

My goal for the next two weeks is to finish the jtreg package, understand how jpkg (which is supposed to create Debian packages) works in Jigsaw and write more examples using the modular JVM.

17 June 2011

John Goerzen: APRS: World s Best Social Mapping and Wide-Area Ad-Hoc Wireless Mesh Network

That was quite a headline, and I m going to try to back it up below. APRS is the Automatic Packet Reporting System. It s a system for exchanging brief packets of information. It is most frequently used for mapping applications, but it really does a lot more than that. It has its biggest home in the amateur radio world, but isn t limited to that, either. The most common way to use APRS is to have some device hooked up to a GPS transmit packets with the GPS information in them. These packets can then be plotted on a map in real-time or with history. That in itself isn t particularly newsworthy these days. An interesting thing about APRS is that it s not just positioning. Let s say that there was a search-and-rescue operation. A person could draw a rectangle on the map indicating the search area, and within about 3 seconds everyone else s map also shows that rectangle. People have even been known to play chess by sharing and moving objects on APRS! The next piece that makes this interesting is that APRS is an ad-hoc mesh network. In its traditional implementation, VHF amateur radio, a radio emits a packet with a geolocation in it (a beacon ) and any other radio within direct range of that can receive it. Radios can display basic information (such as distance to the other radio, heading, etc.) or hooked up to a laptop or mapping device for a better display. So if everyone is within a few miles, APRS works without any pre-existing infrastructure at all. This makes it wonderful for use in disaster areas, and was put to heavy use in Joplin after the tornado there. But what about radios that are too far away? Any APRS station could also be a digipeater. When a packet is transmitted, it has a maximum hop count. A digipeater hears the packet, decrements the maximum hop count, and re-transmits it. With this mechanism, packets can travel hundreds of miles. It creates a highly resilient network, one that can route around trouble without even having to have an explicit backup route. I could bring in a digipeater in my car it can be small enough to hold in my hand and instantly improve APRS reception in an area. One interesting aspect is that packets can be digipeated more than the maximum hop count. For instance, if a packet leaves my radio and is picked up by a digipeater to my west and one to my east, it can keep on traveling in both directions. This is part of what leads to resiliency. APRS also functions over the Internet. There is a large network of interconnected Internet servers that exchange all global APRS traffic amongst themselves. Gateways between the radio (RF) and Internet (APRS-IS) services exist, and are called iGates. They are not generally required, but make useful websites like aprs.fi and email gateways possible. As long as an iGate is within a reasonable number of hops from you, you re effectively linked. And again, if one iGate drops off, another iGate is probably monitoring your traffic too and you never notice. It s an ad-hoc mesh network that is actually reliable how about that? On the PC side, there are many programs for using APRS. The most common one for Windows is called UI-View, but I don t use Windows so I can t comment. On Linux, there are programs (such as aprx) for running your digipeater, but the best-known program is Xastir. Xastir lets you download map files to your local disk, and can interface with the APRS-IS Internet service, radios, weather stations, or simply other arbitrary machines to exchange information. Xastir is a very nice program and is well worth the install, despite its somewhat dated-looking interface. APRS clients, such as APRSDroid, exist for Android and iOS platforms as well. So let s say you re doing something like helping handle food/water stations for a long bike ride. Even if you don t have anybody with an amateur radio license, you can use APRS to great effect. At your headquarters, you can run Xastir and turn on its server mode . This puts everyone on a map. Then you can have everyone turn on APRS on their phone, and have it report to your custom server instead of APRS-IS. Now you have instant visibility into your entire team s location and status. If you have transport people driving supplies between locations, that s especially helpful. In an amateur radio scenario, you would instead have people with radios at each location, and one laptop hooked up to a radio at HQ. This provides an added bonus of not relying on third-party infrastructure such as cellphone towers. APRS also has a messaging system, similar in concept to text messaging. It works the same as other things. If I want to send a message to Jane, my radio simply emits a packet that lists the message and Jane as a recipient. It s digipeated up to its maximum hop count. If Jane is within RF range of one of those digipeaters, she gets the message and her radio ACKs it. Otherwise, it s delivered into the APRS-IS network probably several times, which isn t a problem and the APRS-IS network delivers it to the iGate closest to her, and from there it gets digipeated the rest of the way to her. Here s an example of something created with APRS. While I was on a bus choir tour last weekend, I had a radio with me that was beaconing all the while. Now it was a small handheld radio inside a large metal bus, so it didn t always have a digipeater in range. But still, you can go see a detailed map with the trail and even see exactly what path each packet took before it hit the Internet. If you want to try out Xastir, please grab at least version 2.0 the version in squeeze has some bugs.

Timo Jyrinki: Why I don't like blogger.

Hmm, I think it's happening again. I changed the Planet Ubuntu to only feed on posts tagged 'ubuntu' from now on, and now as an example tagged one post as 'ubuntu', in addition to this one I'm now writing. Google told me that using "Label actions" to change labels wouldn't affect the feed, but it looks like the feed timestamp for my previous, older post was again updated (not the "Published", but "Updated")... so I guess I won't be tagging any further old posts, unless it turns out the previous problem I had was that the Published stamp was also renewed and the Planet Ubuntu ignores Updated stamp.

If you have a solution, please tell it.

Update: Solution found! Thanks, artfwo.

Timo Jyrinki: Kernel Mode-Setting (KMS) on Neo FreeRunner + Debian

Edit Aug 18th: video available (ogv, also on youtube)

Forget AMD, Intel, NVIDIA! Thomas White's incredible work with Neo FreeRunner's puny graphics dec...accelerator, Smedia Glamo 3362 is starting to bear fruits. Not listening to such comments as the chip will never be used anywhere else and why not spend your time doing something more relevant , he has chosen to actually do what he likes and sees as an interesting challenge. And that's often the spirit of free software, so I don't really agree with the naysayers.

The camera shot on the right, running KMS-enabled X.org driver for the glamo chip on my Debian installation (visible software matchbox-window-manager, fbpanel, zhone), is a bit optimistic looking since Zhone happens to draw correctly. A lot of the drawing is not yet synced correctly, which shows as all text and images in eg. GTK applications being garbled. But as little as two days ago one couldn't yet much launch applications without X crashing, so the newest commits by Thomas were a big step forwards. I'm using Debian, and he's not, so I try to find time to help in debugging even though I really can't much help with the driver code.

The driver is not just one piece of code, but consists of a kernel drm driver (direct rendering manager) using GEM, libdrm support for the kernel driver and finally the X.org driver supporting these other components and offering buzz-words like DRI2. There is also a beginning of a Mesa 3D driver, though it is so far just a skeleton driver since the 2D/KMS/EXA/DRM parts are what should be done first before dwelling into the OpenGL realm.

Openmoko the company is basically nowadays just producing new Neo FreeRunners to resellers, but the community of the so far only Free phone is thriving. The final (Openmoko produced, from where gta02-core continues) version of the phone, with the famous buzzing problem fixed, appeared on store shelves some time ago, offering better out-of-the-box phone functionality in addition to all the mini computer features. It was also offered to Debconf visitors on discount.

It's not only cool to have kernel mode-setting, though it is indeed very cool as well. The reason for much sorrow in the whole OM project has been the graphics chip, and some of the problems with the chip are only finally solvable with the kernel doing the mode-setting. So it's both a very modern thing to use KMS, but there are also clear potential benefits of having accurate control over this very sensitive piece of silicon. The current non-KMS X.org driver for example busy loops in order to try to feed the chip at a correct pace, causing CPU usage every time there is any drawing going on.

This is somehow reminding me about getting the best out of 8-bit computers and other hardware with some specific limitations. The glamo chip is very limited in some ways, but also capable in some other ways like theoretically offering MPEG-4 decoding, OpenGL ES 3D support. It's a mixed bag of things, and I can very well imagine it's indeed an interesting challenge to work on it, if you just have the skills (I don't) and nerves (proper debugging tools help).

It's so nerdy to drool over lines in the X.org log, but I just can help it:
X.Org X Server 1.6.3
Release Date: 2009-7-31
X Protocol Version 11, Revision 0
Build Operating System: Linux 2.6.28 armv5tel Debian
Current Operating System: Linux neo 2.6.29-GTA02_mydrm #2 PREEMPT Mon Aug 17 18:24:39 EEST 2009 armv4tl
...
II) Using KMS!
...
(WW) Glamo(0): EXA hardware acceleration initialising
(II) EXA(0): Driver allocated offscreen pixmaps
(II) EXA(0): Driver registered support for the following operations:
(II) Solid
(II) Copy
(II) Glamo(0): Initialized EXA acceleration
...
(II) Glamo(0): RandR 1.2 enabled, ignore the following RandR disabled message.
...
(II) Glamo(0): [glamo-dri] Name of DRM device is '/dev/dri/card0'
(WW) Glamo(0): [DRI2] Version 1 API (broken front buffer rendering)
(II) Glamo(0): [DRI2] Setup complete
...
(II) AIGLX: enabled GLX_MESA_copy_sub_buffer
(II) AIGLX: Loaded and initialized /usr/lib/dri/glamo_dri.so
(II) GLX: Initialized DRI2 GL provider for screen 0
(II) Glamo(0): Adding framebuffer....!
(II) Glamo(0): 8 480 640 16 16 960
(II) Glamo(0): rootPixmap = 0x1d0a00
(II) Glamo(0): Done

Timo Jyrinki: Ubuntu 10.04 and 3G modems usb-modeswitch or not?

I'm writing a blog entry instead of just replying to myself on a mailing list, since this subject might be interesting to a largish portion of the community. Most people assume things just magically work out by themselves and then after a release wonder why it wasn't so.

So, I raised a question on ubuntu-devel-discuss mailing list about whether usb-modeswitch should be included in the default Ubuntu 10.04 LTS installation instead of it only being in the universe repository. Reason was simply that firstly, I've read some general wonderings from the community about why it isn't already so. Secondly, without it my 3G modem (Huawei E1552) was first not functional, but right after installing usb-modeswitch package Network Manager worked smoothly with it.

Since I didn't get much discussion going on besides some statements that it doesn't work for everybody and even has caused extra problems for some others (in 9.10), I asked for a small round of comments on the subject on IRC. The channels are logged so I believe I can quote those a bit.

Colin Watson said among else:
"surely we just want the kernel to DTRT [do the right thing] by default ... this is the upstream trend ... it already DTRT for quite a few devices"
"I'd be very concerned about advertising that it's the Way to get things to work, and thus undermining getting things fixed in the kernel"
"I agree that usb-modeswitch is often a way to get otherwise non-working hardware to work ... I'm just not very convinced it's a real properly supportable option"
"but I'm just another user from this point of view, albeit one who ended up in quite a few discussions with various appropriate upstreams last time round :)"
Later on Paul Sladen continued a bit, among else:
"usb-modeswitch is a very long-winded way of sending a single usb-mass-storage command to the device's first profile"
"grep -hr MessageContent usb_modeswitch.d/ sort -n uniq -c sort -rn shows the level of duplication in the configuration files"
"...I do like that it is done in userspace though, as so easy to disable if you wanted something else".
Meanwhile I also IM:d a bit with Antti Kaijanm ki from whom I probably originally coined the idea of following whether usb-modeswitch is integrated into Ubuntu or not. He mainly said that he doubts the feasibility of updating kernel's USB storage quirks in a stable release, compared to having stable release upgrades about the usb-modeswitch-data udev rules. He also proposed discussing maintainability and usability by distros with the Debian maintainer and upstream, although it sounds like Colin might have had some talks already.

So there we are now. Apparently at this point it is hoped that as many quirks as possible are inserted in the kernel, see for example drivers/usb/storage/unusual_devs.h in the kernel. But if you have concerns about if 10.04 LTS will be kept up-to-date regarding 3G modems and want to somehow participate in bringing usb-modeswitch into Ubuntu default installation or simply discuss how to handle the kernel SRUs properly, it's time to stand up and do something.

Should there perhaps be a process with which the usb-modeswitch developers would get the information they are gathering more easily to the upstream kernel _and_ (older) distribution kernels? Is this simply a case of being easier to address an issue with a "hackish" approach, or is the usb-modeswitch actually the right way to go?

Note that I'm no expert in this area, I simply become interested in various subject from time to time :)

-Timo, going to figure out a kernel quirk for his 3G modem

13 June 2011

Vincent Bernat: Anycast DNS

In a recently published article, Paul Vixie, past author and architect of BIND, one of the most popular internet domain server, explains why DNS servers should use anycast for increased reliability and faster answers. The main argument against unicast is that some resolvers may not be smart enough to select the best server to query and therefore, they may select a server on the other side of the planet. Since, you cannot go faster than light, you get a penalty of about 100 milliseconds in this case.

Unicast & anycast Most addresses that you handle are unicast addresses, i.e. the address of a single receiver. Any time you send a packet to an unicast address, it will be forwarded to this single receiver. Another common network addressing system is multicast (which happens to include broadcast). When you send a packet to a multicast address, several receivers may get it. The packet is copied as needed by some nodes. Anycast addresses allow you to send a packet to a set of receivers but only one of them will receive it. The packet will not be copied. Usually, the idea behind anycast is that the packet is sent to the nearest receiver. There is no universal way to determine if an IP address is an unicast address or an anycast address just by looking at it. For example 192.5.5.241 is an anycast address while 78.47.78.132 is an unicast address. ISC, which maintains BIND, operates the F root domain server, one of the 13 Internet root name servers. This server is queried by using 192.5.5.241 IPv4 anycast address or 2001:500:2f::f IPv6 anycast address. ISC has published a document explaining how anycast works. Several nodes around the globe announces the same subnet using BGP. When one router running BGP has to send a query to the anycast address, it will have several choices in its BGP table. It will usually select the shortest path and route the query to the corresponding router. Some nodes may choose to restrict the announcement about the subnet containing the anycast address. They are called local nodes (by opposition to other global nodes). This allows them to serve query only from their clients and use cheaper connectivity and hardware.

Lab Let s play with anycast by setting up a lab to experiment with it. We want to setup an IPv6 anycast DNS service using two global nodes, G1 and G2, and one local node, L1. We build some kind of mini-Internet. See : Anycast lab

Setting up the lab This lab will use UML guests for all nodes. For more details on how this works, look at my previous article about network labs with User Mode Linux. You first need to grab the lab:
$ git clone https://vincentbernat@github.com/vincentbernat/network-lab.git
$ cd network-lab/lab-anycast-dns
$ ./setup
You will need to install the dependencies needed by the script on your host system. The script was tested on Debian unstable. If some of the dependencies do not exist in your distribution, if they don t behave as in Debian or if you don t want to install them on your host system, you can use debootstrap to create a working system:
$ sudo debootstrap sid ./sid-chroot http://ftp.debian.org/debian/
[...]
$ sudo chroot ./sid-chroot /bin/bash
# apt-get install iproute zsh aufs-tools
[...]
# apt-get install bird6
[...]
# exit
$ sudo mkdir -p ./sid-chroot/$HOME
$ sudo chown $(whoami) ./sid-chroot/$HOME
$ ROOT=./sid-chroot ./setup
We are running a lot of UML guests in this lab. We assign 64 MB to each host. UML guests do not use host memory until they need it but because of cache subsystem, you can expect that each guest will use 64 MB of memory. This means that you need about 2 GB of RAM to run this lab. Moreover, each guest will create a file representing its memory. Therefore, you need about 2 GB of free space in /tmp too. Routers are named after their AS number. The router of AS 64652 is just 64652. Paris, NewYork and Tokyo are the exception. The lab will use only one switch but we will use VLAN to create several L2 networks. Each link in our schema is an L2 network. Each link will get its own L3 network. Those networks are prefixed with 2001:db8:ffff:. Leaf AS have their own L3 network 2001:db8:X::/48 where X is the AS number minus 60000. Look at /etc/hosts on each host to get the IP of all hosts.

Mini-Internet routing AS 64600 is some kind of Tier-1 transit network (like Level 3 Communications). It has three points of presence: Paris, New York and Tokyo. AS 64650 is an European regional transit network, AS 64640 is an East Asian regional transit network and AS 64610, 64620 and 64630 are North American regional transit network which are peering each other. Other leaf AS are local ISP. This is not really what Internet looks like but shaping our lab like a tree allows us to not deal with a lot of BGP stuff: we ensure that the shortest path (in number of BGP hops) from one client to one server is the best path. Each BGP router will get a basic BGP configuration containing only the list of peers. Since Paris, Tokyo and NewYork are in the same AS, they will use iBGP. They are fully meshed for this purpose but this does not provide redundancy because iBGP routes are not redistributed inside iBGP. A more complex setup would be needed to provide redundancy inside AS 64600. We don t use an IGP inside leaf AS as we should: for the sake of simplicity, we redistribute directly connected routes into BGP. While in my previous lab, I used Quagga, I will use BIRD in this one. It is a more modern daemon with a clean design on how routing tables may interact. It also performs better. However, it lacks several features from Quagga. The configuration for BIRD is autogenerated from /etc/hosts. We use only one routing table inside BIRD. This table will be exported to the kernel and to any BGP instance. Some directly connected routes will be imported (those representing client networks) as well as all routes from BGP instances. Here is an excerpt of 64610 s configuration:
protocol direct  
   description "Client networks";
   import filter  
     if net ~ [ 2001:db8:4600::/40 40,48  ] then accept;
     reject;
    ;
   export none;
 
protocol kernel  
   persist;
   import none;
   export all;
 
protocol bgp  
   description "BGP with peer NewYork";
   local as 64610;
   neighbor 2001:db8:ffff:4600:4610::1 as 64600;
   gateway direct;
   hold time 30;
   export all;
   import all;
 
We do not export inter-router subnets to keep our routing tables small. This also means that those IP addresses are not routable in spite of the use of public IP addresses. Here is the routing table as seen by 64610 (use birdc6 to connect to BIRD):
bird> show route
2001:db8:4622::/48 via 2001:db8:ffff:4610:4620::2 on eth3 [bgp4 18:52] * (100) [AS64622i]
                   via 2001:db8:ffff:4600:4610::1 on eth0 [bgp1 18:52] (100) [AS64622i]
2001:db8:4621::/48 via 2001:db8:ffff:4610:4620::2 on eth3 [bgp4 18:52] * (100) [AS64621i]
                   via 2001:db8:ffff:4600:4610::1 on eth0 [bgp1 18:52] (100) [AS64621i]
2001:db8:4612::/48 via 2001:db8:ffff:4610:4612::2 on eth2 [bgp3 18:52] * (100) [AS64612i]
2001:db8:4611::/48 via 2001:db8:ffff:4610:4611::2 on eth1 [bgp2 18:52] * (100) [AS64611i]
As you can notice, it knows several paths to the same network, however, it only selects the best one (based on the length of the AS path):
bird> show route 2001:db8:4622::/48 all
2001:db8:4622::/48 via 2001:db8:ffff:4610:4620::2 on eth3 [bgp4 18:52] * (100) [AS64622i]
        Type: BGP unicast univ
        BGP.origin: IGP
        BGP.as_path: 64620 64622
        BGP.next_hop: 2001:db8:ffff:4610:4620::2 fe80::e05f:a3ff:fef2:f4e0
        BGP.local_pref: 100
                   via 2001:db8:ffff:4600:4610::1 on eth0 [bgp1 18:52] (100) [AS64622i]
        Type: BGP unicast univ
        BGP.origin: IGP
        BGP.as_path: 64600 64620 64622
        BGP.next_hop: 2001:db8:ffff:4600:4610::1 fe80::3426:63ff:fe5e:afbd
        BGP.local_pref: 100
Let s check that everything works with traceroute6 on C1:
# traceroute6 G2.lab
traceroute to G2.lab (2001:db8:4612::2:53), 30 hops max, 80 byte packets
 1  64652-eth1.lab (2001:db8:4652::1)  -338.277 ms  -338.449 ms  -338.502 ms
 2  64650-eth2.lab (2001:db8:ffff:4650:4652::1)  -338.324 ms  -338.363 ms  -338.384 ms
 3  Paris-eth2.lab (2001:db8:ffff:4600:4650::1)  -338.191 ms  -338.329 ms  -338.282 ms
 4  NewYork-eth0.lab (2001:db8:ffff:4600:4600:1:0:2)  -338.144 ms  -338.182 ms  -338.235 ms
 5  64610-eth0.lab (2001:db8:ffff:4600:4610::2)  -338.078 ms  -337.958 ms  -337.944 ms
 6  64612-eth0.lab (2001:db8:ffff:4610:4612::2)  -337.778 ms  -338.103 ms  -338.041 ms
 7  G2.lab (2001:db8:4612::2:53)  -338.001 ms  -338.045 ms  -338.080 ms
Let s see what happens if we shut the link between NewYork and 64610. This link is VLAN 10. Since vde_switch does not allow to shut a port, we will change the VLAN of port 9 to VLAN 4093. After at most 30 seconds, the BGP session between NewYork and 64610 will be torn down and packets from C1 to G2 will take a new path:
# traceroute6 G2.lab
traceroute to G2.lab (2001:db8:4612::2:53), 30 hops max, 80 byte packets
 1  64652-eth1.lab (2001:db8:4652::1)  -338.382 ms  -338.499 ms  -338.503 ms
 2  64650-eth2.lab (2001:db8:ffff:4650:4652::1)  -338.284 ms  -338.317 ms  -338.151 ms
 3  Paris-eth2.lab (2001:db8:ffff:4600:4650::1)  -337.954 ms  -338.031 ms  -337.924 ms
 4  NewYork-eth0.lab (2001:db8:ffff:4600:4600:1:0:2)  -337.655 ms  -337.751 ms  -337.844 ms
 5  64620-eth0.lab (2001:db8:ffff:4600:4620::2)  -337.762 ms  -337.507 ms  -337.611 ms
 6  64610-eth3.lab (2001:db8:ffff:4610:4620::1)  -337.531 ms  -338.024 ms  -337.998 ms
 7  64612-eth0.lab (2001:db8:ffff:4610:4612::2)  -337.860 ms  -337.931 ms  -337.992 ms
 8  G2.lab (2001:db8:4612::2:53)  -337.863 ms  -337.768 ms  -337.808 ms

Setting up anycast DNS Our anycast DNS service has three nodes. Two global ones, located in Europe and North America and one local one targeted at customers of AS 64620 only. The IP address of this anycast DNS server is 2001:db8:aaaa::53/48. We assign this address to G1, G2 and L1. We also assign 2001:db8:aaaa::1/48 to 64651, 64612 and 64621. BIRD configuration is adapted to advertise this network:
protocol direct  
   description "Client networks";
   import filter  
     if net ~ [ 2001:db8:4600::/40 40,48  ] then accept;
     if net ~ [ 2001:db8:aaaa::/48 ] then accept;
     reject;
    ;
   export none;
 
We use NSD as a name server. This is an authoritative only name server. The configuration is pretty simple. We just ensure that it is bound to the IPv6 anycast address (otherwise, it may answer with the wrong address). The zone file is customized to answer the name of the server:
#(C1) host -t TXT example.com 2001:db8:aaaa::53
Using domain server:
Name: 2001:db8:aaaa::53
Address: 2001:db8:aaaa::53#53
Aliases:
example.com descriptive text "G1"
#(C5) host -t TXT example.com 2001:db8:aaaa::53
Using domain server:
Name: 2001:db8:aaaa::53
Address: 2001:db8:aaaa::53#53
Aliases:
example.com descriptive text "G2"
If we try from C3, we may be redirected to L1 while we are not a customer of AS 64620. A common way to handle this is to let 64621 tag the route with a special community that will let know 64620 that the route should not be exported.
protocol direct  
   description "Client networks";
   import filter  
     if net ~ [ 2001:db8:4600::/40 40,48  ] then accept;
     if net ~ [ 2001:db8:aaaa::/48 ] then  
          bgp_community.add((65535,65281));
          accept;
      
     reject;
    ;
   export none;
 
The community used is known as the no-export community. BIRD knows how to handle it. However, we hit here some limitations of BIRD:
  1. The correct way to let 64622 knows about this route is to build a BGP confederation for AS 64620 and let AS 64622 and AS 64621 be part of it. This means that those three AS will be seen as AS 64200 from outside of the confederation. Moreover, the no-export community would allow to export the route only within the confederation. This is exactly what we need but BIRD does not support confederations.
  2. BIRD first selects the best route to export then apply filtering. This means that even if 64620 knows non-filtered route to 2001:db8:aaaa::/48, it will select the route to L1, filter it out and export nothing to 64622.
Therefore, we need to fallback to ask BIRD to not export the route to AS 64600, 64610 and 64630 on 64620. We could use a special community to filter out the route but we keep it simple and just filter the route directly.
filter keep_local_dns  
   if net ~ [ 2001:db8:aaaa::/48 ] then reject;
   accept;
 
protocol bgp  
   description "BGP with peer NewYork";
   local as 64620;
   neighbor 2001:db8:ffff:4600:4620::1 as 64600;
   gateway direct;
   hold time 30;
   export filter keep_local_dns;
   import all;
 
Voil . Only C4 will be able to query L1. If AS 64621 is missing, it will query G2 instead. One major point about anycast is to know that redundancy is done at the AS level only. If the name server on L1 does not work correctly, requests will still be routed to it. Therefore, with anycast, you still need to provide redundancy inside each AS (or stop advertising the anycast network as soon as there is a failure).

Beyond DNS Anycast can be used to serve HTTP requests too. This is less commonly used because your connection may be rerouted to another node while it is still running. This is not a problem with DNS because there is no connection (you send a query, you receive an answer). There are at least two CDN (MaxCDN and CacheFly) which uses anycast to ensure low roundtrip time and very high reliability. Setting up an HTTP server with an anycast address is no different than setting up a DNS server. However, you need to ensure maximum stability of the routing path to avoid the connection to be dropped while running. I suppose that this is done by hiring some high-level BGP expert. You can also mix unicast and anycast:
  • anycast DNS answers the IP address of a nearby unicast HTTP server, or
  • anycast DNS answers the IP address of an anycast HTTP server that will issue a redirect to an unicast HTTP server when downloading or streaming large files.

10 June 2011

Rapha&#235;l Hertzog: People behind Debian: Philipp Kern, Stable Release Manager

Philipp is a Debian developer since 2005 and a member of the release team since 2008. Since he took the responsibility of Stable Release Manager, the process has evolved for the best. I asked him to explain how the release team decides what s fit for stable or not. His work on the buildd infrastructure is also admirable but I ll let him describe that. My questions are in bold, the rest is by Philipp. Who are you? I m a 24 year old Computer Science student, living in Karlsruhe, Germany. As a student assistant I m currently in two jobs: one is taking care of getting our university IPv6 network in shape, or rather generally getting fringe technologies into the network. The second is taking care of a s390x machine in the basement which my faculty got sponsored recently. In my spare time I tend my Debian duties and I m active in the student council (Fachschaft) as a sys-admin, software developer and until recently as treasurer. I got accepted as a Debian Developer in 2005. I m only really active since I was invited to join the Release Team in early 2008 after I contributed rewrites of some scripts that got lost in a disk crash of ftp-master. In late 2008 I also took on wanna-build/buildd duties. What s your notable achievement within Debian? For one I worked on the deployment of a single buildd/sbuild combination over all of our buildds and I rebased it on the sbuild in the archive several times already. All buildds basically look the same on all architectures, with only few variations. The chroots are now also created in a mostly predictable manner. Then we finally got build autosigning after years of constant poking. However the policy decision to allow it was made by the ftp-masters anyway, for which I m grateful, as it eases the workload of the buildd maintainers, the Security Team and the Release Team quite a bit. On the release side there s the integration of volatile into the proper structures of Debian. But there I m guilty for choosing the bad name squeeze-updates is, in comparision with security s squeeze/updates. Let s see if we can improve that for wheezy. The policy for stable updates have changed over time. Can you summarize what kind of updates are allowed nowadays? All the changes that were made to the policy were driven by the aim to keep stable (and to some lesser degree oldstable) usable throughout its lifetime. I have to admit that we got slightly more liberal in what we accept since I took over. The previous stable update policy included the fixes of critical bugs that break the system in interesting ways and security fixes. We also opened up the possibility of including important fixes for annoying bugs on a case-by-case basis. The whole don t update that much part of the stable release management is rooted in let s don t change the behaviour of a running system and let s have as few regressions as possible . We currently try to counter that with a review process that only allows self-contained fixes that were tested in unstable first, if applicable. In the future I d also like to have a feedback process that includes the users of the packages to report problems more early. In fact that s also why we now send calls for testing to debian-stable-announce one week in advance (see this mail as an example). So stable is updated through point releases that happen roughly every two months for stable and roughly every four for oldstable. They are accompanied by announcements and new CD/DVD sets. To be able to push some updates with less delay to our users and to avoid that they have to pick them from proposed-updates, there is stable-updates. The current policy for this suite is available in this post on debian-devel-announce. It boils down to everything urgent like tzdata, regression fixes and volatile packages. We noticed that some packages in stable just weren t useful anymore when they got updated through volatile. Hence those are folded into stable, too. We also relaxed the rules a bit so that leaf packages like clive, that rely on external websites not to change their URLs, can be updated too. If somebody wonders what happened to the and a half releases Those were intended to mark the inclusion of new hardware support (mostly by including a co-installable new kernel version). This kind of flag point release is thankfully no longer needed because our marvellous kernel team now backports certain hardware drivers to the kernel version in stable. They have some trouble soliciting test feedback for them, though. It would be cool if more people using stable could respond to their calls for testing. What are your plans for Debian Wheezy? Luckily the wanna-build/buildd side isn t much dependent on release cycles. The stable release management also just takes care of what gets dumped on them by the testing RMs. ;-) The near-term goals are certainly: Apart from that my work is mainly squashing the bugs on buildd.debian.org for the wanna-build part, so if you want something fixed, you need to report it, I m afraid. You have several roles within Debian, in particular stable release manager and member of the wanna-build team. Which role do you enjoy more and why? So the regular duties of the SRM, apart from setting up policies, are reviewing requests for uploads to proposed-updates, accepting them, chasing builds and scheduling point releases. It s mainly steady work. As for the Release Team it s working in a fabulous team where everyone s doing an awesome job. For me the wanna-build role is holding the things together and fixing up stuff that breaks while the world continues to evolve. And occassionally taking a stab at the ftp-masters. So rather different. My heart is probably siding with the buildd side because it s infrastructure but release work is fun, too. What s the biggest problem of Debian? Mainly that we re not quick in picking up new technologies and that we got slow on innovation. Distro-wide changes are usually a very big effort to coordinate and to finally complete. I guess people should be able to push fixes for release goals more eagerly into the archive. Also it seems to me that many DDs are actually disengaging from the release process, especially during the freeze but also before, which makes me a bit sad. However I m still not sure how we could counter that. As we re all volunteers you can t suddenly make others love the idea of releasing something together, I guess. Do you have wishes for Debian Wheezy? It s getting old, but there s certainly multiarch. Also a usable Python 3 is something I d really appreciate. Is there someone in Debian that you admire for their contributions? Peter Palfrader did an awesome job of getting a nicely working team together for DSA and got many tasks as automated as they are now. It s a pleasure to work with them and setting up a new buildd is, for example, a real breeze. He also developed a sane archive snapshot service, kudos for that.
Thank you to Philipp for the time spent answering my questions. I hope you enjoyed reading his answers as I did. Subscribe to my newsletter to get my monthly summary of the Debian/Ubuntu news and to not miss further interviews. You can also follow along on Identi.ca, Twitter and Facebook.

3 comments Liked this article? Click here. My blog is Flattr-enabled.

Next.

Previous.