Search Results: "ron"

17 June 2011

Timo Jyrinki: Kernel Mode-Setting (KMS) on Neo FreeRunner + Debian

Edit Aug 18th: video available (ogv, also on youtube)

Forget AMD, Intel, NVIDIA! Thomas White's incredible work with Neo FreeRunner's puny graphics dec...accelerator, Smedia Glamo 3362 is starting to bear fruits. Not listening to such comments as the chip will never be used anywhere else and why not spend your time doing something more relevant , he has chosen to actually do what he likes and sees as an interesting challenge. And that's often the spirit of free software, so I don't really agree with the naysayers.

The camera shot on the right, running KMS-enabled X.org driver for the glamo chip on my Debian installation (visible software matchbox-window-manager, fbpanel, zhone), is a bit optimistic looking since Zhone happens to draw correctly. A lot of the drawing is not yet synced correctly, which shows as all text and images in eg. GTK applications being garbled. But as little as two days ago one couldn't yet much launch applications without X crashing, so the newest commits by Thomas were a big step forwards. I'm using Debian, and he's not, so I try to find time to help in debugging even though I really can't much help with the driver code.

The driver is not just one piece of code, but consists of a kernel drm driver (direct rendering manager) using GEM, libdrm support for the kernel driver and finally the X.org driver supporting these other components and offering buzz-words like DRI2. There is also a beginning of a Mesa 3D driver, though it is so far just a skeleton driver since the 2D/KMS/EXA/DRM parts are what should be done first before dwelling into the OpenGL realm.

Openmoko the company is basically nowadays just producing new Neo FreeRunners to resellers, but the community of the so far only Free phone is thriving. The final (Openmoko produced, from where gta02-core continues) version of the phone, with the famous buzzing problem fixed, appeared on store shelves some time ago, offering better out-of-the-box phone functionality in addition to all the mini computer features. It was also offered to Debconf visitors on discount.

It's not only cool to have kernel mode-setting, though it is indeed very cool as well. The reason for much sorrow in the whole OM project has been the graphics chip, and some of the problems with the chip are only finally solvable with the kernel doing the mode-setting. So it's both a very modern thing to use KMS, but there are also clear potential benefits of having accurate control over this very sensitive piece of silicon. The current non-KMS X.org driver for example busy loops in order to try to feed the chip at a correct pace, causing CPU usage every time there is any drawing going on.

This is somehow reminding me about getting the best out of 8-bit computers and other hardware with some specific limitations. The glamo chip is very limited in some ways, but also capable in some other ways like theoretically offering MPEG-4 decoding, OpenGL ES 3D support. It's a mixed bag of things, and I can very well imagine it's indeed an interesting challenge to work on it, if you just have the skills (I don't) and nerves (proper debugging tools help).

It's so nerdy to drool over lines in the X.org log, but I just can help it:

X.Org X Server 1.6.3
Release Date: 2009-7-31
X Protocol Version 11, Revision 0
Build Operating System: Linux 2.6.28 armv5tel Debian
Current Operating System: Linux neo 2.6.29-GTA02_mydrm #2 PREEMPT Mon Aug 17 18:24:39 EEST 2009 armv4tl
...
II) Using KMS!
...
(WW) Glamo(0): EXA hardware acceleration initialising
(II) EXA(0): Driver allocated offscreen pixmaps
(II) EXA(0): Driver registered support for the following operations:
(II) Solid
(II) Copy
(II) Glamo(0): Initialized EXA acceleration
...
(II) Glamo(0): RandR 1.2 enabled, ignore the following RandR disabled message.
...
(II) Glamo(0): [glamo-dri] Name of DRM device is '/dev/dri/card0'
(WW) Glamo(0): [DRI2] Version 1 API (broken front buffer rendering)
(II) Glamo(0): [DRI2] Setup complete
...
(II) AIGLX: enabled GLX_MESA_copy_sub_buffer
(II) AIGLX: Loaded and initialized /usr/lib/dri/glamo_dri.so
(II) GLX: Initialized DRI2 GL provider for screen 0
(II) Glamo(0): Adding framebuffer....!
(II) Glamo(0): 8 480 640 16 16 960
(II) Glamo(0): rootPixmap = 0x1d0a00
(II) Glamo(0): Done

15 June 2011

Michal Čihař: Lack of time

You can see it from my blog activity as well, I simply lack time for lot of things. My Gammu mail folders are horribly full of unread mail, it took me there three months to release a new testing version and there are still some patches pending for review. On phpMyAdmin side I spend most time with my GSoC students, who progress really well, but that's pretty much all I can do there. The good news is that phpmyadmin.cz website has new maintainer and is again getting new content. Hopefully it will stay alive for longer time to bring back lost visitors. My Debian contributions are in still pretty good shape and I manage to fix important bugs, though my MPD related packages are still for sale. Still considering whether to go or not to go to Debconf, my decision will be probably based on fact whether I will get travel sponsorship or not (and when it would happen). Besides these free software activities I began to be more active in Spole n aktivity (in Czech) project for organizing various activities to spend free time. Occasionally I also contribute to OpenStretMap, but it's more or less random time when I notice something is wrong or when I manage to visit some place which is not that good mapped. The good thing is that Prague is pretty good shape there, so it does not motivate me to spend too much time on this :-). And of course there is my work at SUSE, working on usual L3 front and improving our Django based incident tracking tool. Maybe I should step down from some of these, but it would be pretty hard for me to choose which one...

Filed under: Debian English Gammu Phpmyadmin 2 comments Flattr this!

Christian Perrier: So, what happened with Kikithon?

I mentioned this briefly yesterday, but now I'll try to summarize the story of a great surprise and a big moment for me. All this started when my wife Elizabeth and my son Jean-Baptiste wanted to do something special for my 50th birthday. So, it indeed all started months ago, probably early March or something (I don't yet have all the details). Jean-Baptiste described this well on the web site, so I won't go again into details, but basically, this was about getting birthday wishes from my "free software family" in, as you might guess, as many languages as possible. Elizabeth brought the original idea and JB helped her by setting up the website and collecting e-mail addresses of people I usually work with: he grabbed addresses from PO files on Debian website, plus some in his own set of GPG signatures and here we go. And then he started poking dozens of you folks in order to get your wishes for this birthday. Gradually, contributions accumulated on the website, with many challenges for them: be sure to get as many people as possible, poking and re-poking all those FLOSS people who keep forgetting things... It seems that poking people is something that's probably in the Perrier's genes! And they were doing all this without me noticing. As usually in Debian, releasing on time is a no-no. So, it quickly turned out that having everything ready by April 2nd wouldn't be possible. So, their new goal was offering this to me on Pentecost Sunday, which was yesterday. And...here comes the gift. Aha, this looks like a photo album. Could it be a "50 years of Christian" album? But, EH, why is that pic of me, with the red Debconf5 tee-shirt (that features a world map) and a "bubulle" sign, in front of the book? But, EH EH EH, what the .... are doing these word by H0lger, then Fil, then Joey doing on the following pages? And only then, OMG, I discover the real gift they prepared. 106, often bilingual, wishes from 110 people (some were couples!). 18 postcards (one made of wood). 45 languages. One postcard with wishes from nearly every distro representatives at LinuxTag 2011. Dozens of photos from my friends all around the world. All this in a wonderful album. I can't tell what I said. Anyway, JB was shooting a video, so...we'll see. OK, I didn't cry...but it wasn't that far and emotion was really really intense. Guys, ladies, gentlemen, friends....it took me a while to realize what you contributed to. It took me the entire afternoon to realize the investment put by Elizabeth and JB (and JB's sisters support) into this. Yes, as many of you wrote, I have an awesome family and they really know how to share their love. I also have an awesome virtual family all around the world. Your words are wholeheartedly appreciated and some were indeed much much much appreciated. Of course, I'll have the book in Banja Luka so that you can see the result. I know (because JB and Elizabeth told me) that many of you were really awaiting to see how it would be received (yes, that includes you, in Germany, who I visited in early May!!!). Again, thank you so much for this incredible gift. Thank you Holger Levsen, Phil Hands, Joey Hess, Lior Kaplan, Martin Michlmayr, Alberto Gonzalez Iniesta, Kenshi "best friend" Muto, Praveen Arimbrathodiyil, Felipe Augusto van de Wiel, Ana Carolina Comandulli (5 postcards!), Stefano Zacchiroli (1st contribution received by JB, of course), Gunnar Wolf, Enriiiiiico Zini, Clytie Siddall, Frans Pop (by way of Clytie), Tenzin Dendup, Otavio Salvador, Neil McGovern, Konstantinos Margaritis, Luk Claes, Jonas Smedegaard, Pema Geyleg, Meike "sp tzle queen" Reichle, Alexander Reichle-Schmehl, Torsten Werner, "nette BSD" folks, CentOS Ralph and Brian, Fedora people, SUSE's Jan, Ubuntu's Lucia Tamara, Skolelinux' Paul, Rapha l Hertzog, Lars Wirzenius, Andrew McMillan (revenge in September!), Yasa Giridhar Appaji Nag (now I know my name in Telugu), Amaya Rodrigo, St phane Glondu, Martin Krafft, Jon "maddog" Hall (and God save the queen), Eddy Petri or, Daniel Nylander, Aiet Kolkhi, Andreas "die Katze geht in die K che, wunderbar" Tille, Paul "lets bend the elbow" Wise, Jordi "half-marathon in Banja Luka" Mallach, Steve "as ever-young as I am" Langasek, Obey Arthur Liu, YAMANE Hideki, Jaldhar H. Vyas, Vikram Vincent, Margarita "Bronx cross-country queen" Manterola, Patty Langasek, Aigars Mahinovs (finding a pic *with* you on it is tricky!), Thepittak Karoonboonyanan, Javier "nobody expects the Spanish inquisition" Fern ndez-Sanguino, Varun Hiremath, Moray Allan, David Moreno Garza, Ralf "marathon-man" Treinen, Arief S Fitrianto, Penny Leach, Adam D. Barrat, Wolfgang Martin Borgert, Christine "the mentee overtakes the mentor" Spang, Arjuna Rao Chevala, Gerfried "my best contradictor" Fuchs, Stefano Canepa, Samuel Thibault, Eloy "first samba maintainer" Par s, Josip Rodin, Daniel Kahn Gillmor, Steve McIntyre, Guntupalli Karunakar, Jano Gulja , Karolina Kali , Ben Hutchings, Matej Kova i , Khoem Sokhem, Lisandro "I have the longest name in this list" Dami n Nicanor P rez-Meyer, Amanpreet Singh Alam, H ctor Or n, Hans Nordhaugn, Ivan Mas r, Dr. Tirumurti Vasudevan, John "yes, Kansas is as flat as you can imagine" Goerzen, Jean-Baptiste "Piwet" Perrier, Elizabeth "I love you" Perrier, Peter Eisentraut, Jesus "enemy by nature" Climent, Peter Palfrader, Vasudev Kamath, Miroslav "Chicky" Ku e, Mart n Ferrari, Ollivier Robert, Jure uhalev, Yunqiang Su, Jonathan McDowell, Sampada Nakhare, Nayan Nakhare, Dirk "rendez-vous for Chicago marathon" Eddelbuettel, Elian Myftiu, Tim Retout, Giuseppe Sacco, Changwoo Ryu, Pedro Ribeoro, Miguel "oh no, not him again" Figueiredo, Ana Guerrero, Aur lien Jarno, Kumar Appaiah, Arangel Angov, Faidon Liambotis, Mehdi Dogguy, Andrew Lee, Russ Allbery, Bj rn Steensrud, Mathieu Parent, Davide Viti, Steinar H. Gunderson, Kurt Gramlich, Vanja Cvelbar, Adam Conrad, Armi Be irovi , Nattie Mayer-Hutchings, Joerg "dis shuld be REJECTed" Jaspert and Luca Capello. Let's say it gain:

Danke
Thank you
(RTL or not?) / toda
Gracias
/ arigat
/ n ndi
Obrigado
Grazie
C m n
Kadin chey la
/ Efharisto
Dank je wel
Tak
D ngge
Merci
Kiitos
Tika hoki
/ dhanyavaadhamulu
Thank you and God save the Queen
Mul umesc
Tack
/ didi madloba
Gr cies
Dakujem
/ Aabhar
/ Dhanyavadagalu
Paldies
/ kop khun krap
Terima kasih
Hvala (hr)
/ Dhanyavad
/ Hvala (sr)
Hvala (sl)
/ Akun
/ Sukriya
Takk
/ Nandri
Maki biyen jianbakik
D kuji
/ Xiexie
/ Aabhari aahe
Faleminderit
/ Kam sah ham ni da
/ Blagodaram
/ Xiexie
Hvala (bs)

13 June 2011

Christian Perrier: So, what happened with Kikithon?

Danke
Thank you
(RTL or not?) / toda
Gracias
/ arigat
/ n ndi
Obrigado
Grazie
C m n
Kadin chey la
/ Efharisto
Dank je wel
Tak
D ngge
Merci
Kiitos
Tika hoki
/ dhanyavaadhamulu
Thank you and God save the Queen
Mul umesc
Tack
/ didi madloba
Gr cies
Dakujem
/ Aabhar li> / Dhanyavadagalu
Paldies
/ kop khun krap
Terima kasih
Hvala (hr)
/ Dhanyavad
/ Hvala (sr)
Hvala (sl)
/ Akun
/ Sukriya
Takk
/ Nandri
Maki biyen jianbakik
D kuji
/ Xiexie
/ Aabhari aahe
Faleminderit
/ Kam sah ham ni da
/ Blagodaram
/ Xiexie
Hvala (bs)

Vincent Bernat: Anycast DNS

In a recently published article, Paul Vixie, past author and architect of BIND, one of the most popular internet domain server, explains why DNS servers should use anycast for increased reliability and faster answers. The main argument against unicast is that some resolvers may not be smart enough to select the best server to query and therefore, they may select a server on the other side of the planet. Since, you cannot go faster than light, you get a penalty of about 100 milliseconds in this case.

Unicast & anycast Most addresses that you handle are unicast addresses, i.e. the address of a single receiver. Any time you send a packet to an unicast address, it will be forwarded to this single receiver. Another common network addressing system is multicast (which happens to include broadcast). When you send a packet to a multicast address, several receivers may get it. The packet is copied as needed by some nodes. Anycast addresses allow you to send a packet to a set of receivers but only one of them will receive it. The packet will not be copied. Usually, the idea behind anycast is that the packet is sent to the nearest receiver. There is no universal way to determine if an IP address is an unicast address or an anycast address just by looking at it. For example `192.5.5.241` is an anycast address while `78.47.78.132` is an unicast address. ISC, which maintains BIND, operates the F root domain server, one of the 13 Internet root name servers. This server is queried by using `192.5.5.241` IPv4 anycast address or `2001:500:2f::f` IPv6 anycast address. ISC has published a document explaining how anycast works. Several nodes around the globe announces the same subnet using BGP. When one router running BGP has to send a query to the anycast address, it will have several choices in its BGP table. It will usually select the shortest path and route the query to the corresponding router. Some nodes may choose to restrict the announcement about the subnet containing the anycast address. They are called local nodes (by opposition to other global nodes). This allows them to serve query only from their clients and use cheaper connectivity and hardware.

Lab Let s play with anycast by setting up a lab to experiment with it. We want to setup an IPv6 anycast DNS service using two global nodes, `G1` and `G2`, and one local node, `L1`. We build some kind of mini-Internet. See :

Setting up the lab This lab will use UML guests for all nodes. For more details on how this works, look at my previous article about network labs with User Mode Linux. You first need to grab the lab:

$ git clone https://vincentbernat@github.com/vincentbernat/network-lab.git
$ cd network-lab/lab-anycast-dns
$ ./setup

You will need to install the dependencies needed by the script on your host system. The script was tested on Debian unstable. If some of the dependencies do not exist in your distribution, if they don t behave as in Debian or if you don t want to install them on your host system, you can use debootstrap to create a working system:

$ sudo debootstrap sid ./sid-chroot http://ftp.debian.org/debian/
[...]
$ sudo chroot ./sid-chroot /bin/bash
# apt-get install iproute zsh aufs-tools
[...]
# apt-get install bird6
[...]
# exit
$ sudo mkdir -p ./sid-chroot/$HOME
$ sudo chown $(whoami) ./sid-chroot/$HOME
$ ROOT=./sid-chroot ./setup

We are running a lot of UML guests in this lab. We assign 64 MB to each host. UML guests do not use host memory until they need it but because of cache subsystem, you can expect that each guest will use 64 MB of memory. This means that you need about 2 GB of RAM to run this lab. Moreover, each guest will create a file representing its memory. Therefore, you need about 2 GB of free space in /tmp too. Routers are named after their AS number. The router of AS 64652 is just 64652. Paris, NewYork and Tokyo are the exception. The lab will use only one switch but we will use VLAN to create several L2 networks. Each link in our schema is an L2 network. Each link will get its own L3 network. Those networks are prefixed with 2001:db8:ffff:. Leaf AS have their own L3 network 2001:db8:X::/48 where X is the AS number minus 60000. Look at /etc/hosts on each host to get the IP of all hosts.

Mini-Internet routing AS 64600 is some kind of Tier-1 transit network (like Level 3 Communications). It has three points of presence: Paris, New York and Tokyo. AS 64650 is an European regional transit network, AS 64640 is an East Asian regional transit network and AS 64610, 64620 and 64630 are North American regional transit network which are peering each other. Other leaf AS are local ISP. This is not really what Internet looks like but shaping our lab like a tree allows us to not deal with a lot of BGP stuff: we ensure that the shortest path (in number of BGP hops) from one client to one server is the best path. Each BGP router will get a basic BGP configuration containing only the list of peers. Since Paris, Tokyo and NewYork are in the same AS, they will use iBGP. They are fully meshed for this purpose but this does not provide redundancy because iBGP routes are not redistributed inside iBGP. A more complex setup would be needed to provide redundancy inside AS 64600. We don t use an IGP inside leaf AS as we should: for the sake of simplicity, we redistribute directly connected routes into BGP. While in my previous lab, I used Quagga, I will use BIRD in this one. It is a more modern daemon with a clean design on how routing tables may interact. It also performs better. However, it lacks several features from Quagga. The configuration for BIRD is autogenerated from /etc/hosts. We use only one routing table inside BIRD. This table will be exported to the kernel and to any BGP instance. Some directly connected routes will be imported (those representing client networks) as well as all routes from BGP instances. Here is an excerpt of 64610 s configuration:

protocol direct  
   description "Client networks";
   import filter  
     if net ~ [ 2001:db8:4600::/40 40,48  ] then accept;
     reject;
    ;
   export none;
 
protocol kernel  
   persist;
   import none;
   export all;
 
protocol bgp  
   description "BGP with peer NewYork";
   local as 64610;
   neighbor 2001:db8:ffff:4600:4610::1 as 64600;
   gateway direct;
   hold time 30;
   export all;
   import all;

We do not export inter-router subnets to keep our routing tables small. This also means that those IP addresses are not routable in spite of the use of public IP addresses. Here is the routing table as seen by 64610 (use birdc6 to connect to BIRD):

bird> show route
2001:db8:4622::/48 via 2001:db8:ffff:4610:4620::2 on eth3 [bgp4 18:52] * (100) [AS64622i]
                   via 2001:db8:ffff:4600:4610::1 on eth0 [bgp1 18:52] (100) [AS64622i]
2001:db8:4621::/48 via 2001:db8:ffff:4610:4620::2 on eth3 [bgp4 18:52] * (100) [AS64621i]
                   via 2001:db8:ffff:4600:4610::1 on eth0 [bgp1 18:52] (100) [AS64621i]
2001:db8:4612::/48 via 2001:db8:ffff:4610:4612::2 on eth2 [bgp3 18:52] * (100) [AS64612i]
2001:db8:4611::/48 via 2001:db8:ffff:4610:4611::2 on eth1 [bgp2 18:52] * (100) [AS64611i]

As you can notice, it knows several paths to the same network, however, it only selects the best one (based on the length of the AS path):

bird> show route 2001:db8:4622::/48 all
2001:db8:4622::/48 via 2001:db8:ffff:4610:4620::2 on eth3 [bgp4 18:52] * (100) [AS64622i]
        Type: BGP unicast univ
        BGP.origin: IGP
        BGP.as_path: 64620 64622
        BGP.next_hop: 2001:db8:ffff:4610:4620::2 fe80::e05f:a3ff:fef2:f4e0
        BGP.local_pref: 100
                   via 2001:db8:ffff:4600:4610::1 on eth0 [bgp1 18:52] (100) [AS64622i]
        Type: BGP unicast univ
        BGP.origin: IGP
        BGP.as_path: 64600 64620 64622
        BGP.next_hop: 2001:db8:ffff:4600:4610::1 fe80::3426:63ff:fe5e:afbd
        BGP.local_pref: 100

Let s check that everything works with traceroute6 on C1:

# traceroute6 G2.lab
traceroute to G2.lab (2001:db8:4612::2:53), 30 hops max, 80 byte packets
 1  64652-eth1.lab (2001:db8:4652::1)  -338.277 ms  -338.449 ms  -338.502 ms
 2  64650-eth2.lab (2001:db8:ffff:4650:4652::1)  -338.324 ms  -338.363 ms  -338.384 ms
 3  Paris-eth2.lab (2001:db8:ffff:4600:4650::1)  -338.191 ms  -338.329 ms  -338.282 ms
 4  NewYork-eth0.lab (2001:db8:ffff:4600:4600:1:0:2)  -338.144 ms  -338.182 ms  -338.235 ms
 5  64610-eth0.lab (2001:db8:ffff:4600:4610::2)  -338.078 ms  -337.958 ms  -337.944 ms
 6  64612-eth0.lab (2001:db8:ffff:4610:4612::2)  -337.778 ms  -338.103 ms  -338.041 ms
 7  G2.lab (2001:db8:4612::2:53)  -338.001 ms  -338.045 ms  -338.080 ms

Let s see what happens if we shut the link between NewYork and 64610. This link is VLAN 10. Since vde_switch does not allow to shut a port, we will change the VLAN of port 9 to VLAN 4093. After at most 30 seconds, the BGP session between NewYork and 64610 will be torn down and packets from C1 to G2 will take a new path:

# traceroute6 G2.lab
traceroute to G2.lab (2001:db8:4612::2:53), 30 hops max, 80 byte packets
 1  64652-eth1.lab (2001:db8:4652::1)  -338.382 ms  -338.499 ms  -338.503 ms
 2  64650-eth2.lab (2001:db8:ffff:4650:4652::1)  -338.284 ms  -338.317 ms  -338.151 ms
 3  Paris-eth2.lab (2001:db8:ffff:4600:4650::1)  -337.954 ms  -338.031 ms  -337.924 ms
 4  NewYork-eth0.lab (2001:db8:ffff:4600:4600:1:0:2)  -337.655 ms  -337.751 ms  -337.844 ms
 5  64620-eth0.lab (2001:db8:ffff:4600:4620::2)  -337.762 ms  -337.507 ms  -337.611 ms
 6  64610-eth3.lab (2001:db8:ffff:4610:4620::1)  -337.531 ms  -338.024 ms  -337.998 ms
 7  64612-eth0.lab (2001:db8:ffff:4610:4612::2)  -337.860 ms  -337.931 ms  -337.992 ms
 8  G2.lab (2001:db8:4612::2:53)  -337.863 ms  -337.768 ms  -337.808 ms

Setting up anycast DNS Our anycast DNS service has three nodes. Two global ones, located in Europe and North America and one local one targeted at customers of AS 64620 only. The IP address of this anycast DNS server is 2001:db8:aaaa::53/48. We assign this address to G1, G2 and L1. We also assign 2001:db8:aaaa::1/48 to 64651, 64612 and 64621. BIRD configuration is adapted to advertise this network:

protocol direct  
   description "Client networks";
   import filter  
     if net ~ [ 2001:db8:4600::/40 40,48  ] then accept;
     if net ~ [ 2001:db8:aaaa::/48 ] then accept;
     reject;
    ;
   export none;

We use NSD as a name server. This is an authoritative only name server. The configuration is pretty simple. We just ensure that it is bound to the IPv6 anycast address (otherwise, it may answer with the wrong address). The zone file is customized to answer the name of the server:

#(C1) host -t TXT example.com 2001:db8:aaaa::53
Using domain server:
Name: 2001:db8:aaaa::53
Address: 2001:db8:aaaa::53#53
Aliases:
example.com descriptive text "G1"
#(C5) host -t TXT example.com 2001:db8:aaaa::53
Using domain server:
Name: 2001:db8:aaaa::53
Address: 2001:db8:aaaa::53#53
Aliases:
example.com descriptive text "G2"

If we try from C3, we may be redirected to L1 while we are not a customer of AS 64620. A common way to handle this is to let 64621 tag the route with a special community that will let know 64620 that the route should not be exported.

protocol direct  
   description "Client networks";
   import filter  
     if net ~ [ 2001:db8:4600::/40 40,48  ] then accept;
     if net ~ [ 2001:db8:aaaa::/48 ] then  
          bgp_community.add((65535,65281));
          accept;
      
     reject;
    ;
   export none;

The community used is known as the no-export community. BIRD knows how to handle it. However, we hit here some limitations of BIRD:

The correct way to let 64622 knows about this route is to build a BGP confederation for AS 64620 and let AS 64622 and AS 64621 be part of it. This means that those three AS will be seen as AS 64200 from outside of the confederation. Moreover, the no-export community would allow to export the route only within the confederation. This is exactly what we need but BIRD does not support confederations.
BIRD first selects the best route to export then apply filtering. This means that even if 64620 knows non-filtered route to 2001:db8:aaaa::/48, it will select the route to L1, filter it out and export nothing to 64622.

Therefore, we need to fallback to ask BIRD to not export the route to AS 64600, 64610 and 64630 on 64620. We could use a special community to filter out the route but we keep it simple and just filter the route directly.

filter keep_local_dns  
   if net ~ [ 2001:db8:aaaa::/48 ] then reject;
   accept;
 
protocol bgp  
   description "BGP with peer NewYork";
   local as 64620;
   neighbor 2001:db8:ffff:4600:4620::1 as 64600;
   gateway direct;
   hold time 30;
   export filter keep_local_dns;
   import all;

Voil . Only C4 will be able to query L1. If AS 64621 is missing, it will query G2 instead. One major point about anycast is to know that redundancy is done at the AS level only. If the name server on L1 does not work correctly, requests will still be routed to it. Therefore, with anycast, you still need to provide redundancy inside each AS (or stop advertising the anycast network as soon as there is a failure).

Beyond DNS Anycast can be used to serve HTTP requests too. This is less commonly used because your connection may be rerouted to another node while it is still running. This is not a problem with DNS because there is no connection (you send a query, you receive an answer). There are at least two CDN (MaxCDN and CacheFly) which uses anycast to ensure low roundtrip time and very high reliability. Setting up an HTTP server with an anycast address is no different than setting up a DNS server. However, you need to ensure maximum stability of the routing path to avoid the connection to be dropped while running. I suppose that this is done by hiring some high-level BGP expert. You can also mix unicast and anycast:

anycast DNS answers the IP address of a nearby unicast HTTP server, or

anycast DNS answers the IP address of an anycast HTTP server that will issue a redirect to an unicast HTTP server when downloading or streaming large files.

Craig Small: Debian Linux on HP DV6-6023TX

It was time to update my creaking old laptop. I ordered HP6-6023TX online and after a few days it arrived just before a long weekend. I first backed up Windows 7 onto a set of DVDs and found that either my blank DVDs are old or this new DVD drive is far more fussy. There were quite a few coasters made until I gave up and bought some new ones. DVDs seemed to be trouble for me that day. I downloaded the Debian 6.0.4 DVD image but couldn t burn it. Brasero would just keep saying calculating checksum and go no further. So after waiting too long, the CD iso was downloaded and installed. Installation of debian onto the laptop was straightforward. That is if you used a wired connection. Wireless needs the non-free firmware and I gave up trying to work out how to install it at debian-installer time as it didn t see the tar.gz full of firmware on the USB stick. Wireless worked perfectly once I loaded everything on. So at this point with not much trouble I had an almost fully working setup. Even the webcam and sound was working. Video Fun The laptop is one of these strange modern ones that has two video cards. The low-spec one is an Intel Sandybridge while the higher one is an AMD/ATI Radeon HD 6700M. The idea being you use the ATI when plugged in and the Intel on batteries. Xorg worked first go, but only on the Intel chip. Trying to force X to use the ATI driver with an Xorg configuration resulted in the Xserver crashing. Next attempt was the fglrx packages within Debian. aticonfig said it didn t know about the card. The reason for this is that the Debian packages use fglrx version11.4 and I needed for my card 11.5 So close but so far away. OK, so building the ATI software can t be that hard can it? A nice set of instructions on the Debian wiki and we had errors. The first problem is that the packaging used is ancient. The second problem is much more serious; it won t compile. I was stuck about here until I hopped onto the Debian irc channel and asked about it and someone found a very useful webpage about compiling the ATI drivers on Debian which importantly had information about the patches you need, especially the blk patch. Success! the program compiled and installed. A small xorg.conf specifying the fglrx driver and the BusID and we were up and running. I now have accelerated graphics on the laptop. Ouch, my eyes! I do actually get outside (one of my hobbies is gardening and that is an outdoor activity) so I don t need a monitor screen trying to give me a suntan. The default screen brightness on the laptop could be described as sub-nova . I m sure there is some alien astronomers several light-years away who will in many years time wonder what that flash from the unimpressive end of the milky-way was about. To protect your eyes and keep the neighborhood complaints down you can reduce the brightness to something sane. You do this in the power management settings which for me is System -> Preferences -> Power Management. I can see again!

Joachim Breitner: 401 page family book published

Over the course of the last five years, some relatives of me and I have been working on a family book which lists all descendants of my grandgrandgrandgrandfather Isaak Riehm, born 1799, and his wife Charlotte. My task was everything related to maintaining the data, running the infrastructure and generating the final book. In the end, we have collected biographical details, often with the person s live story, of 703 descendants, 285 partners and 416 parents-in-law. Together with an extensive preface the book now has a respectable size of 401 pages, includes 144 pictures and maps and over 200 copies have been printed and sold through the family.

Although I do not think that the details of the lives of my family is very interesting for the general public, or even suitable for general release, I do want to show the fruit of our work. Therefore, I have created a stripped-down on-line version that only covers the first four generations. I bundled the book body with the cover and the end sheets into one complete PDF file (caution, large: 54MB, and of course in German). Some nifty embellishments of the book that I especially like are the abstract looking descendant fan in the front-end and back-end sheets, the separating bars between the persons which indicate the livespan of that person between 1799 and 2011 and the maps of the birth places and Isaak s travel destinations in the preface. Now to the technical details: Naturally, I used only Free Software in the genesis of the work (The cover is an exception, as it was not created by me). The most important is definitely gramps, a free genealogy system that held all our data. It is written in python and extensible with plug-ins, so I was able to create a custom report that takes the data relevant for us, filters for people who have agreed to appear in the book, and then generate LaTeX code that typesets the book, ready to be printed. I learned a great deal of LaTeX for this project, which uses over 30 different packages and required me to create a font of my own (which, as a side product, lead to a publication in the Linux-Magazin). For the maps I found the Generic Mapping Tools (GMT) and the freely available data for that of great use. I still had to manually collect the coordinates for all the places I wanted to show up in the map. Here, I relied mostly on Wikipedia. The fan which appears in the end sheets is laid out using a Haskell program (naturally, I have to use Haskell somewhere).

Of course, generating the book is only one side of the story; first, the data has to be collected and edited. For that, I wrote a plug-in to gramps that creates forms for each person. These forms contain the current known data and are used to correct and extend our knowledge. One variant of the form, created using LaTeX, was printed and mailed by snail-mail around the country and beyond, for those who are not reachable by mail. The other variant is an HTML form, hosted on a password-protected web site. Changes to that form cause the server to send an e-mail with a description of the changes (and only the changes, not the fields that are left untouched) to the mailing list of our team. There, the changes can be reviewed, e.g. for spelling and stylistic issues, and finally I copy n pasted them into gramps. These parts are implemented in python as well. Since genealogical data is never complete or up-to-date, I have set up a password-protected mediawiki based web site for the book where we collect corrections and additions to the data in the book. We have never anticipated that it would be so much work and that it would take us so long to finish this project, but we pulled it through and we are very happy with the result. Also, the reception has been very good and it was definitely a success. At this point I d like to express my gratitude to the hundreds of Free Software developers out there whose code we used and without whom it would not have been possible publish the book in the current form at all! Update: Due to some requests, I am making the code that I wrote available. You can download the tarball. Note that the code is a big mess and will not be usable right away. I guess if you can get the code to run and actually adjust to your needs, then you can as well write it yourself. Nevertheless it might be useful as inspiration or reference.

12 June 2011

Neil Williams: Ideas for TDeb packages

This is a follow-up on TDebs with some detail of how existing packages can be adapted to support TDebs.

$ cat debian/package-tdeb.tdeb



[gettextdir] po

[gettextname] package

[po4apodir] doc/po

[po4amandir] doc/package/man/

[po4acommand] po4a-build

What's happening then? Well, the first thing to take into account is that a lot of the values for these settings will be dictated by upstream, which is why it is necessary for maintainers to add this information as a support file in debian/. I tried determining all this information automatically but the script got far too long and still was far from complete.

gettextdir - usually po but some upstream packages can support multiple po generation, so this allows for those.
gettextname - the hardest thing to identify reliably, this is the name of the files to be installed in /usr/share/locale/lang/LC_MESSAGES/ and the name of the of the POT file.
po4apodir - po4a is for documentation translation support, which is more variable, so a po4acommand field is available too.

Other changes
The (currently unsupported in Debian) XC-Package-Type: tdeb option is used in debian/control to mark a new or existing package as a TDeb. To use an existing package, it must comply with the rules for a TDeb including that it must be Architecture: all. The other rules centre around build issues. If any of the files in the existing package would require execution of any part of the normal build process of the package, these would have to be moved out of the package before that package could be a TDeb. The reason is that TDeb generation by translators needs to not require a full build environment for every individual package and must not risk modifying package contents in ways which are related to the build environment. So, unchanged images and other architecture-independent files are OK, any files which need build-dependencies are a problem. I don't know how many packages there are out there with such files. If you have a -data or -common package which contains translated data and that package also includes architecture-independent files, I'd be interested in knowing the name of the packages involved. If those packages also include files which are generated or modified within the package build, I'll be particularly interested in knowing about those source packages.
debconf
The most common use of TDebs during a release cycle is probably going to be debconf template updates by Christian and these are handled without needing a listing in the .tdeb file.
Other formats
I'm looking at QtLinguist support and it should be fairly straightforward using the .tdeb file arrangement.

Once that's done (in a branch, please - these can't be uploaded yet), the package can be tested with the soon-to-be-uploaded emdebian-tdeb package. The old content has been moved into the emdebian-grip-server package and the dependencies of the old version stripped back. The package contains two scripts dpkg-gentdeb and dh_gentdeb which I hope to get included into the respective packages in time for Wheezy. The idea will be that build systems (dh7, CDBS, debhelper-plain etc.) includes a call to the relevant script. (dh_ is just a wrapper around dpkg- in this case). This would allow maintainers to create the first TDeb, get that through NEW and then the fun starts. Once the translation mechanism is set up, I would like a nominated person (likely to be Christian if he agrees) to co-ordinate translation updates for particular packages to allow for a single upload including updates for many translations. This would be package_version+t1_all.tdeb and an appropriate .diff.gz or debian.tar.gz. No binary upload, no impact on testing migrations.

This is mostly theory right now but the scripts do exist in Emdebian SVN.

Still more testing to do but I hope to upload these scripts to unstable in the next few days.

Kai Wasserb ch: r[36]00g: S3TC support on amd64 (especially for Wine)

I already blogged about S3TC support with Wine, but back then I wrote about problems to get that working on amd64 without a proprietary driver. Now I've figured out a way to get it working on amd64 with the free radeon drivers (Gallium versions). It's actually pretty simple, as most things are, once you figured it out. For 64 bit applications, you just need a recent Mesa version and the package libtxc-dxtn0. If you're using a card supported by the r600g driver, you first need a version of the Mesa package, which is build with the Gallium version of the r600 driver (the classic driver should work too, but I haven't tested that), like my unofficial Mesa packages (those packages were built for testing r600g, see this e-mail to debian-x). r600g users also need to set R600_S3TC_ENABLE=1, otherwise the driver won't attempt to load libtxc-dxtn0. For 32 bit applications (like Windows games run with Wine) the initial setup is a little bit more involved: in addition to the above you also need to update a few libraries under /usr/lib32: namely Mesa and libdrm (if you use my Mesa packages you need a i386 copy of libffi too). The quick and dirty approach is to download the i386 packages for libgl1-mesa-dri, libgl1-mesa-glx, libglu1-mesa, libdrm2, libdrm-intel1, libdrm-radeon1 and (if you use my Mesa build) libffi5. Unpack the packages (dpkg-deb -x) and move the files under /path/to/unpack/dir/usr/lib to /usr/lib32. Alternatively you can also build a new ia32-libs package (make sure you change the source.list file of ia32-libs to use at least testing as a base, otherwise you'll end up with basically the same versions as are in the current package). No matter how you updated your drivers you now need the 32 bit version of libtxc-dxtn0. debian-multimedia.org offers a packaged version. After installing all the previous packages, Wine should be able to display (most) S3TC-using programs (in case you use the r600g driver, make sure you have R600_S3TC_ENABLE=1 set). If you're using my Mesa builds you can drop the LIBGL_DRIVERS_PATH=/usr/lib32/dri part again from your Wine invocation. Please note, that there might be some S3TC-using applications with graphical glitches, when using the above method, as the support for S3TC in r600g is still marked as WIP.

11 June 2011

Dirk Eddelbuettel: United Sounds of America: New York

Last night was the kick-off concert in a series the Chicago Symphony has been preparing (and pushing) for the last little while: United Sounds of America: A Journey through Musical Roots. Five concerts featuring New York (last night), Chicago/Route 66 (tonight), New Orleans (tomorrow), Detroit (next Friday) and Austin (next Saturday). The New York evening was organized by pianist Bill Charlap (wikipedia) who also headlined and MC-ed the first set. Covering the birth of Jazz until early post-war bebop in chronological fashion, it was a pretty decent jazz set with a very nice band featuring Kenny Washington (dr), Peter Washington (b), Jeremy Pelt (tp), Jimmy Greene (ts) and Ken Peplowski (cl) --- and local favourite Kurt Elling (wikipedia) who was simply outstanding with the classic songbook material. Having seen him live once before a few years ago, I happened to listen a lot to Elling of late, following the random find of this stream from a live concert in February as well as his two most recent records, the Grammy-winning Dedicated to You (which retakes the wonderful Coltrane/Hartman album from 1963) and the simply amazing very recent The Gate (which you should go and buy right now). And I was not disappointed. He has a great four-octave-spanning baritone voice and great stage presence. The second set was dedicated to post-war folk and singer/songwriter material and organized by Suzanne Vega. That was neat too, if somewhat different in format and more like your standard (rock-ish) concert. Vega brought her own band featuring Gerry Leonard (g), Mike Visceglia (b), Graham Hawthorne (dr) along with guest appearances by Tom Paxton (g, vocals) and Richard Julian (g, vocals). Somehow Vega seems a little trapped in her own success in the 1980s and rehashed a lot of old hits. Nothing wrong that per se as it is good material (more on that below). Only during three encores did she provide new material which was ... excellent. So maybe some rebalancing towards new stuff was neat. Also nice was the additiona of four string players from the Chicago Symphony which had joined the band for a Simon and Garfunkel's song The Boxer. Oh, and of course seeing Vega perform Tom's Diner was nice, especially in such a fast and rocking version, even enhanced by the those strings. Just a few months ago I had gone over the passage from the original a-capella version of Tom's Diner to the various beat-box remixes which were then remixed by Vega in various live performances (e.g. videos of a capella, rockish, another rockish and beatboxish versions). Good fun, and it is nice to see she is playing along and enjoying it as well. All told, a really nice iniative by the CSO. If you're in Chicagoland, go and see some of the remaining shows.

Clint Adams: Debian on the DreamPlug

The DreamPlug is like a GuruPlug Server-Plus with a few notable differences:

A separable power supply and thus none of the heating problems that plagues the GuruPlugs. My GuruPlug neither has a fan nor overheats, but other people seem to whine about these things incessantly
Sound card with TOSLINK out
Instead of 512 megs of NAND, it has 2 megs of SPI flash and a 2-gig microSD card inside
It doesn't come preloaded with Debian

So let's say you want to put Debian on your DreamPlug. You do, of course, because running some other operating system is in poor taste, and you'd be embarrassed telling your friends. How do you do it? You can't use d-i. Let me back up. In the onboard SPI flash, there is a copy of Das U-Boot. This is a fork of mainline U-Boot, with no complete and corresponding source available. To the casual observer, this might look like an egregious GPL violation, but, on the other hand, no one seems to know how to rewrite the SPI flash except from U-Boot itself, so if you happen to corrupt said flash, your plug is bricked and openocd can't save you. I would be happy to be proven wrong on this last point, but for not let's assume that you're screwed and unable to replace your boot loader with free software. The next problem is the kernel, which lives on the microSD card. This is potentially free software, though I can't swear that complete and corresponding source code is available. What I can claim is that linux-2.6.33.6.tar.bz2 on Google Code will build a Linux that works on the plug. I will also point out that this doesn't match Linux 2.6.33.6 from kernel.org. Some other time I should rant about ARM Linux but for now I will just say that if you use the U-Boot of questionable provenance and either the Linux that's shipped with the plug or the one at the above link, you will be able to boot a mostly Debian system. I'll mention the third exception later. One thing you can do is to take the plug apart, remove the microSD card from its slot, and then stick it into a reader that you can then manipulate on another machine. If you're choosing this route and wish to take this opportunity to replace the microSD card with something larger than 2 gigs, keep in mind that you'll have to either preserve the uImage file from the first partition or build a new one. I am not fond of cross-architecture anything. I prefer to do things natively rather than cross-compile or foreign-debootstrap, and I own enough armel boxes that I can do this easily. So at this point I would just debootstrap a new Debian base onto the second partition of the microSD card. The first partition contains the uImage (which is your Linux image in a wrapper), and must be a FAT filesystem since you can't replace your boot loader with something that can read a better filesystem. The second partition will be your rootfs and can be whichever filesystem your Linux can read. If you do not have another armel system to play with, you might want to familiarize yourself with Marco's Sheevaplug instructions. You can install to a USB stick as well. Just remember that U-Boot needs a FAT filesystem containing the uImage on it and that you can't use a kernel in Debian yet. You'll also want to ignore the instructions regarding U-Boot, since that must be done differently. I'll be back later with some more details.

10 June 2011

Neil Williams: Putting the magic smoke back into the ring main

Haven't been in my new home that long and through a long chain of contacts, my neighbour was finally able to contact me. (Just goes to show that having Debian friends nearby who like board games turns out to make some things JustWork.)
So, when I got the message and got home the flat was fine - just with no electrical power. Murphy's Law ensured that my laptop was also low on power, as was my mobile phone and my landline is cordless and the router was, naturally, not working. My communication options were suddenly rather limited.

The 50yr old wiring in this block finally surrendered the magic smoke and caused a fire downstairs (mostly melted insulation) before it disconnected. I got home around 1pm, electricians to fix the problem arrived about 5pm and haven't finished quite yet. (It's now 12:11am). A relatively large hole in the lawn was dug (a hard task in itself seeing as this area is officially in a drought and the ground is like iron), two junction boxes rebuilt, new connection made from downstairs to upstairs, the work just went on and on.

I'll get some accurate details when I manage to speak to my brother - I do a little bit of hardware but I deal in 1W or a few hundred mA, when a 80A cable melts I'm just a bit out of my depth. I've not seen a fuse that size since I left school.

Thanks to those Debian and non-Debian people who helped me actually contact the relevant people today. I was due to get so many things done today, instead I got through the worry to end up incredibly bored - it turns out to be surprisingly difficult to find things to do when there is no mains power, all the batteries in mobile devices are flat and it's too dark to read.

Nathan Handler: Google Summer of Code Report #1

Here is my first official Google Summer of Code report. As I still had classes, the first week of coding was rather slow. However, after my first phone call with my mentor, Matt Zimmerman, things started to pick up. I spent most of my time working on a dashboard/portal for the Debian Derivatives Exchange (DEX). We decided that the best approach to this would be to store as little information ourselves as possible. Instead, we will try to pull most of our information from other sources, such as Debian s BTS. The main advantage to this is that package maintainers do not need to change any of their workflows, but they still get to benefit from DEX s work. Currently, I have a python script that gets a list of usertags and bugs for debian-derivatives@lists.debian.org from the BTS. This script will end up getting run by cron. It stores the information it gets in static HTML files. Finally, JQuery and Javascript are used to present the final dashboard to the user. I created a mockup showing how this looks. I also have a functioning website that looks and acts like the mockup, but the code is not publicly available yet (see below). DEX uses Alioth to host its git branches and website. I have spent some time trying to familiarize myself with the recent changes that have taken place on Alioth, as well as trying to figure out the best way to store and execute my script to generate the website. So far, it looks like people are still deciding on the best way to handle having websites that are updated from VCS hooks (due to things being split between vasks and wagner). I will continue to look into this, and might try using an alternate host until things settle down. Next, I plan to work on updating the dashboard to provide a way to edit, store, and display certain extra fields of information. This is necessary for certain pieces of information that are not appropriate for adding to bugs and in a few other cases. I will also be taking some time to talk to a few people who were active contributors to the last DEX project. This will allow me to learn more about their experience, find out what changes they feel should be made to make things run smoother, and get feedback on my current Summer of Code work. Finally, although I have been posting updates on my Summer of Code work to my blog [2][3], Matt thinks that having all of this information in one place will be useful. As a result, I will be creating another wiki page about the project that will contain all of the information people need to know. This page will be updated as the summer progresses. I am also hoping to have my code available in a public VCS by the time of my next progress update so that people can check it out and provide feedback. In addition to the dashboard, I will be working on creating a few other tools to assist with the creation and tracking of new DEX projects. I will also be coming up with a way to generate graphs to visually allow us to track our progress on projects versus time as well as show the work done by people at the individual level. This last task is part of a goal to get more people involved in the DEX project. While we may only be a few weeks into the actual Google Summer of Code, I have already learned a lot of new things. I am looking forward to working more with Matt and other members of the community over the course of the summer.

Gunnar Wolf: Reading revolutions: Online digital text and implications for reading in academe A (very informal) review

It's been a long time since I last took some time to read First Monday A great online publication, if you are not familiar with it, that I would categorize (and no, I'm not probably well-informed in it to be authoritative) as dealing with social, psychological aspects of the cultural shifts the online world has brought upon us (often dealing with topics related to Free Software communities, the reason I first met the publication). Firstmonday is an Open Access champion from early on. It follows an approachable but academic format (this means, it is peer-reviewed, its articles give extensive lists of references, and the articles are not the short reads we have got used to finding on the net, but, quoting from their audience profile, English is not the first language of many First Monday readers; A large percentage of First Monday readers are not a part of academia; Cultures, educational backgrounds, and fields of study vary greatly among First Monday readers.) This means, it's at least a great publication for me to follow :) Anyway After a long time not following it, I have just read Reading revolutions: Online digital text and implications for reading in academe, by Barry W. Cull; First Monday, Volume 16, Number 6 - 6 June 2011 Nice, interesting read. As I was planning on telling about the article to a couple of friends more into the subjects than myself, I'll comment+quote some bits on it. Before going any further: The article makes several references to Maryanne Wolf. No relation to her I'm not lulling my (two? are you both still reading?) readers towards her work ;-) The article talks about the differences social, even dips into some physiological aspects that the activity of reading is sustaining due to the shift from an activity done mainly from books (or other similar printed material) to the computer screen. Of course, we all know from our own experience many of the basic traits Shorter attention spans, a different reading pattern (skimming instead of reading; browsing through several related items instead of in-depth reading a single text as a knowledge unit). The article begins with an overview of reading and humankind. Cull quotes Maryanne Wolf's phrase, despite the fact that it took our ancestors about 2,000 years to develop an alphabetic code, children are regularly expected to crack this code in about 2,000 days . An interesting point I never thaught of is the start of reading as a purely mental activity, detaching reason from verbalization, ~1200 years ago:

Interestingly, these early scribes first did their work by reading out loud to themselves. Not until the ninth century did monastic regulations begin requiring silent reading. By the thirteenth century the practice of men reading silently and alone became commonplace. This shift to silent reading was a profound change, one that Darnton suggested involved a greater mental adjustment than the shift to printed text

. One last important point in this older history, that I'm quoting because I know I'll need the reference later on for one of my texts is about reading as a social activity Yes, also related to the quietness I just mentioned:

Interestingly, with the democratization of the printed text, there was a return to reading aloud. Reading was a solitary silent process only for the educated elite who could afford to buy books. For the rest of the population, as Darnton pointed out, reading was a social activity which took place in workshops, barns, and taverns [and] while children played, women sewed, and men repaired tools.

After this introduction (obviously one of the most interesting parts to me), Cull gives numbers showing how reading is evolving (in the USA and Canada), and quotes some prediction on how the future will end up adapting. Of course, I live in a place with a very different society, so I cannot comment much. Then he confronts some studies regarding specifically leisure reading, as it is a much more trustable factor than just literacy (in a world as highly literate as ours is, many people only read when they have to and have never or very seldom experienced the pleasure of reading just for the sake of it), bringing into the discussion the Internet (and computers in general) usage patterns. I found also very interesting the next section, regarding the pattern changes many libraries are facing now, specially academic/research-oriented libraries:

For several millennia, right up until just two decades ago, the central role of a library was to collect and house physical texts: from clay tablets, to scrolls, to printed books (Battles, 2003; Manguel, 2006). While printed text remains essential to most academic libraries, today s libraries have also become a core conduit via which researchers access scholarly texts online. Just within the last few years, Canadian academic libraries, in a situation similar to libraries throughout the Western world, have reached an interesting tipping point librarians now spend the majority of their collections budgets on electronic instead of printed texts.
( )
Libraries are convinced that digital text, now in its infancy, is likely to have a long future. Not only do they purchase electronic texts, but most academic libraries have also become publishers of electronic texts, whether they are digitizing large portions of their book holdings, or focusing on scanning a relatively small number of archival documents from their unique special collections.

And yes, doing some work with our Institute's library, I can confirm this trend. About e-books: I have got quite into that topic since the Kindle won my heart (and my money!) half a year ago. The little device completly changed my reading habits, I have read lots more since I carry it. And yes, I have never considered a full tablet-like device The article talks long about the difference, about the disadvantage that multitasking means to the human brain (oh, do I suffer from it!) I liked this snippet, quoting Steve Jobs a couple of years ago, regarding an Apple e-book reader question:

When Apple was rumoured to be working on an e book reader a few years ago, CEO Steve Jobs expressed his lack of interest: It doesn t matter how good or bad the product is, the fact is that people don t read anymore, he reportedly said, continuing by stating that 40 percent of the people in the U.S. read one book or less last year. The whole conception is flawed at the top because people don t read anymore (Markoff, 2008). Nicholas Carr (2010) has summed up Apple s involvement in the tablet phenomenon this way: Jobs is no dummy. As a text delivery system, the iPad is perfectly suited to readers who don t read anymore .

The issue for me is, I do enjoy reading, but I am an information addict. I know that if I have parallel information flows, my attention will surely dilute between them. Of course, as I read this article on-screen, it was hard for me to take the needed discipline not to be distracted by IMs or IRC highlights during the whole reading (which was also as an excercise for myself ;-) ). I am surprised to see this on student preferences (and even more surprised to see this data comes from my university):

In a study of students at the Universidad Nacional Aut noma de M xico (UNAM), the majority of students preferred print, and 63 percent reported that they could bear reading a document on a computer screen for no more than one hour (Ram rez Leyva, 2003). When it comes to course textbooks, a marked student preference for paper over e books has recently been found (Woody, 2010).

As we approach the end, it talks about another important topics I have often tried (and often failed) to communicate to my users: That of paratext, the meaning of the different texts, covers, items, layouts, etc. that are not part of the text itself but do shape the way we face it. To some of us, this seems obvious. To others, it is so hard to understand It closes with two more topics I will refer to. One is the permanent connectivity. All the time, more people are connected virtually all of their waking time. This affects not only learning habits but priorities. Will this near constant access to information interfere with students desire to comprehend and remember information, necessary to the educational process of turning it into knowledge? Author and university business school lecturer Don Tapscott recently suggested that students might not have to stress about the details those you can check Finally, regarding the continuity and ellaboration found in texts that are each time more common He quotes Maryanne Wolf:

I am worried about kids who are immersed in digital culture. They will get to college and they will have been Twittering so much that they won t have the patience to read those really long cognitively convoluted and complex sentences. They may not have developed those rich networks which are required in order to read at a high level of sophistication. The effort is what we are going to lose. They are becoming not so much a lazy reader, but an atrophied reader

As you can see, not only this post is meant to tell my couple-of-interested-friends to read the article, but it's mainly meant as a mental placeholder for myself. I will be surely refering to some of these items.

Axel Beckert: How to find broken symlinks

Looking through the man page of find there is no obvious way to find broken symbolic links. But there is a simple way involving only find:

$ find -L . -type l
$ find -L . -type l -ls

The option -L (before the path!) causes find to follow symbolic links and the expression -type l causes find to report only symbolic links. But as it follows symlinks, it only reports those it can t follow, i.e. broken ones. The second line also shows where the broken links point to. To easily show that this really works, just use the color indicator of GNU ls instead of find s builtin -ls:

$ find -L . -type l -exec ls -lF --color=yes ' ' +

Et voil , all displayed links show up in red which means they re broken. Kudos to CodeSnippets for showing me the right idea. And thanks to ft of zsh and grml fame for the hint about find -exec command + instead of

find -exec
command   ;

. Hint from mika of grml fame: With zsh it is even less code to type:

% ls **/*(-@)
% ls -lF **/*(-@)

Thanks, mika!

Raphaël Hertzog: People behind Debian: Philipp Kern, Stable Release Manager

Philipp is a Debian developer since 2005 and a member of the release team since 2008. Since he took the responsibility of Stable Release Manager, the process has evolved for the best. I asked him to explain how the release team decides what s fit for stable or not. His work on the buildd infrastructure is also admirable but I ll let him describe that. My questions are in bold, the rest is by Philipp. Who are you? I m a 24 year old Computer Science student, living in Karlsruhe, Germany. As a student assistant I m currently in two jobs: one is taking care of getting our university IPv6 network in shape, or rather generally getting fringe technologies into the network. The second is taking care of a s390x machine in the basement which my faculty got sponsored recently. In my spare time I tend my Debian duties and I m active in the student council (Fachschaft) as a sys-admin, software developer and until recently as treasurer. I got accepted as a Debian Developer in 2005. I m only really active since I was invited to join the Release Team in early 2008 after I contributed rewrites of some scripts that got lost in a disk crash of ftp-master. In late 2008 I also took on wanna-build/buildd duties. What s your notable achievement within Debian? For one I worked on the deployment of a single buildd/sbuild combination over all of our buildds and I rebased it on the sbuild in the archive several times already. All buildds basically look the same on all architectures, with only few variations. The chroots are now also created in a mostly predictable manner. Then we finally got build autosigning after years of constant poking. However the policy decision to allow it was made by the ftp-masters anyway, for which I m grateful, as it eases the workload of the buildd maintainers, the Security Team and the Release Team quite a bit. On the release side there s the integration of volatile into the proper structures of Debian. But there I m guilty for choosing the bad name squeeze-updates is, in comparision with security s squeeze/updates. Let s see if we can improve that for wheezy. The policy for stable updates have changed over time. Can you summarize what kind of updates are allowed nowadays? All the changes that were made to the policy were driven by the aim to keep stable (and to some lesser degree oldstable) usable throughout its lifetime. I have to admit that we got slightly more liberal in what we accept since I took over. The previous stable update policy included the fixes of critical bugs that break the system in interesting ways and security fixes. We also opened up the possibility of including important fixes for annoying bugs on a case-by-case basis. The whole don t update that much part of the stable release management is rooted in let s don t change the behaviour of a running system and let s have as few regressions as possible . We currently try to counter that with a review process that only allows self-contained fixes that were tested in unstable first, if applicable. In the future I d also like to have a feedback process that includes the users of the packages to report problems more early. In fact that s also why we now send calls for testing to debian-stable-announce one week in advance (see this mail as an example). So stable is updated through point releases that happen roughly every two months for stable and roughly every four for oldstable. They are accompanied by announcements and new CD/DVD sets. To be able to push some updates with less delay to our users and to avoid that they have to pick them from proposed-updates, there is stable-updates. The current policy for this suite is available in this post on debian-devel-announce. It boils down to everything urgent like tzdata, regression fixes and volatile packages. We noticed that some packages in stable just weren t useful anymore when they got updated through volatile. Hence those are folded into stable, too. We also relaxed the rules a bit so that leaf packages like clive, that rely on external websites not to change their URLs, can be updated too. If somebody wonders what happened to the and a half releases Those were intended to mark the inclusion of new hardware support (mostly by including a co-installable new kernel version). This kind of flag point release is thankfully no longer needed because our marvellous kernel team now backports certain hardware drivers to the kernel version in stable. They have some trouble soliciting test feedback for them, though. It would be cool if more people using stable could respond to their calls for testing. What are your plans for Debian Wheezy? Luckily the wanna-build/buildd side isn t much dependent on release cycles. The stable release management also just takes care of what gets dumped on them by the testing RMs.

The near-term goals are certainly:

getting d-i to use some development process in the archive that s compatible with the normal flow of packages into testing and
hopefully getting wanna-build to use a sane SQL layout and hence better performance

Apart from that my work is mainly squashing the bugs on buildd.debian.org for the wanna-build part, so if you want something fixed, you need to report it, I m afraid. You have several roles within Debian, in particular stable release manager and member of the wanna-build team. Which role do you enjoy more and why? So the regular duties of the SRM, apart from setting up policies, are reviewing requests for uploads to proposed-updates, accepting them, chasing builds and scheduling point releases. It s mainly steady work. As for the Release Team it s working in a fabulous team where everyone s doing an awesome job. For me the wanna-build role is holding the things together and fixing up stuff that breaks while the world continues to evolve. And occassionally taking a stab at the ftp-masters. So rather different. My heart is probably siding with the buildd side because it s infrastructure but release work is fun, too. What s the biggest problem of Debian? Mainly that we re not quick in picking up new technologies and that we got slow on innovation. Distro-wide changes are usually a very big effort to coordinate and to finally complete. I guess people should be able to push fixes for release goals more eagerly into the archive. Also it seems to me that many DDs are actually disengaging from the release process, especially during the freeze but also before, which makes me a bit sad. However I m still not sure how we could counter that. As we re all volunteers you can t suddenly make others love the idea of releasing something together, I guess. Do you have wishes for Debian Wheezy? It s getting old, but there s certainly multiarch. Also a usable Python 3 is something I d really appreciate. Is there someone in Debian that you admire for their contributions? Peter Palfrader did an awesome job of getting a nicely working team together for DSA and got many tasks as automated as they are now. It s a pleasure to work with them and setting up a new buildd is, for example, a real breeze. He also developed a sane archive snapshot service, kudos for that.

Thank you to Philipp for the time spent answering my questions. I hope you enjoyed reading his answers as I did. Subscribe to my newsletter to get my monthly summary of the Debian/Ubuntu news and to not miss further interviews. You can also follow along on Identi.ca, Twitter and Facebook.

3 comments Liked this article? Click here. My blog is Flattr-enabled.

8 June 2011

Joey Hess: date formats of a decade of usenet

I've finished importing the usenet archive for oldusenet. The fun part was parsing the dates to put the posts in order. No date format was really required on usenet, and so a wide variery of formats were used. Some posts didn't have a Date, but a guess could be made from their Message-ID. Some posts had absurd dates (ie, 1969, 1995), others had dates that were correct in every way.. except the year was left out (oops). One early post had a date of "_". Still, this excerpt of my code managed to parse the rest and so gives a fairly complete picture of how messy dates can possibly be. Read and weep.

  p anyzone "%d %b %y %T"       "15 Jun 88 02:27:41 GMT"
, p anyzone "%a, %d %b %y %T"       "Thu, 22 Jun 89 20:02:03 GMT"
, p anyzone "%a, %d-%b-%y %T"       "Thu, 15-Jun-89 18:01:56 EDT"
, p anyzone "%d %b %y %T"       "8 Jan 90 14:07:27 -0400"
, p anyzone "%d %b %y %H:%M"        "4 Oct 89 19:56 GMT"
, p anyzone "%a, %d %b %y %H:%M"    "Thu, 23 May 91 02:13 PDT"
, p anyzone "%a, %d %b %Y %T"       "Thu, 23 May 1991 07:07:00 -0400"
, p anyzone "%a, %d %b %Y %H:%M"    "Sat, 18 May 1991 17:28 CDT"
, p anyzone "%d %b %Y %T"       "11 Apr 1991 12:02:01 GMT"
, p anyzone "%d-%b-%y %H:%M"        "24-Mar-90 14:22 CST"
, p anyzone "%d %b %y, %T"      "22 May 91, 16:31:37 EST"
, p anyzone "%d %b %Y %H:%M"        "30 June 1991 17:15 -0400"
, p anyzone "%a, %d %b T  %T"       "Fri, 8 Feb T  09:49:39 EST"
-- special cases
, p (tzconst est) "%a %b %d %T EST %Y"  "Tue Jan 11 12:44:36 EST 1983"
, p (tzconst est) "%a %b %d %T EST %y"  "Tue Jan 11 12:44:36 EST 83"
, p (tzconst edt) "%a %b %d %T EDT %Y"  "Tue Jan 11 12:44:36 EDT 1983"
, p (tzconst edt) "%a %b %d %T EDT %y"  "Tue Jan 11 12:44:36 EDT 83"
, p (tzconst utc) "%a %b %d %T GMT %Y"  "Thu Nov  1 23:14:37 GMT 1990"
, p (tzconst pdt) "%d %b %y %T -7"  "11 Jun 91 15:41:21 -7"
-- dates with no timezone specified are guessed
, p nozone "%d %b %y %T"        "9 Jan 90 09:33:59"
, p nozone "%d %b %Y %T"        "10 APR 1990 05:25:28"
, p nozone "%a %b %d %T %Y"     "Fri Feb  6 00:19:47 1981"
, p nozone "%a %b %d %T %y"     "Fri Feb  6 00:19:47 81"
, p nozone "%Y-%m-%d %T"        "1981-11-12 18:31:01"
, p nozone "%y-%m-%d %T"        "81-11-12 18:31:01"
, p nozone "%a, %d %b %y %T"        "Sat, 13 Apr 91 08:37:57"
, p nozone "%a, %d %b %Y %T"        "Sun, 16 Jun 1991 13:23:02"
, p nozone "%d %b, %Y %T"       "1 May, 1991 00:00:00"
, p nozone "%d %b %y %H:%M"     "8 Jan 88 18:03"
, p nozone "%a, %d %b %y %H:%M"     "Wed, 29 May 91 17:14"
, p nozone "1 %b %d %T %Y"      "1 Jan 08 20:59:08 1991"
-- this has to come near the end, as it matches greedily
, g nozone "%a %b %d %T %Y ("       "Wed Oct 27 17:02:46 1982 (Tuesday)"
, g nozone "%a, %d %b %y %T +"      "Tue, 21 May 91 16:46:01 +22323328"
-- extract date from message-id headers
-- (used for messages with no Date field)
, g nozone "<%Y%b%d.%H%M%S."        "<1989Jul6.214048.28313@jarvis.csri.toronto.edu>"

(Parsing the often ambiguous, malformed, etc timezones was fun all its own too, of course.)

Lars Wirzenius: Test driven distribution development

Disclaimer: There is a huge discussion going on right now in the debian-devel mailing list, about different ways to rearrange how Debian development happens, and maybe to provide a Debian users with an additional release. This blog entry is not a comment on that discussion, which I am ignoring. The discussion did, however, prompt a dent about "test driven development". That prompted a short exchange of thoughts with Tom Marble, and this blog post is a cleaned up version of my part in that discussion.

Currently Debian development happens roughly like this: all packages are uploaded to a part of the Debian archive called unstable. Once they've been in use for a while without serious problems, they get automatically copied into another part called testing. Every year or two, testing is frozen and any remaining problems are fixed, after which all of testing gets copied into yet another part of the archive called stable, and that's the actual release. There is a little bit of automatic testing, but only a little. Almost all testing is done by people, by them using unstable or testing on their usual computers. If they have problems, they report them. Contrast this with development using Test Driven Development, or TDD, and other modern development methodologies. Here's a rough summary of what I do when I write (much) of my software.

first write one or more automatic tests (unit or functional ones)
write the actual code, enough to make all tests pass
add more tests
write more code, or change existing code
repeat this until tests describe all the behavior desired of the code

In addition, I measure coverage to make sure all parts of the code gets tested. I usually aim for 100% coverage, except for those parts that very hard or quite pointless to test. (That's easier to achieve than you'd think, but that's the topic of another blog post.) This all sounds like a lot of bureaucratic nonsense, but what I get out of this is this: once all tests pass, I have a strong confidence that the software works. As soon as I've added all the features I want to have for a release, I can push a button to push it out. (Well, actually there's a little bit more to it.) This does mean I never have bugs in my software. Of course I do. However, there's a lot fewer of them. In fact, there's so few of them, after tests pass, that I would almost be happy to make an automatic release after every successful run of "make check". Another aspect of the way I do development is distributed version control. The relevant feature is powerful branching and merging. Most things I develop happen in short-lived single-purpose branches: whenever I start work on a feature or bugfix, I create a branch for it. (Unless I'm feeling particularly lazy, which happens more often than you'd think.) When I've finished the feature, or fixed the bug, I merge the branch back into the trunk branch. This way, the trunk is always in releaseable state. It might not have all the features I want for the next release, but the features that are there, do work. The branch probably has bugs, but if I've written tests that are good enough, I know the software works well enough. Or that's the idea. Sometimes things go wrong. Then I write more tests and the next time goes better. (It doesn't have to be perfect, as long as it gets better every time.) See the contrast between automatic tests with good coverage, and the Debian style of relying on user feedback? There's no need to wait for user feedback with automatic tests. This speeds up development, makes releasing easier, but most importantly takes away any reason to fear making changes, even big changes. Automatic tests and good test coverage are easy achieve in small projects. For a system as huge as Debian, good test coverage is quite hard to achieve. The thing about automatic tests, though, is that even a little bit of it is helpful, and after you have a few tests, it gets easier to add more. The first test is the hardest to get done, since you need to set up all the infrastructure for it. Debian does do a bit of automatic testing already. (See lintian, piuparts, autopkgtest, edos, etc.) I don't want to belittle that, but I think we could do better. Here's what I would love to see:

we have a part of the archive that corresponds to a trunk branch, i.e., it is always in a releasable state (the "testing" area was originally meant to be that; we could make it so now)
releaseable state is determined by two things:
- an automatic test suite
- user feedback, particularly in the form of bug reports (as now)
whenever changes are made, they happen in "branches"
- the branch is not affected by changes elsewhere in the archive, except by manual synching ("merge from trunk")
- individual package uploads, as well as groups of packages such as for transitions, are each in their own branch
- this is sort of similar to a PPA, but more powerful
when a branch is to be merged into trunk, the automatic tests must first pass (or at least no new failures in them can be introduced)
- tests can also be run for any other branch, of course, so that those developing the branch know if they're ready to push their changes into trunk
there's a culture of writing tests for bugs (whenever that is feasible), and for new features
- particularly release goals should be expressed as automatic test suites
there's a culture of sharing tests with upstreams, and with other distributions

Since full test coverage is going to be impossible for Debian, some subset should be targeted. Perhaps something like this would suffice to start with:

a version of debian-installer is generated from a branch
test installation of a new system from scratch
- with a large set of tasks selected
test upgrade from previous stable release to the branch
- with a large set of packages installed (as many as possible, actually)
test upgrade from trunk branch to branch to be merged
- ditto large set of packages
test with lintian, reject on specific tests failing
test with piuparts
test the whole system for specific functionality
- ssh access from outside
- sudo access for a logged in user
- sending mail with SMTP
- web server
- possibly test specific web applications
- possibly test Samba and NFS services
- possibly test printing services (CUPS)
- possibly test essential desktop functionality (automatic login, at least)
if package comes with package specific tests, run those too

These tests would not guarantee that a set of changes would not break Debian, but they would give a high confidence that at least basic stuff would still work after the changes. Now, obviously implementing all of this I'm dreaming of (it is just past midnight, after all) is going to be impossible. There's way too much work to do, there's not enough tools for writing tests, and it would require too much computing power to run, and so on and so forth. But it's late, and I've had a bad day, and I might as well dream of something nice. Anyway, anyone interested in these things should perhaps help drive DEP8.

7 June 2011

Vincent Fourmond: Record sound currently playing to file using ALSA

I've been wondering for a long time about how to record sound which is currently being played on the computer. I know the ALSA framework is powerful enough to do that, and today just proved me so (at least to some extent). It turns out that it's pretty trivial using the file PCM plugin; all you need is a stanza like this in your $HOME/.asoundrc file:

pcm.save  
    type file
    format wav
    slave.pcm front 
    # my main PCM is called front
    file output.wav

Afterwards, you can use this save PCM with the tools that understand it, such as aplay or mplayer:

~ aplay -Dsave stuff.wav
~ mplayer -ao alsa:device=save stuff.wav

(for xine users, you may want to have a look at the audio.device.alsa_front_device configuration element). Funnier, though, is the possibility to use it with programs that don't provide such an easy way to change that, (say, the flash player in a browser ?) via the use of the default PCM specification in $HOME/.asoundrc:

pcm.!default save

After that, all programs using the default PCM will save the sound they play as a save.wav file in their current directory - but nothing stops you from specifiying an absolute directory. Better yet, you may apparently use pipes in stead of the file name, la popen.

Lars Wirzenius: TDDD systest tool proof of concept

Continuing my earlier musings about test driven distro development, and the tools that would require. I imagine something like this:

create a virtual machine with a particular configuration of stable
run a test suite verifying the VM works
upgrade the VM to testing and reboot
re-run test suite, verifying things work in testing

We might also want this scenario:

install testing directly in a VM (instead of upgrading from stable)
run test suite

There might be other scenarios that would be useful to test as well. Even from these two, it's clear there's a need for at least three separate tools:

create a machine image with a particular configuration
- might be based on stable or testing or unstable
- might also be using some other package sources
- might want particular packages installed
- might want particular configuration settings
run a suite of tests against the running image
- which tests to run will depend on the scenario
run various scenarios using the above two tools
- needs reasonably easy ways to specify scenarios
- needs to support tests specific to scenarios

I've written a rudimentary version of the first tool: vmdebootstrap. I've since learned there's a bunch of others that might also work. There's room for more: someone should write (or find) a tool to make a snapshot of a real system and create a VM image that mimicks it, for example. Anyway, for now, I'll assume that one of the existing tools is good enough to get started. For the second tool, I wrote a quick-and-dirty proof-of-concept thing, see systest.py. Here's a sample of how it might be used:

liw@havelock$ ./systest -v --target 192.168.122.139 --user tomjon
test 1/6: cat
test 2/6: only-ssh-port
ERROR: Assertion failed: ['22/tcp', '139/tcp', '445/tcp'] != ['22/tcp']
[status 1]
liw@havelock$ ./systest -v --target 192.168.122.139 --user tomjon \
    cat ping6-localhost ping-localhost simple-dns-lookup ssh-login
test 1/5: cat
test 2/5: ping6-localhost
test 3/5: ping-localhost
test 4/5: simple-dns-lookup
test 5/5: ssh-login
liw@havelock$

The first run failed, because the VM I'm testing against has some extra ports open. Some of the tests will require logging into the machine, via ssh, and for that one needs to specify the user to use. systool may overlap heavily on system monitoring tools, and possibly the production implementation should be based on those. I think it's best to design such a tool for the more general purpose of testing whether a system currently works rather than as an integrated part of a more specific larger tool. This lets the tool be useful for other things than just testing specific things about Debian. (The production implementation would then need to not have all the tests hardcoded, of course. SMOP.) The third tool I have not spent a lot of time on yet. One thing at a time. Given these tools, one would then need to decide how to use them. The easiest way would be to use them like lintian and piuparts: run them frequently on whatever packages happen to be in testing or unstable or experimental, and put links to the test reports to the PTS, and hope that people fix things. That is the easiest way to start things. Once there's a nice set of test cases and scenarios, it may be interesting to think about more aggressive ways: for example, preventing packages from migrating to testing unless the test suite passes with them. If the tests do not pass, one of four things is broken:

the package or packages in question
other packages already in testing
the tests themselves
the test environment

If things are set up properly, the last one should be rare. The other three always require manual inspection: it is not possible to automatically know whether the test itself, or the code it tests, is at fault. it is, however, enough to know that something is wrong. If the tests are written well, they should be robust enough to not be the culprits very often. (Someone wanting to make a rolling distrbution, or even better, making a monthly mini-release, might benefit from this sort of automated testing.)

Next.

Previous.