Search Results: "spa"

19 August 2011

Alexander Reichle-Schmehl: How to read ftp-masters package numbers table

In my previous blog about Debian's growing archive size I pointed as reference to an hard to read table on the ftp-master server, and missed that the table is not self explanatory and on the first glance hard to read. The table currently looks like the following: 
                  e      s     p-u     t      u    t-p-u   l-r    s-u     o    o-p-u   s-r   
---------------------------------------------------------------------------------------------
     all         1096  12034     70  14442  15784     20      1     12   8910    230      0  
    alpha           -      -      -      -      -      -      -      -  13183    297      -  
    amd64        1164  16997    242  18581  19605     39      -     21  13964    342      -  
     arm            -      -      -      -      -      -      -      -  13251    291      -  
    armel         988  16401    239  17879  18680     39      -     21  13411    327      -  
     hppa           -      -      -      -      -      -      -      -  13227    264      -  
  hurd-i386       445      -      -      -  12622      -      -      -      -      -      -  
     i386        1178  17188    242  18727  19751     39      -     21  14377    350      -  
     ia64         951  16036    239  17303  18045     39      -     21  13554    329      -  
kfreebsd-amd64    807  14499    231  15905  16480     39      -     21      -      -      -  
kfreebsd-i386     826  14491    231  15881  16498     39      -     21      -      -      -  
     mips         974  16269    241  17730  18504     39      -     21  13638    323      -  
    mipsel        984  16291    239  17807  18558     39      -     21  13600    319      -  
   powerpc       1078  16677    239  18156  19036     39      -     21  13872    332      -  
     s390        1030  16252    239  17678  18469     39      -     21  13331    327      -  
    source        522  14974     51  16376  17405      1     19      6  12519     75    151  
    sparc         992  16423    239  17895  18673     39      -     21  13542    329      -
The lines represent different hardware architectures Debian supports, like amd64 (for 64-Bit PCs), i386 (for 32-Bit PCs), etc. Two lines are kind of special, the one marked source and the one marked all. Let's start with the source line: That counts the number of source packages. Those are rarely seen by end users. Source packages contain the actual source needed to build a package, as well as Debian specific changes and a blue print on how to create an actual installable binary package from the source. As you can see Debian supports quite some architectures, which leads to a small problem on the mirrors: Quite a lot shipped in Debian is actually architecture independent. For example documentation, most sound and graphic files, or level data for a game. To avoid storing the very same date multiple times, Debian makes haeavy use of so called arch: all packages. You might have noticed them already while installing them: 
Hole:5 ftp://ftp.rrzn.uni-hannover.de/debian/debian/ squeeze/main extremetuxracer-data all 0.4-4 [28,1 MB]
Hole:6 ftp://ftp.rrzn.uni-hannover.de/debian/debian/ squeeze/main extremetuxracer amd64 0.4-4 [273 kB]
In this example I installed a quite small game on the amd64 architecture, which depends on a quite big architecture independent package (notice the all between package name and version) containing it's data.So, to get the actual number of packages a user can install on a specific architecture, one has to add the number in the all line to the number in the specific architecture line. But what column to pick? The columns represent so called suites. The s is for stable, currently Debian 6.0 aka Squeeze and o for oldstable, aka Debian 5.0 Lenny. The t is for testing, aka Wheezy or the upcoming Debian 7.0. The u is for unstable aka sid. The e is for experimental, used for stuff probably not yet ready to be part of a stable release and therefore not uploaded to unstable. The rest is not that interesting (as you see it's mostly empty anyway), like s-u for stable updates, several p-u suites for proposed-updates (that is stuff that might end up in the next point release, and should be tested). And then we have s-r and l-r which I forgot what they are about... Sorry. But back to topic: As I explained briefly a source package is mostly used internally, but may create several binary packages, which a user can install. So the most interesting number for users is the number of binary packages on an arch. So we usually publish this number. So for example, using squeeze on i386 you can install: 17'188 (architecture dependent packages, column s row i386) + 12'034 (architecture independent packages columns s row all) = 29'222 packages. And for completeness: They are build from 14'974 source packages (row source column s). So if look that up for unstable on amd64 you should get to the result 35'389 binary packages build from 17'405 source packages. Slightly less than I reported yesterday, because we removed some old stuff e.g. obsolete libraries in the meantime. Update: J rg just explained me, that the s-r and l-r are there to ensure Debian complies with the GPL. They hold the sources of everything shipped in the original lenny and squeeze releases. Update 2: J rg explained it to me again: s-r and l-r contains sources used without being referenced by a package, and so might get lost otherwise. One example is the kernel used by the debian installation system. You have to boot a kernel somehow, so it can't be packaged. But we still need to save the sources, so it is referenced in these suites and won't get lost.

Vincent Bernat: jchroot: chroot with more isolation

Linux 2.6.24, released more than three years ago, introduced two very interesting features: PID and network namespaces. They allow to create a process that does not share the same PID namespace or the same network namespace than other processes. Those namespaces are then inherited by the children (unless they are assigned new namespaces).

PID & mount namespaces A PID namespace is a view of the processes running on the system. Each PID namespace will have its own numbering of the processes contained in its namespace (including the namespace of the children). Other processes are hidden and therefore, it is not possible to issue any operation requiring a PID on them, like kill(2). Moreover, the process that was created first in a namespace will get PID 1. If it gets killed, the namespace is destroyed and all other processes contained in this namespace are killed as well. systemd, a system and service manager for Linux, targeted as a replacement for the old SysV init, uses could use this feature to launch daemons in their own PID namespace and keep track of them without relying on their cooperation. When it needs to kill a daemon, it just kills the process with PID 1 (which has another PID in the global namespace) and the kernel will do the hard work of killing all processes that were started in this namespace. You can get more details about this namespace in the manual page of clone(2) (search for CLONE_NEWPID). UPDATED: In fact, systemd does not use PID namespaces but cgroups to group processes together. There is no PID numbering here but spawned processes are all enclosed into one cgroup and therefore can be killed easily. For example, here is the result of pstree -p inside a PID namespace: 
root@guybrush:/# pstree -p
bash(1) pstree(8)
         sleep(7)
The PID of sleep is 7. In the global namespace, it is 31762 instead. But it is the same processus! 
root@guybrush:/# ps -eo pid,command   grep sleep
31762 sleep 1000
Linux have also supported mount namespaces since a long time. Each process can have its own mount points that will be visible only by processes in the same namespace or in children namespaces. When the namespace is destroyed (when no process left), the kernel will do the cleanup itself.

chroot A common way to test experimental software or do packaging work is to build a chroot: debootstrap is used to build a new complete system inside a directory and chroot allows to enter into this new system. Tools like schroot can become handy if you need to do that often. Unfortunately, you only get an isolated filesystem. When you start launching daemons and mounting partitions, you need to keep track of all your actions to undo them. Quitting the chroot will not kill any daemon nor unmount any partition. Extending an existing tool, like schroot to use PID and mount namespaces seems the best way to prevent such problems. I have filed bug #637870 for this purpose. However, this is not as simple as it seems since schroot allows to run a process into an existing session. When the session is a chroot, this is quite easy: just chroot into the same root filesystem and launch the process here. With other kind of namespaces, you need to use a new system call, setns(), to change the current namespace of the running process. However, some bits are still missing. First, the GNU C Library does not implement the appropriate wrapper. Second, Linux (as of 3.0) does not contain the support to change the PID namespace of a running process yet. Other tools exist, like lxc (aka Linux Containers). However, it is more targeted at providing a complete container. I have reported that lxc-execute is not able to spawn a standalone shell but got no luck. UPDATED: systemd ships with systemd-nspawn which was exactly what I needed. However, it requires systemd. Otherwise, it seems less basic than the alternative that I describe below (some sensible mount points are setup, the console is correctly configured, signals are correctly transmitted, ). If you use systemd, look at it!

jchroot Building a chroot clone using namespaces is quite easy. The only difficulty is the use of clone(2). Here is a sketch: 
int main()  
  pid_t pid;
  int ret;
  long stack_size = sysconf(_SC_PAGESIZE);
  void *stack = alloca(stack_size) + stack_size;
  /* New process with its own PID/IPC/NS namespace */
  pid = clone(step2,
              stack,
              SIGCHLD   CLONE_NEWPID   CLONE_NEWIPC  
              CLONE_NEWNS   CLONE_FILES,
              NULL);
  /* Wait for the child to terminate */
  while (waitpid(pid, &ret, 0) < 0 && errno == EINTR)
    continue;
  return WIFEXITED(ret)?WEXITSTATUS(ret):EXIT_FAILURE;
 
static int step2(void *a)  
  /* Mount /proc as an example */
  mount("proc", "/proc", "proc", 0, "");
  /* Chroot */
  chroot("/srv/debian");
  chdir("/");
  /* Drop privileges */
  setgid(1000);
  setgroups(0, NULL);
  setuid(1000);
  /* Exec shell */
  execl("/bin/bash", "/bin/bash", NULL);
Add command flags, error handling and fstab parsing for the definition of mount points and you get jchroot. Here is an example of use: 
$ cat fstab
proc     /proc  proc    defaults                  0  0
sys      /sys   sysfs   defaults                  0  0
/home    /home  none    bind,rw                   0  0
/dev/pts /dev/pts none  bind,rw                   0  0
/var/run /var/run tmpfs rw,nosuid,noexec,mode=755 0  0
/etc/resolv.conf /etc/resolv.conf none bind,ro    0  0
$ sudo jchroot -f fstab -n test /srv/debian/squeeze /bin/bash
# hostname
test
# ps auxww
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0  19480  1888 pts/1    S    15:18   0:00 /bin/bash
root         3  0.0  0.0  16536  1184 pts/1    R+   15:18   0:00 ps auxww
# /etc/init.d/postgresql start
Starting PostgreSQL 9.0 database server: main.
# exit
$ ps auxwwc   grep postgres
$
I hope that jchroot won t live for too long and that its features will be integrated into regular chroot tools. I was thinking at proposing a patch for chroot(8). Anyone knows if this would have a chance to be accepted?

18 August 2011

Thorsten Glaser: How not to create DEB files

Once upon a time, there was Deb and Ian. That was about exactly 18 years ago. We don t talk about the 0.939000 format any more, but they eventually settled on:

	$ ar rc pkg_1.0_all.deb debian-binary control.tar.gz data.tar.gz
	$ hexdump -C pkg_1.0_all.deb   head
	00000000  21 3c 61 72 63 68 3e 0a  64 65 62 69 61 6e 2d 62   !<arch>.debian-b 
	00000010  69 6e 61 72 79 20 20 20  31 33 31 33 36 38 33 35   inary   13136835 
	00000020  32 39 20 20 31 30 30 36  20 20 32 30 30 20 20 20   29  1006  200    
	00000030  31 30 30 36 34 34 20 20  34 20 20 20 20 20 20 20   100644  4        
	00000040  20 20 60 0a 32 2e 30 0a  63 6f 6e 74 72 6f 6c 2e      .2.0.control. 
	00000050  74 61 72 2e 67 7a 20 20  31 33 31 33 36 38 33 35   tar.gz  13136835 
	00000060  32 39 20 20 31 30 30 36  20 20 32 30 30 20 20 20   29  1006  200    
	00000070  31 30 30 36 34 34 20 20  31 33 39 31 20 20 20 20   100644  1391     
	00000080  20 20 60 0a 1f 8b 08 00  00 00 00 00 00 03 ed 59      ............Y 
	00000090  eb 6f db 36 10 f7 d7 f0  af b8 3a 5e 9b 74 b1 f5   .o.6......:^.t..

By then, systems were a.out(5), and everything was good. (Of course, if you look at the mtimes, you ll notice I faked this. But it s really equivalent to the real thing.

But oh horror! GNU binutils, not always everyone s friend, switched from using BSD style Unix Archiver libraries in ar(1) to SYSV style libraries on elf(5) systems:

	$ ar rc on-elf debian-binary control.tar.gz data.tar.gz
	$ hexdump -C on-elf   head
	00000000  21 3c 61 72 63 68 3e 0a  64 65 62 69 61 6e 2d 62   !<arch>.debian-b 
	00000010  69 6e 61 72 79 2f 20 20  31 33 31 33 36 38 33 35   inary/  13136835 
	00000020  32 39 20 20 31 30 30 36  20 20 32 30 30 20 20 20   29  1006  200    
	00000030  31 30 30 36 34 34 20 20  34 20 20 20 20 20 20 20   100644  4        
	00000040  20 20 60 0a 32 2e 30 0a  63 6f 6e 74 72 6f 6c 2e      .2.0.control. 
	00000050  74 61 72 2e 67 7a 2f 20  31 33 31 33 36 38 33 35   tar.gz/ 13136835 
	00000060  32 39 20 20 31 30 30 36  20 20 32 30 30 20 20 20   29  1006  200    
	00000070  31 30 30 36 34 34 20 20  31 33 39 31 20 20 20 20   100644  1391     
	00000080  20 20 60 0a 1f 8b 08 00  00 00 00 00 00 03 ed 59      ............Y 
	00000090  eb 6f db 36 10 f7 d7 f0  af b8 3a 5e 9b 74 b1 f5   .o.6......:^.t..

Can you spot the difference?

Of course, ELF is what you want , so there is little choice. Unix Archiver libraries are system dependent, and no format has ever been normed, but DEB files use it as format so what is one to do?

	$ GNUTARGET=a.out-i386-linux ar rc with-aout \
	> debian-binary control.tar.gz data.tar.gz
	$ md5sum pkg_1.0_all.deb with-aout on-elf
	248f78d42f8ca8f2a3560f9800b2bf01  pkg_1.0_all.deb
	248f78d42f8ca8f2a3560f9800b2bf01  with-aout
	09eca70c9b11b6b55bbadcab5c3201fb  on-elf

OK, and what do I do on my Debian/m68k system?

ar(1) uses bfd, and GNU binutils can not only forcibly set the target emulation but also show them:

debian_m68k$ ar -h 2>&1 grep '^ar: supported targets'
ar: supported targets: elf32-m68k a.out-m68k-linux elf32-little elf32-big plugin srec symbolsrec verilog tekhex binary ihex trad-core
debian_i386$ ar -h 2>&1 grep '^ar: supported targets'
ar: supported targets: elf32-i386 a.out-i386-linux pei-i386 elf32-little elf32-big elf64-x86-64 elf32-x86-64 pei-x86-64 elf64-l1om elf64-k1om elf64-little elf64-big plugin srec symbolsrec verilog tekhex binary ihex trad-core
debian_i386$ ar -h 2>&1 grep '^ar: supported targets' # binutils-multiarch
ar: supported targets: elf32-i386 a.out-i386-linux pei-i386 elf32-little elf32-big elf64-alpha ecoff-littlealpha elf64-little elf64-big elf32-littlearm elf32-bigarm elf32-hppa-linux elf32-hppa elf64-x86-64 elf32-x86-64 elf64-l1om elf64-k1om elf64-ia64-little elf64-ia64-big pei-ia64 elf32-m68k a.out-m68k-linux coff-m68k versados ieee a.out-zero-big elf32-tradbigmips elf32-tradlittlemips ecoff-bigmips ecoff-littlemips elf32-ntradbigmips elf64-tradbigmips elf32-ntradlittlemips elf64-tradlittlemips elf32-powerpc aixcoff-rs6000 elf32-powerpcle ppcboot elf64-powerpc elf64-powerpcle aixcoff64-rs6000 aix5coff64-rs6000 elf32-s390 elf64-s390 elf32-shbig-linux elf32-sh-linux elf32-sh64-linux elf32-sh64big-linux elf64-sh64-linux elf64-sh64big-linux elf32-sparc a.out-sparc-linux elf64-sparc a.out-sunos-big pei-x86-64 elf32-m32r-linux elf32-m32rle-linux elf32-spu plugin srec symbolsrec verilog tekhex binary ihex trad-core
mirbsd_i386$ ar -h 2>&1 grep '^ar: supported targets'
ar: supported targets: elf32-i386 coff-a29k-big a.out.adobe aix5coff64-rs6000 a.out-zero-big a.out-mips-little epoc-pe-arm-big epoc-pe-arm-little epoc-pei-arm-big epoc-pei-arm-little coff-arm-big coff-arm-little a.out-arm-netbsd pe-arm-big pe-arm-little pei-arm-big pei-arm-little b.out.big b.out.little efi-app-ia32 efi-app-ia64 elf32-avr elf32-big elf32-bigarc elf32-bigarm elf32-bigarm-symbian elf32-bigarm-vxworks elf32-bigmips elf32-cr16c elf32-cris elf32-crx elf32-d10v elf32-d30v elf32-dlx elf32-fr30 elf32-frv elf32-frvfdpic elf32-h8300 elf32-hppa-linux elf32-hppa-netbsd elf32-hppa elf32-i370 elf32-i386-freebsd elf32-i386-vxworks elf32-i860-little elf32-i860 elf32-i960 elf32-ia64-hpux-big elf32-ip2k elf32-iq2000 elf32-little elf32-littlearc elf32-littlearm elf32-littlearm-symbian elf32-littlearm-vxworks elf32-littlemips elf32-m32r elf32-m32rle elf32-m32r-linux elf32-m32rle-linux elf32-m68hc11 elf32-m68hc12 elf32-m68k elf32-m88k elf32-mcore-big elf32-mcore-little elf32-mn10200 elf32-mn10300 elf32-msp430 elf32-nbigmips elf32-nlittlemips elf32-ntradbigmips elf32-ntradlittlemips elf32-openrisc elf32-or32 elf32-pj elf32-pjl elf32-powerpc elf32-powerpc-vxworks elf32-powerpcle elf32-s390 elf32-sh elf32-shbig-linux elf32-shl elf32-shl-symbian elf32-sh-linux elf32-shl-nbsd elf32-sh-nbsd elf32-sh64 elf32-sh64l elf32-sh64l-nbsd elf32-sh64-nbsd elf32-sh64-linux elf32-sh64big-linux elf32-sparc elf32-tradbigmips elf32-tradlittlemips elf32-us-cris elf32-v850 elf32-vax elf32-xstormy16 elf32-xtensa-be elf32-xtensa-le elf64-alpha-freebsd elf64-alpha elf64-big elf64-bigmips elf64-hppa-linux elf64-hppa elf64-ia64-big elf64-ia64-hpux-big elf64-ia64-little elf64-little elf64-littlemips elf64-mmix elf64-powerpc elf64-powerpcle elf64-s390 elf64-sh64 elf64-sh64l elf64-sh64l-nbsd elf64-sh64-nbsd elf64-sh64-linux elf64-sh64big-linux elf64-sparc elf64-tradbigmips elf64-tradlittlemips elf64-x86-64 mmo pe-powerpc pei-powerpc pe-powerpcle pei-powerpcle a.out-cris demo64 ecoff-bigmips ecoff-biglittlemips ecoff-littlemips ecoff-littlealpha coff-go32 coff-go32-exe coff-h8300 coff-h8500 a.out-hp300hpux a.out-i386 a.out-i386-bsd coff-i386 a.out-i386-freebsd a.out-i386-lynx coff-i386-lynx msdos a.out-i386-netbsd i386os9k pe-i386 pei-i386 coff-i860 coff-Intel-big coff-Intel-little ieee coff-m68k coff-m68k-un a.out-m68k-lynx coff-m68k-lynx a.out-m68k-netbsd coff-m68k-sysv coff-m88kbcs a.out-m88k-mach3 a.out-m88k-openbsd mach-o-be mach-o-le mach-o-fat coff-maxq pe-mcore-big pe-mcore-little pei-mcore-big pei-mcore-little pe-mips pei-mips a.out-newsos3 nlm32-alpha nlm32-i386 nlm32-powerpc nlm32-sparc coff-or32-big a.out-pc532-mach a.out-ns32k-netbsd a.out-pdp11 pef pef-xlib ppcboot aixcoff64-rs6000 aixcoff-rs6000 coff-sh-small coff-sh coff-shl-small coff-shl pe-shl pei-shl coff-sparc a.out-sparc-little a.out-sparc-linux a.out-sparc-lynx coff-sparc-lynx a.out-sparc-netbsd a.out-sunos-big sym a.out-tic30 coff-tic30 coff0-beh-c54x coff0-c54x coff1-beh-c54x coff1-c54x coff2-beh-c54x coff2-c54x coff-tic80 a.out-vax-bsd a.out-vax-netbsd a.out-vax1k-netbsd versados vms-alpha vms-vax coff-w65 coff-we32k coff-z8k elf32-am33lin elf32-ms1 srec symbolsrec tekhex binary ihex netbsd-core

Wow. While binutils share no single supported working target, they can be built multiarch, or (on MirBSD) with --enable-targets=all --enable-64-bit-bfd. Doesn t help if you want to stay portable: GNUTARGET=srec is common on all Debian (sid) binutils versions (single or multiarch), but errors out on older binutils. The a.out-* targets are not common. Sure, you could hack around things, but this is tedious. If you follow things or know me a little, you might already have guessed that I wouldn t let that stand.

pax(1) to the rescue. On MirBSD, we use paxtar, which has cpio(1) and tar(1) front-ends and supports multiple formats (4 cpio and 2 tar variants) and has already been extended a lot and is lovingly called paxmirabilis (mirabilos peace in Latin) it has options to anonymise archives: set uid and gid to zero, set mtime to zero, (for ustar) only write the numeric uid and gid to the archive, (for cpio formats) serialise inodes and device information, write content of hardlinked files only once (breaks partial extraction but saves a lot of space, e.g. 2 MiB off the Grml initrd.gz). And, recently, the ability to append a trailing slash to pathnames of ustar members which are directories (GNU tar does it and I thought some Debian utilities check for it). So why not (the -M dist and fakeroot set the uid/gid to 0)

	$ find debian-binary control.tar.gz data.tar.gz   \
	> mircpio -oHar -Mdist >with-mircpio
	$ mirpax -w -M dist -f with-mirpax -x ar \
	> debian-binary control.tar.gz data.tar.gz
	$ mirtar -M dist -A -cf with-mirtar \
	> debian-binary control.tar.gz data.tar.gz
	$ GNUTARGET=a.out-i386-linux fakeroot ar rc with-aout-ar \
	> debian-binary control.tar.gz data.tar.gz
	$ md5sum with-*
	a466e2fd57cdee141fe585a43245548f  with-aout-ar
	a466e2fd57cdee141fe585a43245548f  with-mircpio
	a466e2fd57cdee141fe585a43245548f  with-mirpax
	a466e2fd57cdee141fe585a43245548f  with-mirtar

Voil . I got it, and even appending is possible. It supports the BSD format with special focus on DEB files, and deals with long filenames, but not symbol or filename tables (used by ranlib(1) or strange formats, respectively, but since we don t create *.a files to use with some native linker/binder/loader, we don t need that anyway).

On extraction (oh, and listing!) it deals with SYSV style filenames as well.

	$ mirtar tvf on-elf
	-rw-r--r--  1 tg       tg          4 Aug 18 16:05 debian-binary
	-rw-r--r--  1 tg       tg       1391 Aug 18 16:05 control.tar.gz
	-rw-r--r--  1 tg       tg      18135 Aug 18 16:05 data.tar.gz
	$ mirtar tvf with-aout
	-rw-r--r--  1 tg       tg          4 Aug 18 16:05 debian-binary
	-rw-r--r--  1 tg       tg       1391 Aug 18 16:05 control.tar.gz
	-rw-r--r--  1 tg       tg      18135 Aug 18 16:05 data.tar.gz

One of the real benefits is that you can use the front-ends interchangably for example, mirtar tzf foo.cpio.gz would work (which GNU tar can t do), and mircpio s ustar implementation, unlike GNU cpio s, is not horribly broken. Of course, there are some drawbacks: it s not GNU tar or GNU cpio, so there are absolutely zero --long-options. Some of their features are missing (but tar s -O is implemented now), so it s no replacement (but very well usable alongside it). The format called pax, committee-designed to replace ustar, isn t yet supported ironically, but that s on the TODO.

So, what do you think?

	tg@frozenfish:~/Debs/dists/sid/wtf/Pkgs/mircpio $ ll *.deb
	-rw-r--r-- 2 tg freewrt 78140 Aug 17 11:04 mircpio_20110817-0wtf2_amd64.deb
	-rw-r--r-- 3 tg freewrt 72262 Aug 17 11:00 mircpio_20110817-0wtf2_i386.deb
	-rw-r--r-- 1 tg freewrt 67446 Aug 17 18:21 mircpio_20110817-0wtf2_m68k.deb

Should I upload this to Debian proper? As for the licence: 3-clause UCB (and 2-clause BSD, which is a subset of it), so no problem. I m asking because the other package which I had been using for a long time and not uploaded, jupp, got uploaded recently (during DebConf) on user input (people wondered why it did not yet exist in Debian proper). I guess the old saying if it s not in Debian, it doesn t exist holds true in many parts of the OSS world. It s up to date wrt. standards btw, and lintian-clean save for two pedantic-class warnings (no upstream changelog file, no homepage link) which aren t fulfillable (could link this wlog entry as homepage). If you know Alioth you re familiar with the software formerly known as SourceForge, formerly known as GForge, currently known as FusionForge. My employer both uses it and contributes to it, we run an adapted (mostly themed, prototyping new functions that often end up in FusionForge itself, and backporting functions from FF to our production codebase ) version.

I ve backported the extratabs plugin to appease project managers and other non-technical people while we move our codebase to FF 5.1, and I did so on an installed version of the plugin rather than the source because the latter was tightly integrated with rather heavy packaging style changes.

	[ ]
	dh_builddeb
	# create fusionforge-plugin-extratabs binary package
	toplev=$$(pwd); cd plugins/fusionforge-plugin-extratabs; \
	p=$$(print -r -- $$(sed -n '/^Package: /s///p' C/control   head -1)); \
	v=$$(print -r -- $$(sed -n '/^Version: /s///p' C/control   head -1)); \
	a=$$(print -r -- $$(sed -n '/^Architecture: /s///p' C/control   head -1)); \
	d=$$ p _$$ v _$$ a .deb; \
	rm -f $$toplev/../$$d control.tar.gz data.tar.gz; \
	(cd control; find .   fgrep -v /.svn   sort   \
	    mircpio -oC512 -Hustar -M0x0B -Mgslash)   gzip -n9 >control.tar.gz; \
	(cd data; find .   fgrep -v /.svn   sort   \
	    mircpio -oC512 -Hustar -M0x0B -Mgslash)   gzip -n9 >data.tar.gz; \
	mirtar -M dist -Acf $$toplev/../$$d debian-binary cont*gz dat*gz; \
	rm -f control.tar.gz data.tar.gz; \
	cd $$toplev; dpkg-distaddfile $$d non-free/devel optional

The hardest part of extending debian/rules with that was to get the autobuild and dpkg-distaddfile call right. This works, even though I d call it a temporary kludge. (No need to tell me I should have used && I know. And I only shell out to mksh(1) because the inner part was already there from before, when I still used ar(1). This was slightly edited for the wlog.) In the meanwhile, apt-extracttemplates can deal with SYSV style filenames in DEB files on Debian sid, but not on Kubuntu hardy, which some people are using as Desktop OS still

Thorsten Glaser: ??? ??? or mira meets d-i

(originally published on 2011-01-26, but reposting so the people on Pl net Debian can have some fun) While helping a cow-orker setting up an encrypted hard disc (basically, putting / and swap into LVM inside cryptsetup, and /boot outside), mirabilos managed to discover an entirely new side of Kubuntu 10.10 on his voyage Warped
wo noch nie ein Debian Developer zuvor gewesen ist oder? (Only a reboot helped at that point. Earlier, the dialogue box was shown only once, but upon re-entry of the partitioning clickibunti d-i tool, neither button did anything save redrawing this interesting, informative and intuitive error message.)

16 August 2011

Craig Small: Debian turns 18

In Australia, 18 means you can do things like vote and drink. While most don t get terribly excited about the first, the second usually happens on your 18th. And so 18 years ago Ian Murdock brought together the new debian distribution. I ve been along for the ride for 14 of those 18 years and it certainly has been a fun time and what a lot of differences it has been since then. Debian is not only one of the best distributions around, but has an important place in ensuring there IS a Free Software distribution. So raise your glasses (mine s the one with the chilled filtered water, its 8am after all) and wish Happy Birthday to Debian, may you have many more. And if you d like some calorie-free cake, hope over to DUG s site where they have been busy making Debian Birthday Cakes. Just looking at that yummyness won t be too bad for you, unless it makes you hungry now.

Ritesh Raj Sarraf: File System corruption and Recovery

So this is the 2nd time I ran into a problem like this again. My FAT32 file system on the external USB HDD, all of a sudden, started reporting:
00:47:32 rrs@champaran:/tmp$ sudo dosfsck /dev/sdb1
dosfsck 3.0.9, 31 Jan 2010, FAT32, LFN
Logical sector size is zero.

I had been taking a lot of care to ensure that I don't run into situation like this. No body likes losing data. The good part is that I've been lucky enough that, even without backups (now who's gonna backup a backup disk), I have recovered all my data. All thanks to Christophe GRENIER for Testdisk.read more

Erich Schubert: Documenting fat-jar licenses

Dear Lazyweb,
What is the appropriate way to document the individual licenses of a fat jar (a jar archive that includes the program along with all its dependencies)?
I've been living in the happy world of Linux distributions, where one would just package the dependencies independently (or just specify which existing packages you use, actually, since most is already packaged by helpful other developers). But for the non-Linux people, a fat jar seems to be important so they can double-click it to run the application.
When building larger Java applications, you end up using a couple of external code such as various Apache Commons and Apache Batik. I'm currently including all their LICENSE.txt files in a directory named "legal", and I'm trying to make it as obvious as possible which license applies to which parts of the jar archive. Is there any best-practice of doing this? I don't want to reinvent the wheel; it'd also like to avoid any common legal pitfalls, obviously.
Feel free to respond either using the Disqus comment function or by email via erich () debian org

15 August 2011

Craig Small: Be careful with dspam and multi-arch

Do you use the debian dspam packages, specifically with one of the database backends? Recently upgraded the packages? Does noone email you anymore? You might be bitten by a bug, or perhaps undocumented misfeature, that bit me this morning. No emails were coming in and yet the mailserver logs said they were coming in, but a closer look at the logs showed this: Aug 15 09:08:17 elmo dspam[24896]: dlopen() failed: /usr/lib/dspam/libpgsql_drv.so: /usr/lib/dspam/libpgsql_drv.so: cannot open shared object file: No such file or directory What happened here? A quick look at the installed packages said the pgsql backend was still installed but this file was not being found. The latest version of libdspam7-drv-pgsql now has multiarch support which means /usr/lib/dspam no longer exists and the driver is found in /usr/lib/x86_64-linux-gnu/dspam. The problem happens because the configuration file specifies the directory and that has now changed. The fix is as simple, update /etc/dspam/dspam.conf with the new path.

14 August 2011

Iain Lane: Dear lazyweb: Cycle touring by train

dangly bikelets Dear lazyweb, In a couple of weeks I'm likely to be going cycle touring around the North East of England. This will involve first taking a train from Nottingham. While looking at train options just now, I was reminded of a little survey that I wanted to conduct. Taking bicycles on trains in the UK can be a hit-or-miss experience. The UK's train routes are operated by a variety of train operating companies who each have different policies on taking bicycles on their trains. Some require a reservation, some do not. Some do not even allow reservations at all. There's no official database of these policies. It is up to you to check all of the companies' websites individually. As far as I can tell it is impossible to make a cycle reservation online. If you don't make a reservation, chances are that you'll be OK, but there are no guarantees you won't find yourself stranded in Bolton when none of your connections will take you. The number of spaces reserved for cycles is extermely limited. Two seems to be the most common number that a train can officially carry (you may be able to stash it in the vestibule). Even given this, it is usual to find that the cycle spaces are taken up by prams, luggage and other crap when you actually board the train, so the effective number may actually be 0. However, it is thankfully not uncommon to find staff and other passengers who are more than willing to go out of their way to help you if they see you are trying to take a bicycle onto a train. :-) Anyway, I'm interested in how this situation is in other non-UK countries. Better or worse? A lot of countries certainly have a more cycle-friendly culture I'd like to know how this kind of thing works there so that I can write a Strongly Worded Letter about how we Should Be Doing Better. (Image by onohiroki, CC-by-nd.)

12 August 2011

Chris Lamb: Careful what you wish for...

Via #241343: 
* Package name    : apt-gentoo
  Version         : 0.0.1
* URL             : http://www-jcsu.jesus.cam.ac.uk/~mma29/apt-gentoo/
* License         : GPL version 2 or above
  Description     : enhanced package installation
 apt-gentoo enhances the Debian package installation experience to make
 it fully competitive with newly-popular source-based distributions.
 As packages are installed, apt-gentoo automatically downloads their
 build logs from the buildd network. The logs are then slowly scrolled
 past on the user's terminal to simulate building the software on the
 local machine.
 [..]
So, something like: 
#!/usr/bin/env python
#
# Insert into /etc/apt/conf.d/50apt-gentoo:
#
# DPkg  
#         Pre-Install-Pkgs   "/path/to/apt-gentoo   true";  ;
#  ;
import time
import random
import urllib
import urllib2
import urlparse
import fileinput
from lxml import etree
from debian import debfile
for filename in fileinput.input():
    pkg_data = debfile.DebFile(filename.strip()).debcontrol()
    url = "https://buildd.debian.org/status/logs.php?%s" % urllib.urlencode( 
        'pkg': pkg_data['Package'],
        'ver': pkg_data['Version'],
        'arch': pkg_data['Architecture'],
     )
    html = etree.parse(urllib2.urlopen(url), etree.HTMLParser())
    link = html.xpath("//table[@class='data logs']//td[3]/a")
    try:
        fetch_url = urlparse.urljoin(
            'https://buildd.debian.org/status/',
            link[0].get('href'),
        )
    except IndexError:
        continue
    html = etree.parse(urllib2.urlopen(fetch_url), etree.HTMLParser())
    try:
        log = html.xpath('//pre')[0].text.split('\n')
    except IndexError:
        continue
    for line in log:
        print line
        time.sleep(random.random() * 0.09)
Package/source and example installation transcript.

11 August 2011

Blars Blarson: planet-debian

Welcome Planet Debian viewers If things are working right, Planet Debian will now be getting my blog posts tagged Debian. The main reason I had not done this before is not much on my blog relates to Debian, and it tends to be much less technical than most of the blogs there. Viewers from there might want to check out my recent post on debconf11 and may check out the rest of my blog archives, where I'm trying to get rid of my house and move in to a motorhome.

Sandro Tosi: Mercurial: how to completely remove a named branch

I like so much the git feature branch workflow, that in the early days of development on Python with Mercurial I created some named branches; well, that is something you should not do.

In Mercurial, the changeset contains the branch name, so you cannot develop on a separated (named) branch and then merge on default and hope that branch goes away, because it will stay.

What do I do now? Python Mercurial repository is quite big (around 200Megs) so I wanted to avoid to re-check it out. Thanks to the help of the folks on #mercurial (on freenode IRC network) I found my solution: strip the branch!

Please note that strip is dangerous. Use it only as last resort, and mind you can lose data with it. That said, it's a very powerful tool :) My main aim was to remove completely those named branches, leave no traces, and lose the changes I made on them. Another important aspect is that I didn't merged those branches on default.

So, how to get rid of a named branch:

$ hg strip "branch($ BRANCHNAME )"

and re-iterate for all the branches you have, that's it. Now, to be completely sure they were removed and no spurious changes are in the repository, you can:

$ hg pull -u
$ hg outgoing

and if it says "no changes found" you're sure that those branches are really gone.

Julian Andres Klode: World, Space, and Licenses

Common licenses for software include the term worldwide . Now, what does worldwide mean? The problem with the term worldwide is that it is ambigous and depending on it s interpretation, violates against DFSG 6 which states: No Discrimination Against Fields of Endeavor . The reason: Space travel. If we take the term worldwide to mean everywhere on earth , the license becomes non-free, as it prohibits the use outside of this planet. Affected by this problem are the patent section of GPL-3, the Apache 2.0 license, the CC licenses, the GFDL, and probably also others. Now what should be used instead? Universal? No, that wouldn t work in case there are multiple ones (while travelling between them (if they exist) could be impossible, it would still be a restriction). The correct team would probably be omniversal meaning everywhere in the omniverse . But really, avoiding locations is probably the best way. In any case, if you have received software from me under a license that uses the term worldwide , you can treat worldwide as everywhere, and are thus free to use it outside of earth (and other planets).
Filed under: General

9 August 2011

Aurelien Jarno: Debian s390x port (aka 31 bits is not enough)

During Debconf 11, I got access to a fast s390 machine, and I have started to work on a Debian s390x port, the 64-bit version of the s390 port. One of my goal was to help the SPARC64 port, as some of the issues are the same: both are 64-bit big-endian, don t support unaligned access and behave differently between -fpic and -fPIC. Why such a port? When talking about 64-bit ports, we usually hear: 4GB is enough, handling 64-bit takes more memory . This really sounds like 640K ought to be enough for anybody . The s390 port is actually 31-bit from the address point of view (one bit is reserved for address space extension from 24 to 31 bits), so each process is limited to 2GB only. Nowadays applications which need more than 2GB are not that uncommon, especially on mainframes. Actually the 2GB limit already causes some problem in Debian: in some cases it s not possible to build haskell applications or even C applications using GCC. On the other hand, we already require a 64-bit kernel on the s390 port (only the userland is 32-bit), and applications are handling more and more 64-bit or greater values (files offset, time counters, uid, etc.). What is the status? Bootstrapping the architecture was not really easy (as for any other new architectures), due to a huge amount of dependencies and build-dependencies loops, as explained by Wookey during Debconf11. Now that this part is mostly done, an autobuilder has been started and currently more than 65% of the packages are built. The s390x port is hosted on debian-ports.org. Unfortunately it is not yet deboostrapable, though that should happen in the next few days (only a few packages are missing). The main issues are currently packages which fail to build from source due to linker, gcc-4.6 and curl changes, or due to the libjpeg and multiarch transitions, and thus are not directly related to s390x. If your package is in this case, it would be a good idea to fix it. Otherwise if it has a lot of reverse dependencies and the bug is opened for a while, just expect an NMU (as allowed by the 0-day NMU policy). Of course for a few packages s390x specific fixes are needed, some of them are already in the BTS. How you can help? The list of bugs blocking the s390x port is available through the s390x usertag, fixing these bugs (a lot of them are general FTBFS) would help a lot. Alternatively if you have access to an s390x machine you can take a look at the packages failing to build. Update: Fixed the explanation about the 32th bit, thanks to Bastian Blank for the comment.

Philipp Kern: DebConf11: Gobby documents

If you still want to grab documents that used to be on gobby.debian.net:

Matt Zimmerman: Where s your data center?

Thanks to the tremendous growth of social applications over the past five years, we have our pick of services for collecting, saving and sharing our experiences online. We each have collections of photos, contacts, messages and more, spread across multiple popular services like Twitter, Facebook, LinkedIn, as well as many less popular services which address particular needs or preferences. We re also producing a wealth of exhaust data through our browsing history, mobile sensors, transactions and other activity streams that we rarely if ever examine directly. This ecosystem is becoming so complex that it s easy to lose track of what you ve created, shared or seen. We need new tools to manage this complexity, to make the most of the wealth of information and connections available to us through various services. John Battelle calls these metaservices , and points to growth in the number of connections between the services we use. I expect that this next age of information tools will center around data rather than services. Data is a common denominator for these online experiences, a bridge across disparate services, technologies, social graphs, and life cycles. Personal data, in particular, has this property: the only thing that links together your photos on Flickr, Facebook, Picasa and Twitpic is you. So where s your data center ? I don t anticipate the emergence of a single service where you do everything. There will continue to be innovation in the form of new and specialized services which meet a particular need very well. There won t be a single service which is everything to everybody. Instead, I foresee us wanting to track, save, use and control all of our stuff across the web. That s why my new colleagues and I are working to make that possible. There s open source code available on github, a vibrant IRC channel (#lockerproject on Freeenode), and lots more I d like to write about it. But it s time to get back to work for now

8 August 2011

Joachim Breitner: Nicer URL formatting in LaTeX

To format URLs in LaTeX, one usually uses the url package (either directly, or indirectly via hyperref). By default, it typesets the URL in a mono-spaced font. Maybe this was a good idea some years ago, when URL were not ubiquitous. Today, I do not think they need neither the emphasis nor the explictit technical association that is caused by changing the font. And last but not least, mono-spaced fonts are very wide.
<embed src="http://www.joachim-breitner.de/various/latex-url-font.svg" type="image/svg+xml" width="400">
So url offers the command \urlstyle. I prefer to set it to sf, because URLs tend to look better without serif. In the picture above (also available as PDF), that is the second line. But this still has issues: The double slash at the beginning is spaced out more than it should, given that, at least to most readers, http:// is a single entitiy. In addition, the underscore is far to wide (this only happens if you use the T1 font encoding). Hence I modified the typesetting of both the slash and the underscore to achieve the appearance in the third line. The code is derived from the one given on this blog post by Nei, with slightly less kerning of the slashes and with extending instead of redefining \UrlSpecials and the underscore handling added. Here is the full code for the picture given above, to use it you need to copy the part between and including \makeatletter and \makeatother into the preamble of your document: 
\documentclass article 
\usepackage url 
\usepackage[T1] fontenc 
\usepackage[top=3pt, bottom=3pt, left=3pt, right=3pt, paperwidth=6.7cm, paperheight=1.6cm] geometry 
\begin document 
\noindent
\url http://example.com/with_underscore/ \\
\urlstyle sf 
\url http://example.com/with_underscore/ \\
\makeatletter
% Inspired by http://anti.teamidiot.de/nei/2009/09/latex_url_slash_spacingkerning/
% but slightly less kern and shorter underscore
\let\UrlSpecialsOld\UrlSpecials
\def\UrlSpecials \UrlSpecialsOld\do\/ \Url@slash \do\_ \Url@underscore %
\def\Url@slash \@ifnextchar/ \kern-.11em\mathchar47\kern-.2em %
     \kern-.0em\mathchar47\kern-.08em\penalty\UrlBigBreakPenalty 
\def\Url@underscore \nfss@text \leavevmode \kern.06em\vbox \hrule\@width.3em 
\makeatother
\url http://example.com/with_underscore/ 
\end document

Flattr this post

Jaldhar Vyas: Debian GNU/Minix Erratically Timed News vol. 1

Wondering what is going on with the port of Debian to the Minix 3 microkernal? Here's an update. dpkg Ported Well, the big news as I mentioned in my last post is that dpkg has been successfully ported. I haven't sent the patches to the maintainers just yet as they need review and clean up but I will do that ASAP. The architecture name is defined as minix_i386. (There is no k because I am using the Minix libc as well as the kernel.) I am currently targetting squeeze. A Maze of Twisty Dependencies All Different Armed with my new dpkg, I have begun building packages which has been an interesting experience. The Debian packaging systems greatest asset dependencies is also its most frustrating during initial bootstrapping. Take debhelper for instance; nowadays most packages depend on it and I personally wouldn't think of creating a .deb without it but it build-depends e.g. po4a which build-depends on...debhelper. Also some programs which are assumed to be present on linux such as ldconfig are simply not available on Minix. So currently theres a lot of dpkg --force-depends and even --force-all going on. Of course this is only a temporary problem until I get a full set of base packages built. The equivs package is coming in handy too. In lieu of working .debs, I have been using Minix native packages to fill in the gaps. They use NetBSD pkgsrc which is a new experience for me. I can't say I'm that impressed but they work and, being uptodate, are a good source of patches. Works in Progress The two big packages I need to work on next are perl 5.10.1 and gcc 4.4.5. I haven't touched the latter yet. Perl is failing in its second-stage bootstrap for reasons I haven't gotten to the bottom of just yet. I suspect it has more to do with deficiencies in my build environment than a problem in perl itself. libc Deficiencies As I mentioned last time, the situation has improved tremendously since Minix switched over to using the NetBSD libc but I am still running into problems with unimplemented APIs. Take getpwent(3) and getpwuid(3) for instance. Being able to map between user ids and names is kind of important you know? Even worse instead of being completely absent some of these exist as stubs. So they compile and link without warning but always return ENOFILE and so on. I was surprised to find that there is no standardized API for getting information on mounted filesystems. (Which e.g. df(1) needs.) Linux uses getmnt(3) similar but incompatible with System V. The BSDs use getmntinfo so probably this is what Minix will implement. GNU programs are good about portability but a lot of Free software is unfortunately Linux-centric. Hopefully the Debian FreeBSD port already has patches for these cases. There are several annoyances like this but I suppose I shouldn't complain too much. I am using an unreleased version of Minix which is under heavy development and I must say the upstream developers have been good about responding to my concerns. Looking forward: fakeroot, grub2, ext2 fakeroot is a very handy development tool that frees a developer from having to create .debs as root. Alas it won't work under Minix as it depends on LD_PRELOAD and Minix doesn't even support shared libraries. fakeroot-ng sounds a bit more promising as it uses the ptrace(2) system call which is implemented in the Minix kernel though I don't know how well. Minix is now multiboot compliant so it is possible to use grub2 instead of the Minix boot loader. Definitely something I'm going to look at soon. Minix now supports ext2 (though not ext3 or ext4) in addition to its native file system. For compatability this should become the default in the Debian port. While the kernel support is part of the standard Minix system, the userland support (e2fsprogs) is not for some reason. So I'll have to see about porting that. Can I Help the Porting Effort? Sorry not yet. Though if you are determined I'll try and help you get set up. Hopefully in a month or less I will be able to distribute a pre-alpha version so I suggest waiting for that. I am definitely going to be looking for outside help.

7 August 2011

Philipp Kern: Debian s390: channel numbers and consoles

Three things I learned about Debian s390 today:

Jan Wagner: Alone at home

Sometime it happens, that there are school holidays. As my lovely girl used to be a teacher, she has some time off at that period. So she left me with our kids yesterday to visit the grandparents.
It was two hours after leaving, when I was really surprized that I was sitting around and thinking about what to do next, as normaly the weekend are family time.
So I decided to take my bike and a paper map to get of for a trip. Since some years, I usually use my bike to escort the girls or when going to work, which is just a 5 minutes ride, so I thought that would be a great opportunitiy.
I took a backpack with raining clothes, mounted the bike helmet on it, in case I needed to go cross country, some water and my music player.
It was a great ride, beside that I realized I m not in the same condition as some years ago. Unfortunately this will likely not change in the next 12 mounth, as we have some projects in the queue we need to deal with. Hopefully we will solve that satisfactorily and have a bit more spare time afterwards.

Anyways .. 27,65km in 1h40 is not so bad, as it was hot and humid.
Ride summary:

Next.

Previous.