PID & mount namespaces A PID namespace is a view of the processes running on the system. Each PID namespace will have its own numbering of the processes contained in its namespace (including the namespace of the children). Other processes are hidden and therefore, it is not possible to issue any operation requiring a PID on them, like kill(2). Moreover, the process that was created first in a namespace will get PID 1. If it gets killed, the namespace is destroyed and all other processes contained in this namespace are killed as well. systemd, a system and service manager for Linux, targeted as a replacement for the old SysV init, uses could use this feature to launch daemons in their own PID namespace and keep track of them without relying on their cooperation. When it needs to kill a daemon, it just kills the process with PID 1 (which has another PID in the global namespace) and the kernel will do the hard work of killing all processes that were started in this namespace. You can get more details about this namespace in the manual page of clone(2) (search for CLONE_NEWPID). UPDATED: In fact, systemd does not use PID namespaces but cgroups to group processes together. There is no PID numbering here but spawned processes are all enclosed into one cgroup and therefore can be killed easily. For example, here is the result of pstree -p inside a PID namespace:

root@guybrush:/# pstree -p
bash(1) pstree(8)
         sleep(7)

The PID of sleep is 7. In the global namespace, it is 31762 instead. But it is the same processus!

root@guybrush:/# ps -eo pid,command   grep sleep
31762 sleep 1000

Linux have also supported mount namespaces since a long time. Each process can have its own mount points that will be visible only by processes in the same namespace or in children namespaces. When the namespace is destroyed (when no process left), the kernel will do the cleanup itself.

chroot A common way to test experimental software or do packaging work is to build a chroot: debootstrap is used to build a new complete system inside a directory and chroot allows to enter into this new system. Tools like schroot can become handy if you need to do that often. Unfortunately, you only get an isolated filesystem. When you start launching daemons and mounting partitions, you need to keep track of all your actions to undo them. Quitting the chroot will not kill any daemon nor unmount any partition. Extending an existing tool, like schroot to use PID and mount namespaces seems the best way to prevent such problems. I have filed bug #637870 for this purpose. However, this is not as simple as it seems since schroot allows to run a process into an existing session. When the session is a chroot, this is quite easy: just chroot into the same root filesystem and launch the process here. With other kind of namespaces, you need to use a new system call, setns(), to change the current namespace of the running process. However, some bits are still missing. First, the GNU C Library does not implement the appropriate wrapper. Second, Linux (as of 3.0) does not contain the support to change the PID namespace of a running process yet. Other tools exist, like lxc (aka Linux Containers). However, it is more targeted at providing a complete container. I have reported that lxc-execute is not able to spawn a standalone shell but got no luck. UPDATED: systemd ships with systemd-nspawn which was exactly what I needed. However, it requires systemd. Otherwise, it seems less basic than the alternative that I describe below (some sensible mount points are setup, the console is correctly configured, signals are correctly transmitted, ). If you use systemd, look at it!

jchroot Building a chroot clone using namespaces is quite easy. The only difficulty is the use of clone(2). Here is a sketch:

int main()  
  pid_t pid;
  int ret;
  long stack_size = sysconf(_SC_PAGESIZE);
  void *stack = alloca(stack_size) + stack_size;
  /* New process with its own PID/IPC/NS namespace */
  pid = clone(step2,
              stack,
              SIGCHLD   CLONE_NEWPID   CLONE_NEWIPC  
              CLONE_NEWNS   CLONE_FILES,
              NULL);
  /* Wait for the child to terminate */
  while (waitpid(pid, &ret, 0) < 0 && errno == EINTR)
    continue;
  return WIFEXITED(ret)?WEXITSTATUS(ret):EXIT_FAILURE;
 
static int step2(void *a)  
  /* Mount /proc as an example */
  mount("proc", "/proc", "proc", 0, "");
  /* Chroot */
  chroot("/srv/debian");
  chdir("/");
  /* Drop privileges */
  setgid(1000);
  setgroups(0, NULL);
  setuid(1000);
  /* Exec shell */
  execl("/bin/bash", "/bin/bash", NULL);
 

Add command flags, error handling and fstab parsing for the definition of mount points and you get jchroot. Here is an example of use:

$ cat fstab
proc     /proc  proc    defaults                  0  0
sys      /sys   sysfs   defaults                  0  0
/home    /home  none    bind,rw                   0  0
/dev/pts /dev/pts none  bind,rw                   0  0
/var/run /var/run tmpfs rw,nosuid,noexec,mode=755 0  0
/etc/resolv.conf /etc/resolv.conf none bind,ro    0  0
$ sudo jchroot -f fstab -n test /srv/debian/squeeze /bin/bash
# hostname
test
# ps auxww
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0  19480  1888 pts/1    S    15:18   0:00 /bin/bash
root         3  0.0  0.0  16536  1184 pts/1    R+   15:18   0:00 ps auxww
# /etc/init.d/postgresql start
Starting PostgreSQL 9.0 database server: main.
# exit
$ ps auxwwc   grep postgres
$

I hope that jchroot won t live for too long and that its features will be integrated into regular chroot tools. I was thinking at proposing a patch for chroot(8). Anyone knows if this would have a chance to be accepted?

HR does silly stuff like this all the time. Somebody s following some policy that was created because somebody verbally resigned nine years ago and then wanted to come back and some executive said where s their letter and HR said we don t have one and the exec said that s not good and we oughta not be doing stuff to help people leave unless they re really leaving and HR said okay we ll have a policy and the exec said that s good. And the exec s not there anymore.

Once upon a time, there was Deb and Ian. That was about exactly 18 years ago. We don t talk about the 0.939000 format any more, but they eventually settled on:

	$ ar rc pkg_1.0_all.deb debian-binary control.tar.gz data.tar.gz
	$ hexdump -C pkg_1.0_all.deb   head
	00000000  21 3c 61 72 63 68 3e 0a  64 65 62 69 61 6e 2d 62   !<arch>.debian-b 
	00000010  69 6e 61 72 79 20 20 20  31 33 31 33 36 38 33 35   inary   13136835 
	00000020  32 39 20 20 31 30 30 36  20 20 32 30 30 20 20 20   29  1006  200    
	00000030  31 30 30 36 34 34 20 20  34 20 20 20 20 20 20 20   100644  4        
	00000040  20 20 60 0a 32 2e 30 0a  63 6f 6e 74 72 6f 6c 2e      .2.0.control. 
	00000050  74 61 72 2e 67 7a 20 20  31 33 31 33 36 38 33 35   tar.gz  13136835 
	00000060  32 39 20 20 31 30 30 36  20 20 32 30 30 20 20 20   29  1006  200    
	00000070  31 30 30 36 34 34 20 20  31 33 39 31 20 20 20 20   100644  1391     
	00000080  20 20 60 0a 1f 8b 08 00  00 00 00 00 00 03 ed 59      ............Y 
	00000090  eb 6f db 36 10 f7 d7 f0  af b8 3a 5e 9b 74 b1 f5   .o.6......:^.t.. 
 

By then, systems were a.out(5), and everything was good. (Of course, if you look at the mtimes, you ll notice I faked this. But it s really equivalent to the real thing.

But oh horror! GNU binutils, not always everyone s friend, switched from using BSD style Unix Archiver libraries in ar(1) to SYSV style libraries on elf(5) systems:

	$ ar rc on-elf debian-binary control.tar.gz data.tar.gz
	$ hexdump -C on-elf   head
	00000000  21 3c 61 72 63 68 3e 0a  64 65 62 69 61 6e 2d 62   !<arch>.debian-b 
	00000010  69 6e 61 72 79 2f 20 20  31 33 31 33 36 38 33 35   inary/  13136835 
	00000020  32 39 20 20 31 30 30 36  20 20 32 30 30 20 20 20   29  1006  200    
	00000030  31 30 30 36 34 34 20 20  34 20 20 20 20 20 20 20   100644  4        
	00000040  20 20 60 0a 32 2e 30 0a  63 6f 6e 74 72 6f 6c 2e      .2.0.control. 
	00000050  74 61 72 2e 67 7a 2f 20  31 33 31 33 36 38 33 35   tar.gz/ 13136835 
	00000060  32 39 20 20 31 30 30 36  20 20 32 30 30 20 20 20   29  1006  200    
	00000070  31 30 30 36 34 34 20 20  31 33 39 31 20 20 20 20   100644  1391     
	00000080  20 20 60 0a 1f 8b 08 00  00 00 00 00 00 03 ed 59      ............Y 
	00000090  eb 6f db 36 10 f7 d7 f0  af b8 3a 5e 9b 74 b1 f5   .o.6......:^.t.. 
 

Can you spot the difference?

Of course, ELF is what you want , so there is little choice. Unix Archiver libraries are system dependent, and no format has ever been normed, but DEB files use it as format so what is one to do?

	$ GNUTARGET=a.out-i386-linux ar rc with-aout \
	> debian-binary control.tar.gz data.tar.gz
	$ md5sum pkg_1.0_all.deb with-aout on-elf
	248f78d42f8ca8f2a3560f9800b2bf01  pkg_1.0_all.deb
	248f78d42f8ca8f2a3560f9800b2bf01  with-aout
	09eca70c9b11b6b55bbadcab5c3201fb  on-elf
 

OK, and what do I do on my Debian/m68k system?

ar(1) uses bfd, and GNU binutils can not only forcibly set the target emulation but also show them:

Wow. While binutils share no single supported working target, they can be built multiarch, or (on MirBSD) with --enable-targets=all --enable-64-bit-bfd. Doesn t help if you want to stay portable: GNUTARGET=srec is common on all Debian (sid) binutils versions (single or multiarch), but errors out on older binutils. The a.out-* targets are not common. Sure, you could hack around things, but this is tedious. If you follow things or know me a little, you might already have guessed that I wouldn t let that stand.

pax(1) to the rescue. On MirBSD, we use paxtar, which has cpio(1) and tar(1) front-ends and supports multiple formats (4 cpio and 2 tar variants) and has already been extended a lot and is lovingly called paxmirabilis (mirabilos peace in Latin) it has options to anonymise archives: set uid and gid to zero, set mtime to zero, (for ustar) only write the numeric uid and gid to the archive, (for cpio formats) serialise inodes and device information, write content of hardlinked files only once (breaks partial extraction but saves a lot of space, e.g. 2 MiB off the Grml initrd.gz). And, recently, the ability to append a trailing slash to pathnames of ustar members which are directories (GNU tar does it and I thought some Debian utilities check for it). So why not (the -M dist and fakeroot set the uid/gid to 0)

	$ find debian-binary control.tar.gz data.tar.gz   \
	> mircpio -oHar -Mdist >with-mircpio
	$ mirpax -w -M dist -f with-mirpax -x ar \
	> debian-binary control.tar.gz data.tar.gz
	$ mirtar -M dist -A -cf with-mirtar \
	> debian-binary control.tar.gz data.tar.gz
	$ GNUTARGET=a.out-i386-linux fakeroot ar rc with-aout-ar \
	> debian-binary control.tar.gz data.tar.gz
	$ md5sum with-*
	a466e2fd57cdee141fe585a43245548f  with-aout-ar
	a466e2fd57cdee141fe585a43245548f  with-mircpio
	a466e2fd57cdee141fe585a43245548f  with-mirpax
	a466e2fd57cdee141fe585a43245548f  with-mirtar
 

Voil . I got it, and even appending is possible. It supports the BSD format with special focus on DEB files, and deals with long filenames, but not symbol or filename tables (used by ranlib(1) or strange formats, respectively, but since we don t create *.a files to use with some native linker/binder/loader, we don t need that anyway).

On extraction (oh, and listing!) it deals with SYSV style filenames as well.

	$ mirtar tvf on-elf
	-rw-r--r--  1 tg       tg          4 Aug 18 16:05 debian-binary
	-rw-r--r--  1 tg       tg       1391 Aug 18 16:05 control.tar.gz
	-rw-r--r--  1 tg       tg      18135 Aug 18 16:05 data.tar.gz
	$ mirtar tvf with-aout
	-rw-r--r--  1 tg       tg          4 Aug 18 16:05 debian-binary
	-rw-r--r--  1 tg       tg       1391 Aug 18 16:05 control.tar.gz
	-rw-r--r--  1 tg       tg      18135 Aug 18 16:05 data.tar.gz
 

One of the real benefits is that you can use the front-ends interchangably for example, mirtar tzf foo.cpio.gz would work (which GNU tar can t do), and mircpio s ustar implementation, unlike GNU cpio s, is not horribly broken. Of course, there are some drawbacks: it s not GNU tar or GNU cpio, so there are absolutely zero --long-options. Some of their features are missing (but tar s -O is implemented now), so it s no replacement (but very well usable alongside it). The format called pax, committee-designed to replace ustar, isn t yet supported ironically, but that s on the TODO.

So, what do you think?

	tg@frozenfish:~/Debs/dists/sid/wtf/Pkgs/mircpio $ ll *.deb
	-rw-r--r-- 2 tg freewrt 78140 Aug 17 11:04 mircpio_20110817-0wtf2_amd64.deb
	-rw-r--r-- 3 tg freewrt 72262 Aug 17 11:00 mircpio_20110817-0wtf2_i386.deb
	-rw-r--r-- 1 tg freewrt 67446 Aug 17 18:21 mircpio_20110817-0wtf2_m68k.deb
 

Should I upload this to Debian proper? As for the licence: 3-clause UCB (and 2-clause BSD, which is a subset of it), so no problem. I m asking because the other package which I had been using for a long time and not uploaded, jupp, got uploaded recently (during DebConf) on user input (people wondered why it did not yet exist in Debian proper). I guess the old saying if it s not in Debian, it doesn t exist holds true in many parts of the OSS world. It s up to date wrt. standards btw, and lintian-clean save for two pedantic-class warnings (no upstream changelog file, no homepage link) which aren t fulfillable (could link this wlog entry as homepage). If you know Alioth you re familiar with the software formerly known as SourceForge, formerly known as GForge, currently known as FusionForge. My employer both uses it and contributes to it, we run an adapted (mostly themed, prototyping new functions that often end up in FusionForge itself, and backporting functions from FF to our production codebase ) version.

I ve backported the extratabs plugin to appease project managers and other non-technical people while we move our codebase to FF 5.1, and I did so on an installed version of the plugin rather than the source because the latter was tightly integrated with rather heavy packaging style changes.

	[ ]
	dh_builddeb
	# create fusionforge-plugin-extratabs binary package
	toplev=$$(pwd); cd plugins/fusionforge-plugin-extratabs; \
	p=$$(print -r -- $$(sed -n '/^Package: /s///p' C/control   head -1)); \
	v=$$(print -r -- $$(sed -n '/^Version: /s///p' C/control   head -1)); \
	a=$$(print -r -- $$(sed -n '/^Architecture: /s///p' C/control   head -1)); \
	d=$$ p _$$ v _$$ a .deb; \
	rm -f $$toplev/../$$d control.tar.gz data.tar.gz; \
	(cd control; find .   fgrep -v /.svn   sort   \
	    mircpio -oC512 -Hustar -M0x0B -Mgslash)   gzip -n9 >control.tar.gz; \
	(cd data; find .   fgrep -v /.svn   sort   \
	    mircpio -oC512 -Hustar -M0x0B -Mgslash)   gzip -n9 >data.tar.gz; \
	mirtar -M dist -Acf $$toplev/../$$d debian-binary cont*gz dat*gz; \
	rm -f control.tar.gz data.tar.gz; \
	cd $$toplev; dpkg-distaddfile $$d non-free/devel optional
 

The hardest part of extending debian/rules with that was to get the autobuild and dpkg-distaddfile call right. This works, even though I d call it a temporary kludge. (No need to tell me I should have used && I know. And I only shell out to mksh(1) because the inner part was already there from before, when I still used ar(1). This was slightly edited for the wlog.) In the meanwhile, apt-extracttemplates can deal with SYSV style filenames in DEB files on Debian sid, but not on Kubuntu hardy, which some people are using as Desktop OS still

(originally published on 2011-01-26, but reposting so the people on Pl net Debian can have some fun) While helping a cow-orker setting up an encrypted hard disc (basically, putting / and swap into LVM inside cryptsetup, and /boot outside), mirabilos managed to discover an entirely new side of Kubuntu 10.10 on his voyage
wo noch nie ein Debian Developer zuvor gewesen ist oder? (Only a reboot helped at that point. Earlier, the dialogue box was shown only once, but upon re-entry of the partitioning clickibunti d-i tool, neither button did anything save redrawing this interesting, informative and intuitive error message.)

No comment Liked this article? Click here. My blog is Flattr-enabled.

• Amazon wants to have offer search (and browsing) of an empty notebook.
• Amazon wants to have empty notebooks as e-books on the Kindle.

I m writing to announce three big changes for the project. First, OpenHatch is changing its organizational structure to reflect our not-for-profit goals. Second, we ll emphasize our new work beyond the website, building and promoting outreach events that bring new people into the free software community. Finally, I am taking a year to do that full-time as the project lead of OpenHatch in Somerville, MA.

• Drive from England to Bosnia, and back. Plus two days of air travel, for a week of travel all told. The trip with the UK convoy back from Bosnia was enjoyable, Steve found a great route thru the Alps, and I much enjoyed finally seeing them. Then we stopped at a golf course in Luxemburg, where my hotel room was a suite ... swanky. We bogged down in Belgium, missed our ferry, which provided a chance to play some Eurogames in Europe. Then I visited family in London. I hope to eventually have some pictures from that trip. If those who had cameras make them available.
• In between the European tour, there was DebConf. As always, it was excellent. I did not come out with the large todo list of exciting things like happened last year. I did continue nibbling through that list. Had some good conversations about haskell, met Intrigeri, who wrote the ikiwiki po plugin. Had some meetings on things I feel I've sorta moved on from to some extent but still have to be available for. Didn't manage significant technical work, but this was not unexpected. The day trip was fun, enjoyed seeing the waterfalls and little mills, and swimming the cold, cold river. The last few days I was out of energy. I did not give any presentations, and only realized during the lightning talks that I should have given one about git-annex.
• I had expected to have most of a week at home after getting back from Europe, and technically did. But it was too annoying and unusual to count. Wildlife ate two trees of pears while I was away. My cat was stressed. I was stressed. It was insanely humid, and the house had been closed up for two weeks, and I had to fight mold and damp.
• So the added trip to the beach that put this month over the top to beyond insane amounts of travel, turned out to be sorta a good thing. Camping in the dunes, kids, good books, sea turtle eggs fenced off a hundred feet away on the beach waiting to hatch. Lots of kite flying, and somehow no sunburn. And no rain until a dawn rainbow followed by lots of wet just as we were breaking camp. Full details in this Ocracode. Nobody but me understands or cares, but that's just fine. :) OBX1.1 P6 L6 SC5d+++b--c- U0 T3 f++-b2 R1dw Bn-b++m++ F+u+ SC++s++g0 H+++f2i4Vs---m0 E+++r+++ T6f++-b0 R1w Bn-b++m++ F++u++ SC++s++g1 H+++f2i5 V+++ E+++r++

1. It starts with a boot manager which is similar to the temporary boot device selection menu of most recent BIOSes.
2. When you have selected the device to boot from, it looks into its EFI System partition, identified by its type GUID, as FAT file system, and executes a boot loader from a file at a specific path: /efi/boot/bootx64.efi (on amd64 platforms) by default, specially for removable devices.
3. Once the boot loader is loaded, the boot process continues as usual and is no longer specific to EFI (although the ABI used by the boot loader is specific, of course).

• It has to be made of free software, of course
• It must keep my data secure, while allowing me to share it when and how I want to
• It needs to handle a range of different data types natively, and be extensible to new types, from photos to real-time sensor data
• It should be able to collect my data from many different places where it is being created and stored
• It should have a rich API, so that I can create applications which access my data
• If I want to, I should be able to host it myself, on my own hardware, without compromising my ability to access and share it