Search Results: "spa"

13 December 2008

Runa Sandvik: So You ve Been Buried Alive

Now, check your pockets and take stock of what resources are available. If you have a cell phone, see if you can get reception. If you get through to a friend, dispense with pleasantries quickly and state the situation calmly and clearly. The newer cell phones have GPS function, which will aid greatly in directing him or her to the right place. If you re worried about using valuable phone minutes, try texting, but keep it to essentials. HLY SHT BD ALV. GT HLP, BRG SHVL, OXY, BR (NT PBR)
Now these are the kinds of tips we need more of.

Mike Hommey: Shared subtrees and per-process namespaces

Now we have seen what per-process namespaces and shared subtrees are and how to operate them, we can try to combine these two features. We ll be using our newns tool from this earlier post to create new namespaces. And for practical reasons, let s say you have a terminal with a 1$ prompt and a second terminal with a 2$ prompt (that will allow me to skip go to the second terminal phrases). In the kernel, per-process namespaces are a bit like bind mounts, such that shared subtrees work with namespaces like they do with bind mounts. As with standard bind mounts, the default shared subtree mode is private, which means mounts done on either namespaces will be private to the namespace. Only mount points active at the time of the new namespace creation will be in both namespaces:
1$ ./newns
1$ mount /dev/sda1 /mnt
1$ ls /mnt
config-2.6.26-1-amd64 grub initrd.img-2.6.26-1-amd64 System.map-2.6.26-1-amd64 vmlinuz-2.6.26-1-amd64
2$ ls /mnt
2$ mount /dev/sda1 /cdrom
2$ ls /cdrom
config-2.6.26-1-amd64 grub initrd.img-2.6.26-1-amd64 System.map-2.6.26-1-amd64 vmlinuz-2.6.26-1-amd64
1$ ls /cdrom
1$ umount /mnt
1$ exit
# Exit from newns
2$ umount /cdrom
shared mode allows both namespaces to share subsequent mounts. As with bind mounts, and for obvious reasons, you need to change the subtree mode before creating the new namespace:
1$ mount --make-rshared /
1$ ./newns
1$ mount --bind /usr /mnt
1$ ls /mnt
bin games include lib lib32 lib64 local sbin share src X11R6
2$ ls /mnt
bin games include lib lib32 lib64 local sbin share src X11R6
Like with shared bind mounts, the new mount point can be unmounted from either namespace:
2$ umount /mnt
1$ ls /mnt
1$ exit
# Exit from newns
A slave subtree will have mount points under its master (shared) subtrees propagated, while propagation won t happen in the other direction. Again, very much like bind mounts:
1$ mount --make-rshared / # For completeness, we already did that earlier
1$ ./newns
1$ mount --make-rslave /
1$ mount --bind /usr /mnt
1$ ls /mnt
bin games include lib lib32 lib64 local sbin share src X11R6
2$ ls /mnt
2$ mount --bind /usr /cdrom
2$ ls /cdrom
bin games include lib lib32 lib64 local sbin share src X11R6
1$ ls /cdrom
bin games include lib lib32 lib64 local sbin share src X11R6
1$ exit
# Exit from newns
2$ umount /cdrom
Contrary to shared and slave, unbindable doesn t add value when used in two different namespaces. This is only something that will have impact on the current namespace:
1$ mount --make-rshared / # For completeness, we already did that earlier
1$ ./newns
2$ mount --make-runbindable /
2$ mount --bind /usr /mnt
mount: wrong fs type, bad option, bad superblock on /usr,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg tail or so
1$ mount --bind /usr /mnt
bin games include lib lib32 lib64 local sbin share src X11R6
1$ exit
# Exit from newns
1$ mount --make-rprivate # Set back to default
Now we ve seen what can be done with namespaces and shared subtrees, let s see what nice features can be implemented with both. As Russell revealed, pam-namespace is able to polyinstanciate some directories following rules set in /etc/security/namespace.conf (see namespace.conf(5)). The sad thing is that it apparently can t just create a new namespace without polyinstanciating a directory, which could be useful if you only want separate namespaces, but no polyinstanciated directory. Russell s recipe goes as follows: What this means is that in the newly created namespace, a user reading in /tmp will actually be reading in /tmp-inst/$USER without the user knowing. Also, if root mounts something on the parent namespace, it will be propagated (/ being shared) to the user namespace. This means that mounting USB storage, for example, will be propagated to the user namespace. This also means that something mounted from the user namespace will also be propagated to the parent namespace. In both cases, this only applies to mounts occuring outside of /tmp, for which mounts don t get propagated. Note that without setting /tmp as private, when PAM would be mounting /tmp/inst/$USER, it would propagate as well, setting /tmp to /tmp/inst/$USER for everyone. So setting /tmp as private is mandatory. There is, however, a flaw in Russell s recipe: If any of the user mounts something under a submount of /, under /var for example, if /var is mounted, it won t be mounted to a user that already had a session opened. For that to be possible, you have to use --make-rshared instead of --make-shared in Russell s recipe. Some nice setup that can be done with all these, is the following: Add the following to /etc/security/namespace.conf:
/tmp tmpfs tmpfs root
Add the following to /etc/pam.d/common-session:
session required pam_namespace.so
Until here, it is pretty much the same as Russell s, except /tmp is a tmpfs instead of a real directory in /tmp-inst/, which can have some advantages. It doesn t seem to be possible to give pam-namespace a size for the tmpfs, though. Add the following to /etc/rc.local:
mount --make-rshared /
mount --bind /tmp /tmp
mount --make-private /tmp
Here again, this is the same as Russell s except for the correction for --make-rshared as seen above. If /tmp is already a mount point (on my systems, it is a tmpfs), you can remove mount --bind /tmp /tmp from above. Add the following to /etc/security/namespace.init:
mount --make-rslave /
Now, this is where it gets interesting: we re setting the whole tree as slave in the user namespace which means that if a user mounts a file system anywhere, it will only be seen in his session. Which means the user can more safely mount encrypted volumes: they won t be available to other users (root can still go wandering in /dev/kmem, though). And you don t even need SE linux for that. The caveat is that if you open a root session with su, from the user session, you are still inside the user boundaries, and don t have access to the virtual filesystem as it is for init. And if you mount something as root then, while it will appear in the user namespace, it won t appear neither in other users namespace nor in init s, which can be a good thing in some cases, but a burden in others. It means you may need to setup a special user that won t get a new namespace in /etc/security/namespace.conf. An alternative model would be to only set the user s home as slave, which means anything mounted by the user in his home directory would stay private in his namespace, while anything mounted outside would be shared with other namespaces. For that, you may want to replace the lines we added to /etc/security/namespace.init with the following:
HOME=$(getent passwd $3 cut -d: -f6)
mount --bind "$HOME" "$HOME"
mount --make-rslave "$HOME"
Either way, the remaining problem is that a root session opened with su from the user session won t get access to the original /tmp. As we saw, there are various use cases for namespaces and shared subtrees. I ll follow-up again on these features soon, as I ll be using them in yet another way to achieve a very different purpose.

Mike Hommey: Shared subtrees

As reply to my previous post about per-process namespaces, Russell wrote about pam-namespace and shared subtrees. As he reports, pam-namespace (that I discovered by the occasion) can do what I was suggesting would be nice for pam-tmpdir. Anyways, I was actually already planning to write about shared subtrees and how they can be useful with namespaces. In this first post, I will introduce shared subtrees, while in a follow-up post I will introduce how they can usefully be used with namespaces. As a preliminary note, you should know that while etch’s kernel supports shared subtrees, etch’s mount doesn’t support the necessary options. Lenny’s mount is unfortunately uninstallable on etch due to its dependency on a newer libc. But you can find a smount utility’s source code in Documentation/filesystems/sharedsubtree.txt in the kernel source, where other explanations about the feature are available. So, you can either backport lenny’s util-linux to etch, or build smount to replicate what I will be doing below. If you are using smount, just replace
mount --make-type path
with
smount path type
As seen in my previous post, bind-mounting allows to attach a file hierarchy at some other place in the virtual file system.
mount --bind / /mnt
will make all the contents of / (/bin, /etc, …) available through /mnt (/mnt/bin, /mnt/etc, …). On the other hand, sub-mounts will be ignored in such a case. For instance, /usr is usually a different mount point from /. It means /mnt/usr will be empty (provided it is empty in the underlying physical filesystem), instead of containing the same as /usr:
$ mount --bind / /mnt
$ ls /mnt/usr
$ ls /usr
bin games include lib lib32 lib64 local sbin share src X11R6
$ umount /mnt
If you also want /usr to be bind-mounted, you must use --rbind instead of --bind:
$ mount --rbind / /mnt
$ ls /mnt/usr
bin games include lib lib32 lib64 local sbin share src X11R6
$ ls /usr
bin games include lib lib32 lib64 local sbin share src X11R6
Obviously, sub-sub-mounts will also be propagated. Since recursively bind-mounting will create a bunch of mount points, unmounting can become a hassle. You can use the following command to unmount everything:
$ awk '$2 ~ /^\/mnt/ print $2 ' /proc/mounts sort -r xargs umount
Now, this is where shared subtrees come in. After the bind mount has been done, if you mount something on either side of the bind mount, the new mount is not propagated. This is called private subtrees, and is the default. But before doing anything else, let’s setup our testing environment:
$ mkdir -p /a/a /b
$ touch /a/b /a/c
$ ls /a
a b c
$ ls /b
After, bind-mounting /a onto /b, let’s see what happens when mounting something under /a, then under /b:
$ mount --bind /a /b
$ ls /b
a b c
$ mount --bind /usr /a/a
$ ls /a/a
bin games include lib lib32 lib64 local sbin share src X11R6
$ ls /b/a
$ umount /a/a
$ mount --bind /usr /b/a
$ ls /a/a
$ ls /b/a
bin games include lib lib32 lib64 local sbin share src X11R6
$ umount /b/a
As I said earlier, these new mounts are not propagated. Note that I used bind-mounts as sub mounts, but it would work all the same with other kind of mounts (device, fuse, etc.). There are 2 other modes that allow to have some propagation: shared and slave. shared allows mounts on both ends to be shared. Note you need to set the mode before bind-mounting:
$ umount /b
$ mount --bind /a /a
# This is needed because /a is not initially a mount point and you can only apply subtree modes to mount points.
$ mount --make-shared /a
$ mount --bind /a /b
$ mount /dev/sda1 /a/a
$ ls /a/a
config-2.6.26-1-amd64 grub initrd.img-2.6.26-1-amd64 System.map-2.6.26-1-amd64 vmlinuz-2.6.26-1-amd64
$ ls /b/a
config-2.6.26-1-amd64 grub initrd.img-2.6.26-1-amd64 System.map-2.6.26-1-amd64 vmlinuz-2.6.26-1-amd64
$ umount /a/a
$ mount /dev/sda1 /b/a
$ ls /a/a
config-2.6.26-1-amd64 grub initrd.img-2.6.26-1-amd64 System.map-2.6.26-1-amd64 vmlinuz-2.6.26-1-amd64
$ ls /b/a
config-2.6.26-1-amd64 grub initrd.img-2.6.26-1-amd64 System.map-2.6.26-1-amd64 vmlinuz-2.6.26-1-amd64
$ umount /b/a
You can even mount on one end and unmount from the other:
$ mount /dev/sda1 /a/a
$ umount /b/a
$ ls /a/a
slave allows mounts on the “master” end (/a in our case) to propagate to the “slave” end (/b), but not the other way around. The “master” end need to be shared :
$ umount /b
$ mount --make-shared /a
# Only for completeness, we already set /a as shared earlier.
$ mount --bind /a /b
$ mount --make-slave /b
$ mount /dev/sda1 /a/a
$ ls /a/a
config-2.6.26-1-amd64 grub initrd.img-2.6.26-1-amd64 System.map-2.6.26-1-amd64 vmlinuz-2.6.26-1-amd64
$ ls /b/a
config-2.6.26-1-amd64 grub initrd.img-2.6.26-1-amd64 System.map-2.6.26-1-amd64 vmlinuz-2.6.26-1-amd64
$ umount /a/a
$ mount /dev/sda1 /b/a
$ ls /a/a
$ ls /b/a
config-2.6.26-1-amd64 grub initrd.img-2.6.26-1-amd64 System.map-2.6.26-1-amd64 vmlinuz-2.6.26-1-amd64
$ umount /b/a
There is a third mode, unbindable, that does something different:
$ umount /b
$ mount --make-unbindable /a
$ mount --bind /a /b
mount: wrong fs type, bad option, bad superblock on /a,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg tail or so
$ mount --bind /a/a /b
mount: wrong fs type, bad option, bad superblock on /a,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg tail or so
As you can see, it disallows bind mounting of /a and its subdirectories to some other place. Finally, to put back the default mode, you can use:
$ mount --make-private /a
Similarly to --bind, there is also a recursive version of each: rshared, rslave, runbindable and rprivate, to apply these to sub-mounts.

12 December 2008

Russell Coker: Per-process Namespaces - pam-namespace

Mike writes about his work in using namespaces on Linux [1]. In 2006 I presented a paper titled Polyinstantiation of directories in an SE Linux system about this at the SAGE-AU conference [2]. Newer versions of the code in question has been included in Debian/Lenny. So if you want to use namespaces for a login session on a Lenny system you can do the following:
mkdir /tmp-inst
chmod 0 /tmp-inst
echo /tmp /tmp-inst/ user root >> /etc/security/namespace.conf
echo session required pam_namespace.so >> /etc/pam.d/common-session Then every user will have their own unique /tmp and be unable to mess with other users. If you want to use the shared-subtrees facility to have mount commands which don t affect /tmp be propagated to other sessions then you need to have the following commands run at boot (maybe from /etc/rc.local):
mount make-shared /
mount bind /tmp /tmp
mount make-private /tmp The functionality in pam_namespace.so to use the SE Linux security context to instantiate the directory seems broken in Lenny. I ll write a patch for this shortly. While my paper is not particularly useful as documentation of pam_namespace.so (things changed after I wrote it), it does cover the threats that you face in terms of hostile use of /tmp and how namespaces may be used to solve them.

Raphael Geissert: Re: Per-process namespaces


Mike Hommey blogged today about Per-process namespaces.
That reminded me to blog about switchsh, a very useful program (now in sid) to help workaround those programs that believe that /bin/sh is always /bin/bash, in other words: it bind-mounts bash as /bin/sh.

Package description follows:
Small program to bind-mount bash as /bin/sh for its child processes.

It can be used when bash is not the default shell interpreter but a given
program makes use of bashisms (features not required by Policy for sh) and one
wants to run it without changing (or can not change) the default sh.


Let's take a look at how switchsh can help us:

$ cat evil_script.sh
#!/bin/sh
echo -ne "Hello!\nPlease enter your name: "
read
echo "So, you are $REPLY, right?"


$ readlink /bin/sh
dash
$ ./evil_script.sh
-ne Hello!
Please enter your name:
read: 4: arg count
So, you are , right?
$ switchsh ./evil_script.sh
Hello!
Please enter your name: Raphael
So, you are Raphael, right?

But lets consider the following case:
$ cat evil_script.pl
#!/usr/bin/perl -W
my $res =  git gc &>/dev/null ;
if ($? eq 0)  
    print "Done\n";
  else  
    print STDERR "bang! something went wrong\n";
 


$ ./evil_script.pl
fatal: Not a git repository
Done
$ switchsh ./evil_script.pl
bang! something went wrong

But what happens when bashisms are used on a binary object? Say in OpenOffice.org, in X.org, Wine, or any other large project that take not less than a couple of hours to build in a "normal" machine? you really want to have switchsh in those cases (and file a bug as soon as you notice that running a program with and without switchsh makes it behave differently).

Mike Hommey: Per-process namespaces

Linux has had this neat feature for quite some time now: since 2.4.19 according to the docs. Yet, it is neither very known nor very used. I couldn’t even find a program that would create a new namespace for its subprocesses, similar to what chroot does with the root of the file hierarchy. This neat feature allows each process to have a different set of mount points. While you most of the time want processes to share their mount points, there are some cases where you may want to have some processes have a different set of mount points. Combined with bind mounts, it can allow some useful setups. In case you are not familiar with bind mounts, they allow to “attach” a part of the file hierarchy to some place else. For example:
$ ls /mnt
$ ls /usr
bin games include lib lib32 lib64 local sbin share src X11R6
$ mount --bind /usr /mnt
$ ls /mnt
bin games include lib lib32 lib64 local sbin share src X11R6
Now, take pam-tmpdir, for example. It sets $TMPDIR and $TMP to point to a user-specific temporary directory. Sadly, it is pretty useless for applications that don’t follow the standards of using these environment variables. Without namespaces, if you’d create a temporary directory and bind-mount it to /tmp, this new /tmp would be visible to everyone, to every process. But with namespaces, you can make this new /tmp only available to subprocesses. If pam-tmpdir were to do this, it would also allow applications trying to write to /tmp without resorting to $TMP or $TMPDIR to be using the temporary space, without impacting external processes, that would still be using the original /tmp. On x86-64, you can run both 64-bits and 32-bits applications. 64-bits applications would take libraries from /usr/lib, and 32-bits applications would search libraries in /usr/lib32. But badly crafted 32-bits applications could be trying to load libraries from /usr/lib, where only 64-bits versions are available. With namespaces, the broken 32-bits application could have /usr/lib32 bind-mounted to /usr/lib without the 64-bits applications knowing. You could certainly get a similar result with the following set of commands:
$ mount --rbind / /chroot
$ mount --rbind /usr/lib32 /chroot/usr/lib
$ chroot /chroot $application
(–rbind also attaches submounts, contrary to –bind) The downside, here, is that external processes will see all this setup under /chroot. The whole setup would be invisible to external processes if namespaces were used. Another nice use of namespaces would be to mount encrypted volumes under a different namespace, so that only a limited set of processes would be allowed to read the decrypted data. The sad thing is that you need the admin capability to create a new namespace, so that would need to be done by a setuid root program. There are, as far as a few hours fiddling showed me, 2 system calls that will setup a new namespace: clone(2), and unshare(2). The second is easier to use, though only available since 2.6.16. But while etch ships 2.6.18, the glibc coming with it doesn’t implement unshare(2), so we need to use syscall(2) instead. The following code will run /bin/sh, or any command given as argument after creating a new namespace. The new process and its subprocesses will inherit the new namespace.
#include <sched.h>
#include <syscall.h>
#include <unistd.h>
int main(int argc, char *argv[])
  syscall(SYS_unshare, CLONE_NEWNS);
  if (argc > 1)
    return execvp(argv[1], &argv[1]);
  return execv("/bin/sh", NULL);

This tool, once built, is called newns in the following example.
$ mkdir /tmp/abc
$ ./newns
$ mount -n --bind /tmp/abc /tmp
$ touch /tmp/a
$ ls /tmp
a
<in another terminal>
$ ls /tmp
abc
$ ls /tmp/abc
a
When using namespaces, it is better not to have mount fill /etc/mtab, using the -n argument. /proc/mounts will contain the proper mount information about the namespace of the process reading it. /proc/PID/mounts will contain the mount information for the given process. As bind mounts also work on files, you can override some files. The following will run dash instead of bash (in subprocesses, too, obviously):
$ ./newns sh -c "mount -n --bind /bin/dash /bin/bash; /bin/bash"
Back to the idea of having encrypted volumes only available to some processes, the following should work (unverified):
$ ./newns sh -c "encfs /tmp/crypt-raw /tmp/crypt; /bin/bash"
Only the opened bash, and its subprocesses, would have access to /tmp/crypt. The newns tool used above could, to allow normal users to be able to fiddle with namespaces, be improved to be a setuid root program that would drop its privileges right after unshare(2) to take the same privileges as the calling process. As you can see, per-process namespaces have a wide range of possible uses ; it’s astonishing that it’s not more used yet, considering its age. Additionally to per-process namespaces, there are also a bunch of other (more recent) features that allow to implement vserver-like features with a vanilla kernel, such as networking netspaces (work under progress, though), PID namespaces and utsname (see uname(2)) namespaces. Actually, these features are designed to be used by vserver and openvz. I am looking forward to having unprivileged mounts implemented, so that users could fool around with bind mounts. Unprivileged namespaces would be a nice addition.

Chris Lawrence: Corollary of the day

King Politics thinks the GOP anti-bailout faction has the politics of the day wrong:
The time for pragmatism is now, but Senate Republicans don t recognize that. Senate Republicans have yet to realize that the GOP lost so many seats in 2006 and 2008 because the American public has a greater desire for pragmatism than ideology.
I think that s true to a point but I suspect it has more to do with post-Katrina George W. Bush than any coherent definition of ideology. As much as the Democrats would like to pretend otherwise, ideological is not a term I d readily apply to the bumbling nature of Bush s second term. Nor am I really convinced that the average voter is doing much more than engaging in post-hoc rationalized-as-something-else economic voting, which doubtless makes me no fun at parties when I play a public opinion scholar. ( Yes, all this crap matters at the margins, and occasionally elections are won at the margins, but most of the time it doesn t matter. ) But I digress. There is a broader lesson, though, in that to the extent the Democrats believe that their recent success is due to their ideology it is at their long-term peril, particularly if they bypass pragmatism in favor of catering to the cobbled-together collection of rent-seekers that passes for the Democratic coalition. To the extent bailing out Detroit is seen as a Democatic handout to its paymasters particularly with the emerging frame of the greedy unions are standing in the way trumping any sort of concept that any deal that tells the UAW to can their contract is essentially an impairment of the obligation of contract (which, believe it or not, is unconstitutional) the auto bailout won t go over well in the 48 or so states that don t host significant Big Three production. On a related note, Steven Taylor notes the defining-down of the filibuster by the media to mean failure to win a cloture vote. While I accept the point graciously, I think this may be more a failure of us as a profession than the media per se; modern congressional procedures (not just filibusters and holds, but also esoterica such as the Rules Committee and UCAs), even superficially treated, aren t a strong point of most American government textbooks, and more often than not that s the only real government orientation budding journalists will get. I make a point of assigning Barbara Sinclair s Unorthodox Lawmaking in my Congress classes, but I doubt the average journalist gets that in-depth in their undergrad days. So here at least I think the blame falls somewhat closer to home than we might want to admit.

Chris Lawrence: Corollary of the day

King Politics thinks the GOP anti-bailout faction has the politics of the day wrong:
The time for pragmatism is now, but Senate Republicans don t recognize that. Senate Republicans have yet to realize that the GOP lost so many seats in 2006 and 2008 because the American public has a greater desire for pragmatism than ideology.
I think that s true to a point but I suspect it has more to do with post-Katrina George W. Bush than any coherent definition of ideology. As much as the Democrats would like to pretend otherwise, ideological is not a term I d readily apply to the bumbling nature of Bush s second term. Nor am I really convinced that the average voter is doing much more than engaging in post-hoc rationalized-as-something-else economic voting, which doubtless makes me no fun at parties when I play a public opinion scholar. ( Yes, all this crap matters at the margins, and occasionally elections are won at the margins, but most of the time it doesn t matter. ) But I digress. There is a broader lesson, though, in that to the extent the Democrats believe that their recent success is due to their ideology it is at their long-term peril, particularly if they bypass pragmatism in favor of catering to the cobbled-together collection of rent-seekers that passes for the Democratic coalition. To the extent bailing out Detroit is seen as a Democatic handout to its paymasters particularly with the emerging frame of the greedy unions are standing in the way trumping any sort of concept that any deal that tells the UAW to can their contract is essentially an impairment of the obligation of contract (which, believe it or not, is unconstitutional) the auto bailout won t go over well in the 48 or so states that don t host significant Big Three production. On a related note, Steven Taylor notes the defining-down of the filibuster by the media to mean failure to win a cloture vote. While I accept the point graciously, I think this may be more a failure of us as a profession than the media per se; modern congressional procedures (not just filibusters and holds, but also esoterica such as the Rules Committee and UCAs), even superficially treated, aren t a strong point of most American government textbooks, and more often than not that s the only real government orientation budding journalists will get. I make a point of assigning Barbara Sinclair s Unorthodox Lawmaking in my Congress classes, but I doubt the average journalist gets that in-depth in their undergrad days. So here at least I think the blame falls somewhat closer to home than we might want to admit.

MJ Ray: Global 300 and chances to invest in a fuel co-op or meet Cooperatives-SW

The new Global 300 list has been released this week:-
the 300 top co-operatives are responsible for an aggregate turnover of 1.1 trillion USD which makes it the size of the 10th economy of the world nearly the size of the Spanish economy. The list also shows that co-operatives are growing in terms of turnover 14%
Also, the Good Fuel Co-operative (a consortium established early in 2008 by five UK biodiesel cooperatives: Goldenfuels, Sundance Renewables, Magpie Recycling, Bolton Alternative Fuels Co-operative and Biofuels.org.uk) has launched a share issue to raise 420,000 to fuel the growth of the biodiesel market. Finally, Cooperatives-SW board will meet in Exeter in January and the AGM will be in Plymouth in April - so if you d like me to let you know the dates when they re announced (for meeting up or otherwise), please leave a comment on this post and tick the Notify me of followup comments box. Thanks!

11 December 2008

Russell Coker: Please Turn off Your Spam Protection

Hi, I’d like to send an email from a small domain that you’ve never heard of or from a big ISP that’s known for being slack about spam (*), I can’t send the mail to you because of your anti-spam measures. I think that this is unfair, it’s discrimination, and you are cutting off your nose to spite your face in rejecting my mail. So please reconfigure your mail server now and accept more spam in your inbox, my message is important enough to justify the extra time you will spend manually deleting mail and the risk of accidentally deleting legitimate mail while deleting heaps of spam (I am in fact more important than you). By not accepting my mail you are being an asshole. I only receive a dozen spam messages a day and I don’t mind it, without even knowing you or having bothered to do a web search to see how well your email address is known I’m sure that you don’t receive any more spam than me and therefore you too can turn off most anti-spam measures and manually sort through the spam. You don’t really have a problem with spam, you are just paranoid, I’m sure that you installed your anti-spam measures before receiving any spam and then never bothered to check the hit rates. My sys-admin knows that one of the DNSBLs has an old entry from when his server was broken, but he won’t request that it be removed - so you can change your server because my sys-admin doesn’t want to click on a URL that you sent him. The RFC-ignorant.org service is used and run by ignorant people - I know this without even reading their web site to discover how it works. The above is a summary of a number of complaints that I have received about my anti-spam measures. I’ve para-phrased them so that they make sense, I have not actually had someone directly say “I’m more important than you so you should just accept more spam”, but the implication is quite clear. Now there are some legitimate reasons for requesting that anti-spam measures be reduced. In the distant past almost everyone had working reverse DNS entries which matched the forward entries and it was common to reject mail from systems that didn’t have valid DNS. Nowadays there are many big ISPs that delegate IP address space without permitting reverse DNS entries, and there are companies that have one department in charge of IP addresses (who don’t have a clue about reverse DNS) and another department running mail servers (who are cluefull). So the environment has changed to make reverse DNS checks a non-viable anti-spam measure. Requesting that people remove such checks is reasonable. Anti-spam measures that attack innocent third parties are bad. Sending “warnings” about viruses has made no sense for many years as all modern viruses fake the sender address, an employee of one company once admitted to me that they were sending out anti-virus “warning” messages as a way of sending unsolicited advertising to random people (I reported them as a spam source). Some time ago on a routine upgrade of ClamAV I accidentally copied in a default configuration file that made it send such warnings - I was grateful when someone informed me of my mistake so that I could fix it. Challenge-response is another technology that causes harm to others. I think it makes a lot of sense for mailing lists (every modern list server will confirm subscription requests), while it does result in sending unwanted mail to innocent third parties (every time a new virus becomes prevalent I receive a number of confirmation messages from list servers), but it’s not something that I will use on a regular email account (but I am prepared to do paid work implementing CR for other people). Requesting that manually implemented blocks be removed is quite reasonable. Occasionally I find that mail from one of my servers is blocked because a previous owner of the IP address space did bad things. In such a situation it is quite reasonable to provide an assurance that the new owner takes abuse issues seriously and to request that the block be removed. Requesting that I make any change to my system without making a minimal effort to get your broken mail server fixed is totally unreasonable. If the system administrator is not prepared to click on a URL to get their system removed from a black-list or if the user is unwilling to report the problem to the sysadmin then I will probably be unwilling to make any change to my system. The only exceptions to this rule are for clients, colleagues, and for people who use mail services that are large and unresponsive to users (IE the users don’t directly pay). I recently made a white-list entry for a large European ISP that is used by a Debian Developer for their work, as the ISP is known to be unresponsive to requests and mail related to Debian work is important to me I added a white-list entry. One thing I am planning to do is to document my anti-spam measures and then allow people the opportunity of suggesting new anti-spam measures that I haven’t tried yet if they want me to turn off any of my current protections. (*) I’m not aware of any big ISP that takes strong measures against spamming customers and customers whose computers are trojaned. I am aware of one having done it in the past, but I suspect that they may have ceased doing so after their brush with bankruptcy. I suspect that many ISPs simply rate-limit their customers connections to the outbound mail relays and hope that they don’t get enough infected customers at any time to get themselves listed as a spam source.

Alastair McKinstry: Splint - compiler warnings

I was installing SPlint - Secure Programming Lint - on our new machine at ICHEC yesterday, when:
Compiling flags.c...
flags.c(151): warning #1011: missing return statement at end of non-void function "argcode_unparse"
          
  ^
flags.c(265): warning #188: enumerated type mixed with another type
    allFlagCodes (code)
    ^
flags.c(292): warning #188: enumerated type mixed with another type
    allFlags (f)
    ^
flags.c(390): warning #188: enumerated type mixed with another type
    allFlags (f)
    ^
flags.c(459): warning #188: enumerated type mixed with another type
    allFlags (f)
    ^
...
Compiler warnings from a Lint tool. A commercial compiler (Intel) admittedly, but nevertheless.

read more

Tags , ,

10 December 2008

Biella Coleman: The Only Winning Move is Not To Play



nice-game-chess, originally uploaded by the biella.
Today I wrapped up both of my classes. Usually the end of class/semester represents and is experienced as nothing short of pure, unadulterated joy and at both ends of the stick (student and professors alike experience similar emotions :-). Like my students, I am pretty beat and more than ready for a break. But I have to admit, I experienced the end of my hacker class with a tinge of sadness. This is the third time I have taught this class and each time, it seems like it becomes harder and harder to contain the class conversation and I enjoy them through and through. The last few classes were brimming with talk, which is what should exactly happen because by the last month students are walking/talking on a foundation built over the course of many weeks. There is not much I would have changed about the syllabus (most of the readings were great-to-stellar) nor the class. But if there is one thing I am sorry about it is that we never watched War Games. The last time I taught the class, I was able to arrange various movie sessions and we watched War Games, Tron, and Sneakers (among a few others) and the students not only learned something about hacker media representations but did a little out-of-class bonding with the movies and lots of food. Since space is such an issue at NYU, I decided not to forage or hunt for a suitable room (I have since found a connection who I think can hook me up with a room). But by missing War Games, we missed an important event in the history of hackerdom. The movie undoubtedly led to many a kids asking for a modem for xmas and I am sure led to a spike in hacking and phreaking 9 months later (or however long it takes for a kid to plead for a modem, get it, learn some new tricks etc). It is also the case that while it helped usher in the image of the hacker as nefarious computer trickster, David Lightman was also a pretty cool, cute, and likable kid. Not a nerd but a geek. It perhaps represents the glimmerings of the transformation of the negative nerd into the positive geek (oh and the list keeps the list of girl geeks growing ), signaling the spread of the computer into mainstream society. So next time I teach the class, there will certainly be more movies, War Games on the top of the list so that we can learn that “the only winning move is not to play.”

David Welton: Slicehost vs Linode

Since hosting is the major expense for DedaSys LLC, and obviously a critical part of much of what I do, getting the right one is very important. Naturally, "the right one" for me may not be the right one for everyone. I am a fair sysadmin for small numbers of machines, so I don't mind doing that myself - I don't need, nor want to pay for hand holding.
Since this post has become fairly popular, I am going to link to my Linode affiliate URL if you'd like to sign up with Linode - thanks! Other than running my sites there, I have no other affiliation with Linode. And, if you're more a fan of Slicehost, here is my referral page on Slicehost.
I recently moved everything over to Slicehost on the recommendations of friends, and am so far fairly happy with the experience. It's cheaper, simpler, and more flexible than Layered Tech was. To boot, I also like the fact that Slicehost is run out of St. Louis, Missouri, rather than some really expensive tech center like San Francisco or Boston. Hosting is basically a commodity, and much like you wouldn't want to put a factory in San Francisco, you don't want your hosting their either. However, I discovered something annoying about Slicehost: they use x86_64 servers, which, per se, doesn't really matter to me - I use open source code that can run on any number of architectures. The problem is that this particular architecture uses more memory than plain old x86. Significantly more. My Layered Tech server with 1 gig of memory was hitting the swap space a bit, but the same code on a slice was swapping quite heavily, despite the fact that I'd moved PostgreSQL off to its own slice. Since I pretty much exclusively run Rails, I decided to look into Phusion's "Ruby Enterprise Edition", which is basically just some nice hacking on Ruby's garbage collection mechanism. What they've done is nice, and I may end up using it, but ultimately it's buying me space that I would also gain from simply moving back to x86. With that in mind, I decided to take another look at what appears to be emerging as one of Slicehost's main competitors, Linode, who *do* use x86 servers. Here are my results, which are admittedly not all that scientific, but what the heck - you're getting them for free, and they were pretty quick to whip up. Service, Support, Setup But before we begin with the numbers, I want to add a few words of caution. One of the big things about hosting, to me at least, is how they deal with unexpected random negative events. Connections going down, disk breakage, DoS attacks, and so on. It's really hard to get an idea of just what sort of people you're dealing with until you've "ridden the river together". I don't really know what sort of response times, uptime, and all that either Slicehost or Linode have, so there are potentially some big intangibles there that are not quite as easy to draw pretty charts for. In terms of the console/setup/management tools, I liked Slicehost's simplicity more - Linode gives you more options, but they're a bit fiddly and cluttered feeling at times. For instance, Linode lets you pick how much swap space to give your disk image, but doesn't include a bit of javascript to balance out the swap and regular partitions, so that if you type in a larger swap number, you have to do the math and subtract that from the other number. Annoying, but not a big deal. Linode lets you pick various data centers in the US (no Europe, yet). Slicehost gives you the option to do backups of your disk images, which is nice, and something that Linode lacks. Comparing plans Since I'm interested in comparing hardware and machines I actually have access to, here are the plans I am currently signed up with, and the salient numbers:
Slicehost 1024 Slicehost 512 Linode 720
Memory 1024 512 720
Bandwidth 400 200 400
Disk 40 20 24
Cost 70 38 39.95
Memory / Dollar: Bandwidth / Dollar: Disk / Dollar: Linode mostly comes out ahead, but not by that much, except for bandwidth. However, let's take a crude look at x86_64 vs x86 memory usage. First, ./script/console (a Rails console) for the same code base on two different machines, showing the virtual memory size (VSZ) and resident set size (RSS): And just to look at something else that's a bit smaller, I ran the following script:
puts File.open("/proc/self/status", "r").read
And got the following data: Being charitable with those numbers, we get x86 taking up 77% the memory that x86_64 does. So let's add that to our comparison of how much memory you're getting per dollar on Slicehost and Linode: Wow! That's a fairly significant difference. Performance That's most of what we can glean from published numbers, and running a few simple experiments. However, there's another factor that's important: performance. "Virtual Private Server" systems, or simply VPS's, got a bad name in the past because they were "overbooked" - too many virtual servers competing for the same CPU resources on one machine. Slicehost and Linode both look like they want to avoid that kind of bad reputation, and so far all the systems I have used have felt snappy and responsive. Now here's where we get really unscientific... I decided to try and do something, even if it wasn't much, to get some kind of objective measurement of what kind of CPU I was getting on the machines. This is really unscientific, because the machines do have other duties (I don't have the cash, time or inclination to set up random servers just for testing), and of course, who knows what else is sharing the computer with my systems: if you get a relatively unused computer, you can apparently pick up extra cycles for yourself, over and above your guaranteed minimum. But c'est la vie, and so I decided to do some numbers anyway. I picked a C implementation of mandelbrot from the language shootout more or less "for the hell of it" (I'm going to keep repeating "unscientific" and invite someone to do better tests than I have). I ran this code every 20 minutes for a couple days, and then averaged out the run times: As expected, the higher power Slicehost 1024 machine wins out in terms of raw speed (less time is better), but not by that much over the Linode system. Indeed, when we factor in the relative prices, Linode comes out ahead, again: So it looks like we're not paying a CPU penalty for having more memory, bandwidth, disk, etc... Notes
  • Nothing in these statistics is indicative of future directions that both Linode and Slicehost might take. Slicehost was recently acquired by Rackspace, and that could have effects on the service they provide. Maybe one or both will come under financial pressure and start "overbooking" their servers.
  • The spreadsheet I used is available via Google docs, for the curious. I did most of the work in OpenOffice, and then uploaded it though, because working with the web spreadsheet was kind of painful. Unfortunately, you can't select non-contiguous data ranges in the Google spreadsheet, so the labels are interspersed with the data. Yuck. Also, Google docs doesn't handle 'time' values nicely, which is what the performance data was originally. The spreadsheet is here: http://spreadsheets.google.com/pub?key=pVDeZF6ire2YwATBYb4mBKQ
  • The standard deviation on the processing times is higher for both slices than the linode system. Not quite sure what that means in terms of what's going on under the hood, but I thought it was interesting.

9 December 2008

Nico Golde: ssh on-the-fly port forwarding

ssh is one of the programs I didn't really every read the manpage of (only to lookup stuff which I already knew is possible but forgot how). However there are some nice features you miss if you don't. Besides ControlMaster I just stumbled over another feature.

It is possible to add port forwarding on the fly without the need to open a new terminal and ssh connection.
ssh provides some escape characters to do some stuff in your current connection.
In the default setup the basic escape character is ~ (you can specify this by using EscapeChar in your configuration).

If you follow this by # character (and thus type ~#) you get a list of all forwarded connections.
Using ~C you can open an internal ssh shell that enables you to add and remove local/remote port forwardings
  1. ssh> help
  2. Commands:
  3.       -L[bind_address:]port:host:hostport    Request local forward
  4.       -R[bind_address:]port:host:hostport    Request remote forward
  5.       -KR[bind_address:]port                 Cancel remote forward
  6.  
  7. ssh> -L 8080:localhost:8080
  8. Forwarding port.
  9. echo foo
  10. foo
Some other more or less usefull escape sequences, have a look at man 1 ssh -> ESCAPE CHARACTERS

Nico Golde: ssh on-the-fly port forwarding

ssh is one of the programs I didn't really every read the manpage of. However there are some nice features you miss if you don't. Besides ControlMaster I just stumbled over another feature.

It is possible to add port forwarding on the fly without the need to open a new terminal and ssh connection.
ssh provides some escape characters to do some stuff in your current connection.
In the default setup the basic escape character is ~ (you can specify this by using EscapeChar in your configuration).

If you follow this by # character (and thus type ~#) you get a list of all forwarded connections.
Using ~C you can open an internal ssh shell that enables you to add and remove local/remote port forwardings
  1. ssh> help
  2. Commands:
  3. -L[bind_address:]port:host:hostport Request local forward
  4. -R[bind_address:]port:host:hostport Request remote forward
  5. -KR[bind_address:]port Cancel remote forward
  6. ssh> -L 8080:localhost:8080
  7. Forwarding port.
  8. echo foo
  9. foo
Some other more or less usefull escape sequences, have a look at man 1 ssh ->> ESCAPE CHARACTERS

MJ Ray: Farewell Oliver Postgate: Chairman Bagpuss, I almost knew you

There have been glowing tributes in the news today following the death of Oliver Postgate, creator of Bagpuss, Ivor the Engine and Noggin the Nog and co-creator of the Clangers. But it’s surprised me how few of the mainstream news sites have linked to Oliver Postgate’s personal website. I knew where it was because it’s hosted by an ISP that TTLLP have worked for. So far, I’ve seen one link to it from the Guardian, buried in a sidebar alongside one of the commentators waffling about him. Are British TV and newspapers scared of Chairman Bagpuss’s “orthodox Miaoist” views? The site’s not that up-to-date (understandably), but it has given me a bit of amusement and warmth today after this sad news. It’s also serendipitious after yesterday’s wikipedia filtering caused a surge in interest in “Wind of Change” band Scorpions that Oliver Postgate’s final homepage was called “The Law of Unintended Consequences” - if you enjoyed the narrative style of Bagpuss, Ivor and the Clangers, go enjoy reading it:-
“Because not being intended, unintended consequences are not on the list of what was expected to happen, so if one does happen, nobody has to notice it or connect it with whatever was intended. Or, if they do notice it they just write it off as part of the law of unintended consequences , and of course as [God's] laws are immutable, eternal and that, they know they can t do anything about it even if they wanted to, which half the time they don t.”

Gerfried Fuchs: Better backports.org Support

Since last weekend backports.org has two more services supporting tracking of what's going on. I will list them in chronological order: Tracking of Security issues in backports packages: This was one of the many topics that was discussed at the Security Team meeting in Essen at the end of last month and Florian Weimer implemented a first version of tracking security issues within backports.org. It currently compares the backported version against the one fixed for unstable, so at the moment it still has some false-positives (e.g. libspf2 because the fix was taken from lenny-security), but this is still a big step forward and helpful to track outstanding issues here, too. Diffs between etch-backports and lenny: Something similar was available at some other places before but strangely got discontinued. In here you can see which packages in backports are older (meaning only debian revision difference), outdated (newer upstream version in lenny) newer (backported from unstable?), have a wrong version schema or are not available in lenny (propably even removed from unstable). It will hopefully help people to get their backports in sync. At least it's an indicator of how well the packages are tracked. Hope you consider them already as useful as myself, even though there is obvious space for improvements. But they are both quite helpful already in their current state so that shouldn't hinder you from using them. :)

Wouter Verhelst: The advantages of a home-grown comment system

A while ago, I was investigating the possibility to move blog backend to something else; something like, say, drupal, so that I wouldn't have to maintain it too much myself. For those who care, this blog is a very strange contraption based on subversion, blosxom, php, and a postres backend for the comments. I've now dropped that migration thought, for a very simple reason: comment spam. A while back, the spammers had found my blog, and started placing comments using a bot. They might think that these comments will eventually show up on my website, but that's wrong; I always moderate away spam. First, they posted a few comments. I changed a few details about the form, and the bots (who only went to the POST URL, never checking whether the form they requested was unchanged) failed miserably. Then, they posted loads and loads of comments, in a weird attempt to overrun me with comments. One psql run and one DELETE statement later, those comments were gone -- and one minor form change later, the bots failed miserably, again. This went on for a few months, until the stupid spammer finally understood that he was wasting loads of his time in trying to post something on a website that wasn't ever going to enable his comments, anyway. Or so I thought. You see, the thing they're really after is my pagerank. In itself, grep.be is nothing special -- but planet.grep.be is, and apparently some of that pagerank score trickles to the other grep.be subdomains. Or, at least, that's what the spammers seem to think, in case it isn't (I wouldn't know). When the comment spam on my blog had been silent for a few weeks, I found that the spammers had found my gallery installation, and had posted some several thousand comments there. One DELETE statement later they were gone, but a few hours later some of them were back. I don't know the gallery source code well enough to pull the same trick here (disable comments until manual moderation), and unless I want to do something like using a CAPTCHA on my gallery site (I don't) or disabling comments altogether (I don't, either), I'll just have to resort to removing these comments from the database. At least by doing that, I can remove them all in one go. And at least the spammers are easily recognizable. For now. Perhaps I'll have to look into changing gallery's source code after all... or, perhaps, just use something else. Sigh. Stupid spammers

Wouter Verhelst: Rails: followup

I received quite some comments on my rant about Ruby on Rails a few days ago; I guess I must have received some inbound link on a ruby-fanboy-website or some such. Most of it is just bashing me since, I'll admit, my post wasn't very constructive (and which according to my policy of only blocking spammers I've not moderated away), some of it just posits another opinion about packaging (which is valid), and one Eric Anderson gave a good and constructive set of arguments that I just have to reply to. First of all, yes, I hate RubyGems more than I hate Ruby on Rails. However, Ruby on Rails is the major success story in the Ruby community, and the main reason why people use that language; it's not unfair to say that I won't use RoR because of RubyGems. But perhaps I should've been clearer. Anyway. Eric mentions that RubyGems "solves two problems that other packaging systems are not addressing": the ability to install two versions of the same library, and the ability to install a single package on multiple operating systems. Well, first of all, the latter one simply is not true. There's CPAN.pm, which does much the same thing as RubyGems; but much of CPAN can be easily turned into a Debian (or RPM, or whatnot) package. The few cases where it can't, you're usually dealing with a broken CPAN package, anyway. There are similar things for Python (egg), and, heck, even TeX (CTAN). The fact that other scripting languages have proper and working packaging systems only outlines that there really isn't an excuse for the horrors of RubyGems. Second, the fact that you even need to be able to install multiple versions of a library really shows that there's a problem with your development ecosystem. C, Perl, and Python programmers are used to stable APIs, and prefer not to change their APIs all that much. Most packaging systems reflect that preference, and deal with incompatible API changes the way they should: as the exception, rather than the rule. The people in the Ruby ecosystem clearly have a different opinion on that, which probably results in the difference in packaging system requirements. Personally, I prefer the non-Ruby way—I'll leave the discussion about "what happens when you need to do an urgent security update of one of your libraries but the API has changed three times since you last looked at it and now the library update breaks your application" as an excercise to the reader. In closing, I'd like to reply to this one sentence that Eric put in his comment:
And of course there is no reason software cannot start as a gem (for people close to the ruby community) but as it's popularity increases packages can be created that are more tightly integrated into the native package manager.
Well, in an ideal world, he'd be right: there would be no reason. Unfortunately, in the RubyGems world—which is far, far away from an ideal world—the fact is that packaging RubyGems in a sane way so that they'll integrate with the rest of a Debian system is simply impossible. You simply cannot currently go from a gem and create a package out of that, because of how gems are done. That, more than anything, was what I was objecting to. I can understand that many Ruby developers do not care about this much; that they are used to RubyGems, and that they can geet anything they need using that system. However, they should understand that different people have different preferences, and that packages for systems other than RubyGems is going to be necessary if their software is to become popular beyond the select set of people who care enough about Ruby. I personally know of plenty of people (myself included) who prefer things to be installed as operating-system level packages on the Linux systems they install; and even if it's not your personal preference to do things that way, you should at the very least not stand in the way of those who do.

8 December 2008

Biella Coleman: dino-anon (or how to end your hacker class on a lulzy note)



dino-anon, originally uploaded by the biella.
As I mentioned in my last entry, the end of the semester is one of overwhelming, seemingly unstoppable entropy. People and Professors are tired and to help move us through this thickness, I try to pick fun readings. Today I also had the opportunity to have a group of NYC-based Anonymous come to class today to give us a window into the politics of protesting Scientology and this of course worked magnificently against the end-of-semester-entropy. In a nutshell: they did not fail or in their own lingo, it was an Epic Win. They provided a really nice set of presentations (and then also proceeded to collectively draw the lulzy image featured above) about their origins, their purpose and some of the tensions within the group. Then we had a class discussion about a number of topics, including their argot and “offensive” (offensive) [I am not sure whether to put " " or not] online language. While I am still ambivalent about their use of “fag” and “nigger”, I greatly admire Anonymous for their politics of pleasure and aesthetic audacity (not to mention creativity). They insist that protest must be fun, must be wild, must go beyond simply rational discourse. If you are interested in this topic, check out Stephen Duncombe’s excellent (and very readable) book Dream, which I happened to be (somewhat randomly) reading right now and it helped give me a new fresh perspective about the politics driving Anon.
Duncombe is a professor at NYU Passionate (with a Capital P) about politics with a lot of real world political experience who examines the importance of desire, fantasy, and pleasure for progressives and leftists who tend to shy away or even denigrate such impulses. And it is certainly the case that Anonymous is experimenting, consciously and unconsciously, with these very questions and impulses which helps explain why many people jumped on board. I hope to post the audio from the class a little later on.

Next.

Previous.