Search Results: "tar"

22 January 2013

Eddy Petrișor: (Rsnapshot) backup and security - I see problems

In my previous post I was asking for suggestions for backup solutions that would be open/free software, do backups over the network to a local HDD, be cross platform to allow Windows and Linux clients and not be too CPU/memory hungry (on the server).

Several people suggested rsnapshot, BackupPC, areca-backup, and rsync. Thank you for all your suggestions, you have been a tremendous help. I have decided to give rsnapshot a try since it was suggested to me that it would actually do what is supposed to do for Windows clients, too (which was initially my perceived show stopper for rsnapshot).

Still, when getting to the implementation, I was a little disappointed by the very permissive access that needs to be provided on the client machines, since the backup is initiated from the backup server. Even the so called more secure suggested solutions seem way too permissive for my taste, since losing the control over the backup system means basically giving total access to the data from all client machines, which is quite a big problem in my opinion.

The data-transfer mechanism employed by rsnapshot is simply
  1. S ==(connects and reads all data)==> C
  2. S stores data in the final storage area
Am I the only one seeing a problem with this idea? If the server can connect to all your client machines and read all areas as it pleases, even if you restrict it to some directories, the data is already compromised when the backup server is compromised (think .ssh private keys, files with wireless network passwords and so on; I won't say card information - you don't keep credit/debit card information on your computer, or at least not in plain text, do you?).

What I would consider a better alternative would be a server-initiated dialogue which goes a little like this (S is server, C is client, '=' represent connections via ssh):
  1. S ---(requests backup initiation procedure)---> C
  2. S waits for a defined period of time that C connects back to send (already encrypted) data; if it doesn't arrive, it aborts
  3. S <===(sends encrypted data to be backed up)=== C
  4. S <-(signals the completion of data transfer)-- C
  5. S stores the data in the final storage area
This way, the server can allow access from the clients only in designated areas (even a chroot is possible) from designated clients, access can even be provided only after a port knocking procedure and only during the backup time frame (since the server initiates the negotiation, it can expect only then the knocks, but only then), so the server is quite well secured. The connection to the server can even be done through an unprivileged account, it can even be one account per client machine which can be limited to a scponly shell, if you care for that level of security.

On the other hand, the client information is secure since it can be encrypted directly on the client machine and sent only after encryption, the client machine can decide and control what it sends, while the backup server can only store what the client provides. Also, if the server is compromised, the clients' data and system aren't compromised at all, since the data is on the backup machine, but is encrypted with a key only known on the client (and a backup copy of it can be stored somewhere safe).

I am aware this approach can be problematic for permission (user/group preservation), but it doesn't happen if there is a local <-> remote user mapping or simply the numeric IDs are kept.

I am also aware this means smarter clients and might mean the Windows machine might not be able to implement this completely, but a little more security than "here is all my data" can still be achieved, can't it?

What do other people think? Am I insane or paranoid?

I think I can implement this type of protocol in some scripts (at least one for server and one for clients) and use the backup_script feature of rsnapshot to keep this clean and nice within rsnapshot.

What might prove problematic with this approach is that rsync spedup is lost (might be?) because the copy is done to a temporary directory which, I assume, is empty, so tough luck. Another problem seems to be that every time the backup is done, the client has to encrypt each of the files to backup, which seems to be a real performance penalty, especially if the data to be backed up is quite large.

Is there an encryption layer that does this automatically at file level in the same/similar manner that LUKS does for entire block devices? Having the right file names, but with scrambled/encrypted contents seems to be the ideal solution from this PoV.

Thanks for reading and possible suggestions you might point me to.

P.S.: I just thought of this, if there was an encryption layer implemented with fuse which is mounted in some directory on the client machine, the default rsnapshot mechanism could actually work, and this would mitigate the data accessibility issue and the performance issue since that file system could be contained within a chroot and the encryption/scrambling would be done transparently on the client, so no data is plainly accessible. Does anybody know such a FUSE implementation that does on-the-fly file encryption?

P.P.S.: EncFS does exactly what I want with its --reverse option which is exactly designed for this purpose:
Normally EncFS provides a plaintext view of data on demand. Normally it stores enciphered data and displays plaintext data. With --reverse it takes as source plaintext data and produces enciphered data on-demand. This can be useful for creating remote encrypted backups, where you do not wish to keep the local files unencrypted.
Great!

21 January 2013

Dirk Eddelbuettel: digest 0.6.1

digest version 0.6.1 is now on CRAN, and I will push the corresponding version into Debian shortly. Duncan Murdoch added AES support, and helped me fix two issues which (annoyingly) made the Rout.save output differ on another platform. CRANberries provides the usual summary of changes to version 0.6.0. As usual, our package is available via the R-Forge page leading to svn and tarball access, my digest page and the local directory here.

Petter Reinholdtsen: First prototype ready making hardware easier to use in Debian

Early this month I set out to try to improve the Debian support for pluggable hardware devices. Now my prototype is working, and it is ready for a larger audience. To test it, fetch the source from the Debian Edu subversion repository, build and install the package. You might have to log out and in again activate the autostart script. The design is simple: I still need to come up with a better name for the system. Here are some screen shots showing the prototype in action. First the notification, then the password request, and finally the request to approve all the dependencies. Sorry for the Norwegian Bokm l GUI.



The prototype still need to be improved with longer timeouts, but is already useful. The database of hardware to package mappings also need more work. It is currently compatible with the Ubuntu way of storing such information in the package control file, but could be changed to use other formats instead or in addition to the current method. I've dropped the use of discover for this mapping, as the modalias approach is more flexible and easier to use on Linux as long as the Linux kernel expose its modalias strings directly. Update 2013-01-21 16:50: Due to popular demand, here is the command required to check out and build the source: Use 'svn checkout svn://svn.debian.org/debian-edu/trunk/src/hw-support-handler/; cd hw-support-handler; debuild'. If you lack debuild, install the devscripts package. Update 2013-01-23 12:00: The project is now renamed to Isenkram and the source moved from the Debian Edu subversion repository to a Debian collab-maint git repository. See build instructions for details.

Raphael Geissert: January's Debian mirrors update

It's been slightly over a month since December's update to http.debian.net. Since then, Debian's mirrors network has grown by 6 more archive mirrors. Many thanks to the Debian sponsors running them!

There are now about 370 archive mirrors serving it over http, an increase of 40 (12%) since April last year. The number of backports mirrors is now at 82, and 25 for archive.debian.org.

On the http.debian.net front there haven't been many changes since last month. Some major changes are in the works, but they didn't make it into January's code update. There were, however, a few issues with one of the hosts during the first couple of days of January. Apologies for the inconveniences it may have caused.

A new version of ftpsync addressing some issues should hopefully be released some time next month. Stay tuned to the debian-mirrors mailing list for a call for testers and probably a survey for mirror administrators.

20 January 2013

Hideki Yamane: digging CVEs

Sometimes I would check security-tracker to find problems that is already fixed but not tagged as so.

Today, I found problems that not in unstable package but not tagged as fixed for Eucalyptus, because it was removed from unstable once but re-introduced to archive again, then data in security-tracker was not updated as well. So update it (CVE-2010-3905 and CVE-2011-0730).

And also found vulnerablity in firebird2.5 was fixed in upstream svn, so pick it and made it as a patch, just report to BTS (for squeeze, too).

For isc-dhcp, it has a vulnerability from BIND9... I've been surprised but dhcp includes bind9 source since 4.2, so BIND9 vulnerability affects isc-dhcp. Pick a diff from bind9 for squeeze security update and made a patch (Bug#698597) .

Russ Allbery: Review: How I Proposed to My Wife

Review: How I Proposed to My Wife, by John Scalzi
Publisher: Subterranean
Copyright: 2007
ISBN: 1-59606-480-3
Format: Kindle
Pages: 25
How I Proposed to My Wife: An Alien Sex Story is really a novelette, which I would normally give a paragraph review in a larger collection. But it was independently published and is independently purchasable, so it gets its own page. I read this as part of the Subterranean Scalzi Super Bundle, which collects a bunch of Scalzi's independently-published short fiction. It was originally a chapbook that accompanied Subterranean #4. Those familiar with Scalzi's other work should expect something more like Agent to the Stars than the Old Man's War series. The first-person protagonist is a new writer for New World Man think the sort of glossy magazine you see in supermarket racks. Every issue, they do an alien story; this issue, they decide to do an article on alien courtship and, of course, assign the new writer to it. Most of the story is a mildly entertaining series of arranged dates in which Charlie has somewhat awkward conversations with various aliens about how their date protocols work, and then ends up in situations that are sexual to the aliens but mostly just bizarre and embarassing. If you've read any other Scalzi humor, you know what to expect: good-natured, light, mild parody with a touch of slapstick. There is, of course, a twist that gets the story to the point foreshadowed in the title. I didn't find it particularly believable, but you don't read this story for its complex and realistic world-building. The ending joke, which is back onto the firmer ground of purely human romantic conventions, is probably the funniest moment in the story. If you like Scalzi's humor, you probably won't regret spending the $1 that the story currently costs on Amazon. (You can also still get it on a shareware basis direct from Scalzi, but given that the minimum useful donation is 30 cents, it's probably easier to just buy from Amazon unless you don't want Amazon's format or DRM.) On the other hand, you probably also won't regret forgetting this story exists and going on with the rest of your life. Rating: 6 out of 10

19 January 2013

Dirk Eddelbuettel: Sing The Truth at Symphony Center

Just back in from an amazing concert at the Chicago Symphony Center. Vocalists Angelique Kidjo, Dianne Reeves and Lizz Wright supported by all-star band of Geri Allen on piano, Romero Lubambo on electric and acoustic guitar, James Genus on electric bass guitar, Munyungo Jackson on percussion and Terri Lyne Carrington on drums (and musical director). The concert alternates between solos and joint pieces with more joy and soul than I heard in a long time. Great evening.

Charles Plessy: Debian-Installer in a Cloud, part 2.

I finally manage to create an image of the Debian Installer automatically, with no key or password transiting on the Cloud or the network, and without using the ec2-api-tools, which are non-Free. Thanks to Eucalyptus for implementing the attribute InstanceInitiatedShutdownBehavior in euca2ools 2.1.2. In brief, I start a micro-instance with an additional volume, and I pass it a script for cloud-init, which will download the Installer on the volume and shut down the instance. Then, I register the volume as a machine image. Those machine images of the Installer are to be preseeded via instance metadata to install Debian on another volume. I am still struggling with the pre-configuration. More details will follow in the next article. In the meantime, here are the commands I a using. On the local computer, start an instance on the Cloud. Until #696595 is fixed and cloud-init is installed by default, I use a Ubuntu image. 
HELPER_AMI=ami-7609bb77 # Ubuntu 12.04 LTS Precise amd64 EBS (Asia North-East)
Start the instance with an extra volume of 1 GiB, which will persist after termination (/dev/sdb=:1:false). Pass the script debigen-install-installer-cloud (see below). 
HELPER_INSTANCE=$( euca-run-instances \
        --instance-initiated-shutdown-behavior terminate \
        --instance-type t1.micro \
        --block-device-mapping /dev/sdb=:1:false \
        --user-data-file debigen-install-installer-cloud \
        $HELPER_AMI  
        tee /dev/stderr   grep INSTANCE   awk ' print $2 ')
Wait that the boot finished. 
while [ ! $(euca-describe-instances $HELPER_INSTANCE   grep INSTANCE   cut -f 6   tee /dev/stderr) = "running" ]
    do sleep 30
done
Get the volume's identifier 
TARGET_VOLUME=$( euca-describe-volumes  
        grep $HELPER_INSTANCE   grep '/dev/sdb'  
        tee /dev/stderr   awk ' print $2 ')
Wait that the scripts shuts down the instance. 
while [ ! $(euca-describe-instances $HELPER_INSTANCE   grep INSTANCE   cut -f 6   tee /dev/stderr) = "terminated" ]
    do sleep 30
done
Snapshot the volume and register it as a machine image. 
PV_KERNEL=aki-44992845 # Boot PV-Grub on hd0 (Asia North-East)
TARGET_SNAPSHOT=$( euca-create-snapshot $TARGET_VOLUME  
        tee /dev/stderr   awk ' print $2 ')
while euca-describe-snapshots $TARGET_SNAPSHOT   grep -q pending ; do sleep 30 ; done
euca-register \
    --name test_di \
    --description test_of_debian_installer \
    --snapshot $TARGET_SNAPSHOT \
    --kernel $PV_KERNEL \
    --architecture x86_64
Here is the script called debigen-install-installer-cloud, used above. 
#!/bin/sh -ex    
mke2fs -L debian-installer /dev/xvdb -F
mount LABEL=debian-installer /mnt/
cd /mnt    
ARCH=amd64
DIST=squeeze
DI_VERSION=20110106+squeeze4+b2
MIRROR=jp
BASEURL=http://ftp.$MIRROR.debian.org/debian/dists/$DIST/main/installer-$ARCH/$DI_VERSION/images/netboot/xen
wget $BASEURL/initrd.gz $BASEURL/vmlinuz    
mkdir -p boot/grub    
cat > boot/grub/menu.lst <<__END__
default 0
timeout 3
title  Debian Installer ($DI_VERSION $ARCH)
root   (hd0)
kernel /vmlinuz root=LABEL=debian-installer ro console=hvc0 auto=true priority=critical url=http://169.254.169.254/latest/user-data DEBIAN_FRONTEND=text
initrd /initrd.gz
__END__
sleep 30     
halt

18 January 2013

Russell Coker: Phone Calls and Other Distractions

Harald Welte has written about the distraction of phone calls and how it impacts engineering work [1]. He asks why people feel that they are entitled to interrupt him given the cost to his work. Some years ago while working as a programmer I was discussing such things with a colleague who worked for the consulting part of the same company. He was really surprised when I told him that a phone call at the wrong time would cost me at least 30 minutes work and possibly an hour or more. His work was also quite technical and demanding but the difference between software development (where you need to think about a lot of program state to consider where the problem might be) and consulting (where you have to deal with a smaller number of configuration file options and sometimes getting debugging information to someone like me) is considerable. So the attitudes towards receiving calls also tends to be quite different. Computer work requires more concentration, thought, and knowledge of system state than many (most?) career choices. If someone finds that an unexpected phone call costs them no more than a few minutes work then it s quite reasonable of them to phone other people whenever they feel like it generally by default people think that everyone else is just like them. In terms of managing interruptions to my work, I generally encourage people to email me and that works reasonably well. So I don t have too many problems with distracting phone calls. I used Jabber for a while a few years ago but I didn t reinstall my Jabber server after it became corrupt because of the distraction. I believe that was due to using Jabber in the wrong way. I should have just started a Jabber client when I wasn t doing anything important and then killed it when I started doing some serious coding. Having a Jabber message interrupt me when I m watching a TED talk or reading blogs is no big deal. In fact I could tell everyone who has my phone number that if they see me on Jabber then they can just phone me if they wish while knowing that it won t distract me from anything serious. I wonder if I could configure a Jabber client to only receive messages when a program such as mplayer is running. I have configured my laptop and workstation to never alert me for new mail. If I m not concentrating then I ll be checking my email frequently and if I am concentrating I don t want a distraction. I have configured my phone to give one brief vibration when it gets mail and not make any sound, I will only notice that if I m not concentrating on anything. It s a standard Android feature to associate ring tones with phone numbers, it s a pity that the K9 MUA doesn t allow associating email addresses with notifications. There are some people who s email could usefully trigger an audible alert. There is an K9 feature request from 2009 to allow notifications only when the IMAP flag Flagged is set which would allow the mail server to determine which users are important, but there s no sign that it will be implemented soon. I ve started playing with Google+ recently due to Ingress team interaction being managed through it. Google+ seems quite poor in this regard, it defaults to making a ring tone for lots of different events. Turning that off is easy enough but getting notifications only about things that are important to me seems impossible. I would like to get an audible alert when someone makes a Google+ post with an Ingress code (because they expire quickly and because they only seem to be posted at times when I m not busy) but not get audible alerts about anything else. I m sure that most people who use Google+ would like to have different notifications for various types of event. But the Android client has options for whether there should be vibration and/or noise and for which events get the notifications. No options for different notifications for different events and for treating some community posts differently from others. It seems that the default settings for most programs suit people who never need to spend much time concentrating on a task. It also seems that most programs don t offer configuration options that suit the needs of people who do concentrate a lot but who also sometimes receive important phone calls and email. It s ironic that so many applications are designed in the least optimal way for the type of people who develop applications. The Google+ developers have an excuse as doing what I desire would be quite complex. But there are other programs which should deal with such things in a user friendly manner.
Related posts:
  1. Globalisation and Phone Calls I just watched an interesting TED talk by Pankaj Ghemawat...
  2. A Mobile Phone for Sysadmin Use My telco Three have just offered me a deal on...
  3. Health and Status Monitoring via Smart Phone Health Monitoring Eric Topol gave an interesting TED talk about...

Ian Campbell: Using Valgrind to Debug Xen Toolstacks

What is Valgrind? Valgrind is a framework for building dynamic analysis tools. Several useful tools are included in the Valgrind distribution including tools to check dynamic memory usage (memcheck), a cache profiler (cachegrind), heap profiler (massif) and thread debugger (helgrind) among others. Valgrind also provides a framework which can be used to build other tools. The Valgrind tool which I find most useful and the one which I have most experience with is memcheck. This tool can detect all manner of memory management problems, including use after free, using uninitialized data, memory leaks, double free. Between them these can result in savings of many hours of staring a core dumps and gdb backtraces. How Does memcheck Work? At its core Valgrind is a dynamic binary translation engine, which is used to instrument the code at runtime. In the case of memcheck this is used to track properties of the memory in a process, including aspects such as whether each bit (yes, bit) of memory has been initialized since it was allocated. It also tracks this information as data is loaded into registers, so it can know if a given register is currently tainted with uninitialized data or not. As well as instrumentation through binary translation Valgrind also includes instrumented versions of the C dynamic memory allocation functions which are used to track whether a each bit of memory is currently considered free or allocated, as well as tainting registers when they contain a pointer to memory which has been freed. A large part of memcheck's functionality is built upon Valgrind's ability to determine when memory has been initialized. However although binary translation can be used to instrument when the application itself has initialized some memory it is not currently possible to instrument the behaviour of system calls, in other words Valgrind cannot automatically determine when memory has been initialised e.g. after a read(2) system call. For this reason Valgrind has baked in knowledge about the behaviour of system calls on various platforms. For example lets consider the read(2) system call. The prototype of read(2) is: 
    ssize_t read(int fd, void *buf, size_t count);
This system call reads up to count bytes from the file descriptor fd into the memory pointed to by buf and returns the number of bytes read. Here is the code within Valgrind which handles this system call (found in coregrind/m_syswrap/syswrap-generic.c): 
PRE(sys_read)
 
   *flags  = SfMayBlock;
   PRINT("sys_read ( %ld, %#lx, %llu )", ARG1, ARG2, (ULong)ARG3);
   PRE_REG_READ3(ssize_t, "read",
                 unsigned int, fd, char *, buf, vki_size_t, count);
   if (!ML_(fd_allowed)(ARG1, "read", tid, False))
      SET_STATUS_Failure( VKI_EBADF );
   else
      PRE_MEM_WRITE( "read(buf)", ARG2, ARG3 );
 
POST(sys_read)
 
   vg_assert(SUCCESS);
   POST_MEM_WRITE( ARG2, RES );
These two functions are called before and after the system call respectively. The main piece of functionality is the calls to PRE_MEM_WRITE and POST_MEM_WRITE. In this case PRE_MEM_WRITE is used to check that the memory starting at ARG2 (recall that this is the supplied buffer) and extending for ARG3 bytes (the system calls count argument) consists of allocated memory. After the system call in complete the post function calls POST_MEM_WRITE to indicate to Valgrind that RES bytes of the buffer have know been initialized. The above is a relatively simple case, but the above two hooks must be supplied for each system call on each platform which Valgrind wishes to support. One of the most problematic system calls is the ioctl(2) system call. The ioctl call takes a file descriptor, a request number and a pointer to a per-ioctl argument structure and implements per-device control requests. There are literally dozens of devices many with their own specific ioctls. For this reason a sizable portion of Valgrind code is dedicate to decoding the behaviour of the myriad of potential ioctls and providing pre- and post-call instrumentation for them. Interacting with the Hypervisor from Userspace Under Xen the toolstack is a normal userspace process running in the control domain (typically domain 0). As part of its operation the toolstack needs to communicate with the hypervisor and to do this it uses hypercalls. However a userspace process cannot simply make hypercalls on its own, instead it must request that the OS kernel make the hypercall for it. This prevents just any userspace process making a hypercall, since the kernel will first check the privilege of the process making the request. Under Linux a userspace process makes hypercalls by using the ioctl(2) system call on a special "Privileged Command" (privcmd) device. In this case the toolstack uses the IOCTL_PRIVCMD_HYPERCALL ioctl request which takes a hypercall number and the hypercall arguments as its argument. Other OSes which can be used as a control domain posses similar interfaces. Making memcheck Work With Xen Toolstacks? As might be expected the majority of the work to allow Valgrind to work on Xen toolstack processes was teaching it about the IOCTL_PRIVCMD_HYPERCALL ioctl. Fortunately it was not necessary to teach Valgrind about every hypercall but only about those which are used by toolstacks. This is a smallish subset of all hypercalls including the domctl interfaces and a handful of others. The initial batch of patches cover all of the system calls needed to start new domains, shut them down and migrate them using the XL toolstack. New hypercalls will be added as they are discovered to be missing. Getting and Using This Functionality The initial set of patches were accepted into Valgrind's Subversion repository trunk revision 13081 in October 2012 and are expected to be part of the 3.9.0 release. Until 3.9.0 is released it will be necessary to get this functionality from SVN. Fortunately this is relatively simple and follows the usual autofoo dance. Although the usual caveats regarding running unreleased software apply. 
  $ svn co svn://svn.valgrind.org/valgrind/trunk valgrind
  $ cd valgrind
  $ ./autogen.sh
  $ ./configure
  $ make
  $ sudo make install
Once Valgrind is built and installed simply call it passing the command to be debugged as an option: 
  # valgrind xl list
  ==16186== Memcheck, a memory error detector
  ==16186== Copyright (C) 2002-2012, and GNU GPL'd, by Julian Seward et al.
  ==16186== Using Valgrind-3.9.0.SVN and LibVEX; rerun with -h for copyright info
  ==16186== Command: xl list
  ==16186== 
  Name                                        ID   Mem VCPUs    State   Time(s)
  Domain-0                                     0   512     4     r-----     332.5
  ==16186== 
  ==16186== HEAP SUMMARY:
  ==16186==     in use at exit: 0 bytes in 0 blocks
  ==16186==   total heap usage: 16 allocs, 16 frees, 93,529 bytes allocated
  ==16186== 
  ==16186== All heap blocks were freed -- no leaks are possible
  ==16186== 
  ==16186== For counts of detected and suppressed errors, rerun with: -v
  ==16186== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 39 from 6)
Here we see that xl list has no leaks! However if I comment out the call to libxl_dominfo_list_free in tools/libxl/xl_cmdimpl.c:main_list() then instead Valgrind reports: 
[...]
  ==16203== HEAP SUMMARY:
  ==16203==     in use at exit: 90,112 bytes in 1 blocks
  ==16203==   total heap usage: 16 allocs, 15 frees, 93,529 bytes allocated
  ==16203== 
  ==16203== LEAK SUMMARY:
  ==16203==    definitely lost: 0 bytes in 0 blocks
  ==16203==    indirectly lost: 0 bytes in 0 blocks
  ==16203==      possibly lost: 90,112 bytes in 1 blocks
  ==16203==    still reachable: 0 bytes in 0 blocks
  ==16203==         suppressed: 0 bytes in 0 blocks
  ==16203== Rerun with --leak-check=full to see details of leaked memory
  ==16203== 
  ==16203== For counts of detected and suppressed errors, rerun with: -v
  ==16203== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 39 from 6)
[...]
So it has caught the memory leak! Rerunning with the suggests --leak-check=full goes even further and tells me where the leaked memory was allocated: 
[...]
  ==16204== 90,112 bytes in 1 blocks are possibly lost in loss record 1 of 1
  ==16204==    at 0x4024480: calloc (vg_replace_malloc.c:593)
  ==16204==    by 0x4050CE4: libxl_list_domain (libxl.c:548)
  ==16204==    by 0x805EEC7: main_list (xl_cmdimpl.c:3931)
  ==16204==    by 0x804D6DD: main (xl.c:285)
[...]
To ease debugging of domain creation I find it useful to use the -F option to xl create to stop xl from forking and daemonising. Most xl subcommands do not daemonize and so need no special treatment. For more information on running Valgrind see the Valgrind Documentation. Conclusion Valgrind is a powerful tool in the arsenal of the C programmer, and can be used to catch a large number of hard to debug and common issues before they happen can save many hours of tedious debugging. The ability to use tools such as this on Xen toolstack processes is of enormous benefit and has already found several bugs in the XL toolstack.

Michael Banck: 18 Jan 2013

What I have been up to over the last couple of years

I have now worked at credativ for four years. A sizable part of that time I spent working with the Munich City Council's LiMux project. The focus of our work is helping them with their LDAP/web-based GOsa user and system installation(based on FAI)/configuration/management system. Personally, I mostly did project management and QA besides occasional coding, bug fixing and on-site consulting.

Contrary to a lot of our other customers, the Munich city council openly acknowledges the work by external companies and lets us talk about it as well. I wrote a lengthy blog post (now available in english) last month about our work. It explains how the LiMux project is using GOsa and FAI and work we did for them.

The project took a long time, but managed to pull off the mass migration over the last two years and reached their goal of 12000 migrated workstation last November. It is now widely regarded as a "success story" by german news sites, and I am glad to have had a small part in it. I really think this was (and still is) an important project, especially after several others backpedalled and Munich remained the last high-profile public sector migration in the german speaking countries.

Besides that, the LiMux project members are really nice people I get along with well both on working hours and during the Munich Debconf11 bid and organizing various BSPs at their office. It is a bit of a shame that not more of their work and customizations are available in Debian, but hopefully this well happen one day. At least our changes to GOsa are up on github.

Russ Allbery: Review: Soulless

Review: Soulless, by Gail Carriger
Series: Parasol Protectorate #1
Publisher: Orbit
Copyright: October 2009
ISBN: 0-316-07165-X
Format: Kindle
Pages: 357
Alexia Tarabotti is a spinster in London society in the time of Queen Victoria. She is possessed of a hopelessly overwrought and society-focused mother, two half-sisters whose only thoughts are for fashion and gossip, a mixed Italian heritage, an unpopular body shape and complexion, and a secret. In a world in which vampire hives, werewolf packs, and the occasional ghost are out of hiding and openly participating in society, and where the Bureau of Unnatural Registry (headed by a particularly intimidating werewolf who is also an earl) keeps track of the undead, Alexia is something still rarer, and not entirely acceptable. She is preternatural, able to draw the supernatural out of the undead with a touch. She has no soul. In Carriger's Victorian urban fantasy, laced with a gentle helping of steampunk, vampires, werewolves, and ghosts all have similar conditions. All three are undead; even the change for werewolves happens after normal death. But the change doesn't happen to everyone killed by the corresponding type of undead. The requirements are quite exacting and include the quantity of soul in the person being changed. Only those with a surfeit of soul, too much to stay dead, can be successfully changed. Vampires and werewolves both are surrounded by hopeful humans, called drones and clavigers respectively, who hope to be transformed and gain immortality. But no one can tell in advance how much soul someone has. Alexia is the precise opposite of the undead. Rather than a surfeit of soul, she has none at all, a condition inherited from her father but unknown to the rest of her family. Her kind, the preternatural, make up the vampire and werewolf hunters of legend, called soul-suckers, curse-breakers, and exorcists by the supernatural set. She's viewed with great suspicion by most, but with fascination by some for her ability to revert them to humanity with physical contact. Not that this has been helpful in finding a husband, or, really, for much of anything else in her life. Soulless opens with Alexia being accousted by a rogue vampire in the library, where she went for refuge from a private ball. Not only does he not know who she is, despite every hive in the area telling all of their new vampires about her, but he doesn't seem to even know what a preternatural is, and she has to kill him with her parasol. That is the first unusual occurance in an escalating mystery involving new vampires appearing from nowhere, loner vampires and werewolves disappearing, and a truly prodigious use of chloroform. Not to mention quite a few encounters with Lord Maccon, Earl of Woolsey, the Alpha werewolf who runs the Bureau of Unnatural Registry. The start of Soulless feels a bit uneven and stilted until the story gets moving and the reader gets used to Carriger's voice. It's written in the style of a comedy of manners and takes delight in long words and a semi-formal style. For example:
The vampire recovered his equanimity quickly enough. He reared away from Alexia, knocking over a nearby tea trolley. Physical contact broken, his fangs reappeared. Clearly not the sharpest of prongs, he then darted forward from the neck like a serpent, diving in for another chomp. "I say!" said Alexia to the vampire. "We have not even been introduced!"
Sometimes this works. Sometimes it doesn't. The frequency with which it works seemed to me to increase as the book went along. The best thing this book has going for it is the character of Alexia Tarabotti. Rather than the typical kick-ass female protagonist of a modern urban fantasy novel, Alexia is not cynical, battle-hardened, or courageously sarcastic. Her attitude for most of the book is something closer to bemusement crossed with exasperation. She sees herself as an outsider, has largely given up on fitting into any sort of proper society, and has little use for the expectations of her family. But she's still embedded within her culture, still shocked (at least momentarily) by breaches in etiquette, and quite able to fit in, if a bit uncomfortably. Her delightfully logical tone and constant open-minded self-analysis is what made this book for me. The plot is fairly straightforward (and has a bit too much anti-science flavor for me). The romance angle, which is reasonably obvious from early on in the book, was entertaining mostly in Alexia's combination of thoughtful analysis (based in part on her father's extensive and not entirely acceptable library) and runaway physical reactions, although I thought the latter was slightly overdone. But all the elements of the story improve noticably towards the end of the book. I was unenthused by the halfway point, but by the time I finished Soulless, I decided I would probably read the sequel. This is more in the enjoyably fun reading pile than a book that I would go to great effort to acquire, and it has a somewhat rocky start, but I enjoyed it and would mildly recommend it. Followed by Changeless. Rating: 7 out of 10

17 January 2013

Craig Small: Pre-selecting ToscaWidgets jqgrid values in TurboGears

My Turbogears project has recently reached an important milestone; one of the back-end processes now runs pretty much continuously and the plugins it use (or at least the ones I can see) are also working. That means I can turn to the front-end which displays the data the back-end collected. For some of the data I am using a ToscaWidgets (or TW2) widget called a jqGridWidget which is a very nice jquery device that separates the presentation and data using a json query. I ve mentioned previously about how to get a jqGridWidget working but left the pre-filtering out until now. This meant that my grid showed all the items in the database, not just the ones I wanted to see, but it was a start. Now this widget displays things called Attributes which are basically children of another model called Hosts. Basically, Attributes are things you want to check or track about a Host. My widget used to show all Attributes but often on a Host screen, I want to see its hosts only. So, this is how I got my widget to behave; I m not sure this is THE CORRECT way of doing it, but it does work. First, in the Hosts controller you need to create the widget and pass along the host_id that the controller has obtained. I was not able to use the sub-classing trick you see in some TW2 documentation but actually make a widget instance. 
class HostsController(BaseController):
    # other stuff here
    class por2(portlets.Portlet):
        title = 'Host Attributes'
        widget = AttributeGrid()
        widget.host_id = host_id
Next, the prepare() method in the Widget needs to get hold of the host_id and put it into the postData list. I needed to do it before calling super() because the options become one long string in the sub-class. 
class AttributeGrid(jqgrid.jqGridWidget):
    def prepare(self, **kw):
        if self.host_id is not None:
            self.options['postData'] =  'hostid': self.host_id 
        super(AttributeGrid, self).prepare()
This means the jqgrid when it asks for its JSON data will include the hostid parameter as well. We need that method to see the host ID so we can filter the database access. Finally in the JSON method for the Attribute we pick up and filter on the hostid. 
    @expose('json')
    @validate(validators= 'hostid':validators.Int() )
    def jqsumdata(self, hostid=0, page=1, rows=1, *args, **kw):
        conditions = []
        if hostid &gt; 0:
            conditions.append(model.Attribute.host_id == hostid)
        attributes =model.DBSession.query(model.Attribute).filter(and_(*conditions))
From there you run through the attributes variable and build your JSON reply.
Related articles

Craig Small: Pre-selecting ToscaWidgets jqgrid values in TurboGears

My Turbogears project has recently reached an important milestone; one of the back-end processes now runs pretty much continuously and the plugins it use (or at least the ones I can see) are also working. That means I can turn to the front-end which displays the data the back-end collected. For some of the data I am using a ToscaWidgets (or TW2) widget called a jqGridWidget which is a very nice jquery device that separates the presentation and data using a json query. I ve mentioned previously about how to get a jqGridWidget working but left the pre-filtering out until now. This meant that my grid showed all the items in the database, not just the ones I wanted to see, but it was a start. Now this widget displays things called Attributes which are basically children of another model called Hosts. Basically, Attributes are things you want to check or track about a Host. My widget used to show all Attributes but often on a Host screen, I want to see its hosts only. So, this is how I got my widget to behave; I m not sure this is THE CORRECT way of doing it, but it does work. First, in the Hosts controller you need to create the widget and pass along the host_id that the controller has obtained. I was not able to use the sub-classing trick you see in some TW2 documentation but actually make a widget instance. 
class HostsController(BaseController):
    # other stuff here
    class por2(portlets.Portlet):
        title = 'Host Attributes'
        widget = AttributeGrid()
        widget.host_id = host_id
Next, the prepare() method in the Widget needs to get hold of the host_id and put it into the postData list. I needed to do it before calling super() because the options become one long string in the sub-class. 
class AttributeGrid(jqgrid.jqGridWidget):
    def prepare(self, **kw):
        if self.host_id is not None:
            self.options['postData'] =  'hostid': self.host_id 
        super(AttributeGrid, self).prepare()
This means the jqgrid when it asks for its JSON data will include the hostid parameter as well. We need that method to see the host ID so we can filter the database access. Finally in the JSON method for the Attribute we pick up and filter on the hostid. 
    @expose('json')
    @validate(validators= 'hostid':validators.Int() )
    def jqsumdata(self, hostid=0, page=1, rows=1, *args, **kw):
        conditions = []
        if hostid &gt; 0:
            conditions.append(model.Attribute.host_id == hostid)
        attributes =model.DBSession.query(model.Attribute).filter(and_(*conditions))
From there you run through the attributes variable and build your JSON reply.
Related articles

Francesca Ciceri: T-Shirt Surgery: DebConf11 t-shirt redux

Please DO TRY this at home debconf11 t-shirt before and after the surgery Here's a little tutorial on how to modify a plain t-shirt, inspired by this one. What you'll need: Step 1: the neckline cutting the neckline Put the t-shirt on a flat surface, then start cutting out the collar, to create a new neckline: cut two slits about 2 cm from the collar (from both sides), then cut out a circle. Flip the tee and do the same on the back side. Step 2: the bottom hem cutting the bottom hem Try your shirt on and mark the desired lenght. I decided to create a curved hem, to make it more feminine: to do it, you'll just need to cut an half-moon shape on the bottom line of the t-shirt. Step 3: the shoulder strap (optional) adding a shoulder strap Just to add a little twist, I decided to sew a ribbon as shoulder strap. You'll just need to measure the required lenght with the tee on, and mark the place to sew the ribbon. After sewing the strap, you can also add a little bow on the front side. Don't throw away the scraps, use them to make a fabric necklace! Some resources on t-shirt surgery:

Tollef Fog Heen: Gitano git hosting with ACLs and other shininess

gitano is not entirely unlike the non-web, server side of github. It allows you to create and manage users and their SSH keys, groups and repositories from the command line. Repositories have ACLs associated with them. Those can be complex ("allow user X to push to master in the doc/ subtree) or trivial ("admin can do anything"). Gitano is written by Daniel Silverstone, and I'd like to thank him both for writing it and for holding my hand as I went stumbling through my initial gitano setup. Getting started with Gitano can be a bit tricky, as it's not yet packaged and fairly undocumented. Until it is packaged, it's install from source time. You need luxio, lace, supple, clod, gall and gitano itself. luxio needs a make install LOCAL=1, the others will be installed to /usr/local with just make install. Once that is installed, create a user to hold the instance. I've named mine git, but you're free to name it whatever you would like. As that user, run gitano-setup and answer the prompts. I'll use git.example.com as the host name and john as the user I'm setting this up for. To create users, run ssh git@git.example.com user add john john@example.com John Doe, then add their SSH key with ssh git@git.example.com as john sshkey add workstation < /tmp/john_id_rsa.pub. To create a repository, run ssh git@git.example.com repo create myrepo. Out of the box, this only allows the owner (typically "admin", unless overridden) to do anything with it. To change ACLs, you'll want to grab the refs/gitano/admin branch. This lives outside of the space git usually use for branches, so you can't just check it out. The easiest way to check it out is to use git-admin-clone. Run it as git-admin-clone git@git.example.com:myrepo ~/myrepo-admin and then edit in ~/myrepo-admin. Use git to add, commit and push as normal from there. To change ACLs for a given repo, you'll want to edit the rules/main.lace file. A real-world example can be found in the NetSurf repository and the lace syntax might be useful. A lace file consists of four types of lines: Rules are processed one by one, from the top and terminate whenever a matching allow or deny is found. Conditions can either be matches to an update, such as ref refs/heads/master to match updates to the master branch. To create groupings, you can use the anyof or allof verbs in a definition. Allows and denials are checked against all the definitions listed and if all of them match, the appropriate action is taken. Pay some attention to what conditions you group together, since a basic operation (is_basic_op, aka op_read and op_write) happens before git is even involved and you don't have a tree at that point, so rules like: 
define is_master ref refs/heads/master
allow "Devs can push" op_is_basic is_master
simply won't work. You'll want to use a group and check on that for basic operations and then have a separate rule to restrict refs.

Russ Allbery: A few last thoughts on Aaron Swartz

Daniel Kahn Gillmor has a very good blog post. You should read it. It includes a thoughtful rebuttal to some of my earlier thoughts about activism. I think I'm developing a richer understanding of where I see boundaries here, but after my last post, I also realized that by focusing on the specific details of what should have been a minor alleged crime, I'm derailing. Swartz did so much else. I made a note to come back to the more theoretical discussion in six months; now isn't the time. Now is the time to celebrate open content and all of the things Swartz achieved. (But thank you very much to the multiple people who have pointed out flaws in my reasoning and attempted approach.) Hopefully, it's also an opportunity to keep the pressure on for a saner and less abusive judicial system that doesn't threaten people with ridiculous and disproportional punishment in order to terrify them into unwarranted plea bargains. The petition I mentioned has reached nearly 40,000 signatures and passed the threshold (at the time it was posted) for forcing a White House response. Probably more importantly, it also seems to be creating the feedback cycle that I was hoping to see: the popularity of the petition is causing this story to stay in the news cycle and continue to be written about, which in turn drives more signatures to the petition. I'm not particularly hopeful that the Obama administration cares about the vast and deep problems with our criminal justice system, but I'm somewhat more hopeful that they, like most politicians, hate news cycles that they don't control. The longer this goes on, the stronger the incentive to find some way to make it go away, which could lead to real disciplinary action. A key committee in the US House of Representatives is starting a formal investigation. One of my local representatives has proposed modifying the US federal law on computer fraud and abuse to remove violations of terms of service from the definition of the crime. (I don't have much hope that this will pass when proposed by the minority party in a fairly hostile House, but the mere act of proposing it keeps the news focus on.) Glenn Greenwald has a (typically long-winded) round-up of news in the Guardian. Note that both Greenwald and Declan McCullagh link directly to the petition in articles in mainstream news outlets. One thing that slacktivism can do is perpetuate a news cycle until it gets more uncomfortable for people in power. It's still nowhere near as effective as the types of activism that Swartz was so good at, but in this specific case I think one gets a reasonable return on one's five-minute investment of effort. I'm going to stop talking about this now, since other people are a lot better at this sort of post than I am. But one last link: the medical community has a related open content problem, and theirs is also killing people. Possibly people you know. If all of this has inspired you, as it has me, to care even more about open content, be watching the push for open access to clinical trial data. More background is in Ben Goldacre's TED talk.

Benjamin Mako Hill: 1-800-INTERNET.COM

I just returned home from Aaron Swartz s funeral in Chicago. Aaron was a good friend. The home I ve returned to is an apartment that was Aaron s before it was mine, that I have lived in with Aaron during several stints, and that I still share with many of his old books and posters. Although, I ve spent what feels like most of the last five days reading things that people have written about Aaron, I m still processing and digesting myself. Right now, I m very sad and at a loss for words. While I reflect, I wanted to share this video recently put online by Finne Boonen. The video was made in 2006 at a Web 1.0 Elevator Pitch Competition held at Wikimania 2006 about a year after that both Aaron and I moved to Cambridge and met. The goal of the contest was to pitch Web 1.0 DotCom business ideas to a team of real Web 1.0 investors like it was still 1999. Aaron and I formed a team along with SJ Klein (who I traveled to the funeral with this week), and Wikimania general counsel and interim executive director Brad Patrick. The video shows how as Danny O Brien has reminded us Aaron was funny. He came up with many our teams best lines in addition to checking our Web 1.0 boxes for tech guru and Stanford dropout. Our pitch for 1-800-INTERNET.COM is in the video below. The transcript was done by Phoebe Ayers in Facebook and the video is also available in WebM. <iframe allowfullscreen="" frameborder="0" height="315" src="http://www.youtube.com/embed/NzFjWY6Fd_g" width="420"></iframe>
SJ: You know, Mako and I had some pretty good ideas for improving connectivity to the internet, and we think we can reach 90% of the world s population. So think about this. You re sitting in a Starbucks, and you need to connect to the internet. But you can t, because there s no internet. But what is there, near every Starbucks? There s a payphone! You pick up the payphone, and you call . 1-800-INTERNET. You can connect to our bank of researchers on our fast T1 connections and get any information you need! So, we don t actually have 1-800-INTERNET yet, we have 1-800-225-3224, so the first thing we need to do is buy the number. So here s Mako, who is our web designer from UC Santa Cruz and Bradford, our financial guru, and Aaron, who s handling all of our technical implementation. But Mako, you should explain the earballs. Mako: So, so, so yeah, so most people on the Internet are going for the eyeballs, but they ve just left all of these earballs. So I have some experience in web design, and it s true that this isn t really a website, but we still need good web design. So, so, I ve actually got a really experienced team, we can go into later, and we have some really great earcons not icons, but earcons.. And it s going to be all together, not apart like some of the websites. It s going to be together. Brad: so how does this work technically? Aaron: Well, I mean, so I only spent one year at Stanford but that s Ok, because there are new developmental technologies, we re going to throw away all that old stuff, we re going to use really reliable and efficient well-designed code that everyone can clearly understand, and write the whole thing in Perl. I know this is a risk, but I am confident that Perl is going to destroy those old C websites. No one will write websites in C anymore once we do this, it s going to be so much faster, and so dynamic, everythings going to be like, on top of everything. It s going to be great. Bradford: So here s the business model. It s really really simple, and it s a really really great idea. It s all about the licensing. Because what we re going to have are these underlying audio ads, While you re on the phone you re going to hear this subliminal advertising message. And the way it works is really really cool, because it s really really low volume, it s high impact! And it s even better, because we license it, and the way it works is when a caller calls 1-800-Internet, they re hearing the ad, but so is the representative, so we get to bill em twice! So that s it: All: 1-800-INTERNET.COM
We did not win and I still believe that we were robbed.

16 January 2013

Michal &#268;iha&#345;: Weblate has per project ACL

I've got yet another question whether Weblate has any sort for ACL to limit access to project. As usual, my answer was no, but I started to think how I would implement such feature. After a little bit of thinking and playing with code, I've realized, that this is not that hard as I thought and implemented it in few minutes. So there is another ofter requested feature which will come in Weblate 1.4 - ACL. It provides simple way to limit access to some projects to some users. The default behavior is still that anybody logged in can do anything, but this can be changed per project by enabling ACL for it. More details are available in our documentation.

Filed under: English Weblate 0 comments Flattr this!

Francois Marier: Moving from Blogger to Ikiwiki and Branchable

In order to move my blog to a free-as-in-freedom platform and support the great work that Joey (of git-annex fame) and Lars (of GTD for hackers fame) have put into their service, I decided to convert my Blogger blog to Ikiwiki and host it on Branchable. While the Ikiwiki tips page points to some old instructions, they weren't particularly useful to me. Here are the steps I followed.

Exporting posts and comments from Blogger Thanks to Google letting people export their own data from their services, I was able to get a full dump (posts, comments and metadata) of my blog in Atom format. To do this, go into "Settings Other" then look under "Blog tools" for the "Export blog" link.

Converting HTML posts to Markdown Converting posts from HTML to Markdown involved a few steps:
  1. Converting the post content using a small conversion library to which I added a few hacks.
  2. Creating the file hierarchy that ikiwiki requires.
  3. Downloading images from Blogger and fixing their paths in the article text.
  4. Extracting comments and linking them to the right posts.
The Python script I wrote to do all of the above will hopefully be a good starting point for anybody wanting to migrate to Ikiwiki.

Maintaining old URLs In order to make sure I wouldn't break any existing links pointing to my blog on Blogger, I got the above Python script to output a list of Apache redirect rules and then found out that I could simply email these rules to Joey and Lars to get them added to my blog. My rules look like this: 
# Tagged feeds
Redirect permanent /feeds/posts/default/-/debian http://feeding.cloud.geek.nz/tags/debian/index.rss
Redirect permanent /search/label/debian http://feeding.cloud.geek.nz/tags/debian
# Main feed (needs to come after the tagged feeds)
Redirect permanent /feeds/posts/default http://feeding.cloud.geek.nz/index.rss
# Articles
Redirect permanent /2012/12/keeping-gmail-in-separate-browser.html http://feeding.cloud.geek.nz/posts/keeping-gmail-in-separate-browser/
Redirect permanent /2012/11/prefetching-resources-to-prime-browser.html http://feeding.cloud.geek.nz/posts/prefetching-resources-to-prime-browser/

Collecting analytics Since I am no longer using Google Analytics on my blog, I decided to take advantage of the access log download feature that Joey recently added to Branchable. Every night, I download my blog's access log and then process it using awstats. Here is the cron job I use: 
#!/bin/bash
BASEDIR=/home/francois/documents/branchable-logs
LOGDIR=/var/log/feedingthecloud
# Download the current access log
LANG=C LC_PAPER= ssh -oIdentityFile=$BASEDIR/branchable-logbot b-feedingthecloud@feedingthecloud.branchable.com logdump > $LOGDIR/access.log
It uses a separate SSH key I added through the Branchable control panel and outputs to a file that gets overwritten every day. Next, I installed the awstats Debian package, and configured it like this: 
$ cat /etc/awstats/awstats.conf.local
SiteDomain=feedingthecloud.branchable.com
LogType=W
LogFormat=1
LogFile="/var/log/feedingthecloud/access.log"
Even if you're not interested in analytics, I recommend you keep an eye on the 404 errors for a little while after the move. This has helped me catch a critical redirection I had forgotten.

Limiting Planet feeds One of the most common things that happen right after someone migrates to a new blogging platform is the flooding of any aggregator that subscribes to their blog. The usual cause being the change in post identifiers. Unsurprisingly, Ikiwiki already had a few ways to avoid this problem. I chose to simply modify each tagged feed and limit them to the posts added after the move to Branchable.

Switching DNS Having always hosted my blog on a domain I own, all I needed to do to move over to the new platform without an outage was to change my CNAME to point to feedingthecloud.branchable.com. I've kept the Blogger blog alive and listening on feeding.cloud.geek.nz to ensure that clients using a broken DNS resolver (which caches records for longer than requested via the record's TTL) continue to see the old posts.

Next.

Previous.