Search Results: "cas"

25 January 2011

Paul van Tilburg: Also Giving up on Ruby Packaging

I have just sent a mail to the Debian/Ruby Extras mailing list stating that I will also give up most of my Ruby packaging efforts and reduce my involvement in the team. I am following Lucas Nussbaum in his decision. Although our reasons are slightly different, I subscribe most of what he says in his blog post. Besides that, I haven t done much Ruby programming in the past years, making me lose the motivation for the huge load of packages I am responsible for and thus subconsciously perceive that as a burden, as I feel guilty not having/making time to work on them. So, what does this mean? I will handover (co)maintainership of over 50 libraries. Many of which have dead upstream, unfortunately. Also will I be looking for someone to take over the administrative tasks of heading the team such as mailing list moderation and Alioth repository access. I will keep maintaining some programs that I still use, such as Camping, and am approachable for (mentor) uploads. Meanwhile I will continue my work on my PhD thesis, which is progressing well and should lead to my defence (hopefully) somewhere around September this year. That said, I will try to keep tinkering with Ruby and other F/OSS projects. Oh, and also to get more motivated and some inspiration

Christoph Berg: HP 2605dn not recognizing new cartridge

When my wife bought her HP 2605dn color laser, she had also ordered a set of replacement cartridges because the originals would supposedly not last very long. It turned out that they would last over two years, though. Now, the black cartridge was empty. The replacements are original HP components, but recycled/refilled second hand from a different source. I put in the black one, and the trouble started with the new drum having a big scratch that would appear in every print. After two years, returning the cartridge was not an option. As the old drum was still good, the idea was to put the new toner into the old cartridge. A bit of googling (and visual inspection) quickly revealed that you are not supposed to do this, the device doesn't have any facilities to access anything. Some post did indicate though that the cartridge actually consists of two parts that are only held together by two pivot pins and two small springs. If you pull the two parts apart a bit, you can feel the axis the parts are moving around. So, I would try to use the old drum part with the new toner container part. Removing the springs is easy (also removing the flap covering the drum, along with a tiny third spring). The two pivot pins are pretty inaccessible, but I managed to pull out both using a pointy tong. One pin is made of metal, if you need more space you can drill a small hole right next to it. The other pin is made of plastic with a small hole in it, in one case I succeeded by screwing in a little screw and pulling at it. Reassembling the parts is not so hard. I put the "new" cartridge into the printer, waited for the initialization to finish, only to find out the printer would still think the cartridge was empty and should be replaced. Duh. I then discovered a small chip attached to one corner of the drum unit, probably carrying product ID and some serial number. Luckily, it was easy to be removed and exchanged with the other drum unit. Still, the printer thought the cartridge was empty. Some (windows) driver re-installations later it was clear that there is no "ignore fill level and print anyway" option. (Cups didn't print either.) The HP Support Forum has a long Ink cartridge empty but it's not thread with lots of complaints but few answers. (Their problem is mostly with new HP cartridges becoming "empty" after a few days.) Someone suggests trying to cold reset the printer. For the 2605dn, that's holding the left and right buttons while powering on. I removed the cartridges for doing so. When I had put them back in, the printer asked me to confirm that non-HP material was installed, and subsequently didn't display any fill level for the black cartridge anymore, just " ? " in the display. Printing works now. Yay. (Lesson learned: I still hate dealing with printers.)

John Goerzen: rdiff-backup, ZFS, and rsync scripts

rdiff-backup vs. ZFS As I ve been writing about backups, I ve gone ahead and run some tests with rdiff-backup. I have been using rdiff-backup personally for many years now probably since 2002, when I packaged it up for Debian. It s a nice, stable system, but I always like to look at other options for things every so often. rdiff-backup stores an uncompressed current mirror of the filesystem, similar to rsync. History is achieved by the use of compressed backwards binary deltas generated by rdiff (using the rsync algorithm). So, you can restore the current copy very easily a simple cp will do if you don t need to preserve permissions. rdiff-backup restores previous copies by applying all necessary binary deltas to generate the previous version. Things I like about rdiff-backup:
  1. Bandwidth-efficient
  2. Reasonably space-efficient, especially where history is concerned
  3. Easily scriptable and nice CLI
  4. Unlike tools such as duplicity, there is no need to periodically run full backups old backups can be deleted without impacting the ability to restore more current backups
Things I don t like about it:
  1. Speed. It can be really slow. Deleting 3 months worth of old history takes hours. It has to unlink vast numbers of files and that s pretty much it, but it does it really slowly. Restores, backups, etc. are all slow as well. Even just getting a list of your increment sizes so you d know how much space would be saved can take a very long time.
  2. The current backup copy is stored without any kind of compression, which is not at all space-efficient
  3. It creates vast numbers of little files that take forever to delete or summarize
So I thought I would examine how efficient ZFS would be. I wrote a script that would replay the rdiff-backup history first it would rsync the current copy onto the ZFS filesystem and make a ZFS snapshot. Then each previous version was processed by my script (rdiff-backup s files are sufficiently standard that a shell script can process them), and a ZFS snapshot created after each. This lets me directly compare the space used by rdiff-backup to that used by ZFS using actual history. I enabled gzip-3 compression and block dedup in ZFS. My backups were nearly 1TB in size and the amount of space I had available for ZFS was roughly 600GB, so I couldn t test all of them. As it happened, I tested the ones that were the worst-case scenario for ZFS: my photos, music collection, etc. These files had very little duplication and very little compressibility. Plus a backup of my regular server that was reasonably compressible. The total size of the data backed up with rdiff-backup was 583 GB. With ZFS, this came to 498GB. My dedup ratio on this was only 1.05 (meaning 5% or 25GB saved). The compression ratio was 1.12 (60GB saved). The combined ratio was 1.17 (85GB saved). Interestingly 498 + 85 = 583. Remember that the data under test here was mostly a worst-case scenario for ZFS. It would probably have done better had I had the time to throw the rest of my dataset at it (such as the 60GB backup of my iPod, which would have mostly deduplicated with the backup of my music server). One problem with ZFS is that dedup is very memory-hungry. This is common knowledge and it is advertised that you need to use roughly 2GB of RAM per TB of disk when using dedup. I don t have quite that much to dedicate to it, so ZFS got VERY slow and thrashed the disk a lot after the ARC grew to about 300MB. I found some tweakables in zfsrc and the zfs command that let me tweak the ARC cache to grow bigger. But the machine in question only has 2GB RAM, and is doing lots of other things as well, so this barely improved anything. Note that this dedup RAM requirement is not out of line with what is expected from these sorts of solutions. Even if I got absolutely stellar dedup ratio of 2:1, that would get me at most 1TB. The cost of buying a 1TB disk is less than the cost of upgrading my system to 4GB RAM, so dedup isn t worth it here. I think the lesson is: think carefully about where dedup makes sense. If you re storing a bunch of nearly-identical virtual machine images the sort of canonical use case for this go for it. A general fileserver well, maybe you should just add more disk instead of more RAM. Then that raises the question: if I don t need dedup from ZFS, do I bother with it at all, or just use ext4 and LVM snapshots? I think ZFS still makes sense, given its built-in support for compression and very fast snapshots LVM snapshots are known to cause serious degradation to write performance once enabled, which ZFS doesn t. So I plan to switch my backups to use ZFS. A few observations on this:
  1. Some testing suggests that the time to delete a few months of old snapshots will be a minute or two with ZFS compared to hours with rdiff-backup.
  2. ZFS has shown itself to be more space-efficient than rdiff-backup, even without dedup enabled.
  3. There are clear performance and convenience wins with ZFS.
    4. Backup Scripts So now comes the question of backup scripts. rsync is obviously a pretty nice choice here and if used with inplace perhaps even will play friendly with ZFS snapshots even if dedup is off. But let s say I m backing up a few machines at home, or perhaps dozens at work. There is a need to automate all of this. Specifically, there s a need to:
    1. Provide scheduling, making sure that we don t hammer the server with 30 clients all at once
    2. Provide for run before jobs to do things like snapshot databases
    3. Be silent on success and scream loudly via emails to administrators on any kind of error and keep backing up other systems when there is an error
    4. Create snapshots and provide an automated way to remove old snapshots (or mount them for reading, as ZFS-fuse doesn t support the .zfs snapshot directory yet)
    To date I haven t found anything that looks suitable. I found a shell script system called rsbackup that does a large part of this, but something about using a script whose homepage is a forum makes me less than 100% confident. On the securing the backups front, rsync comes with a good-looking rrsync script (inexplicably installed under /usr/share/doc/rsync/scripts instead of /usr/bin on Debian) that can help secure the SSH authorization. GNU rush also looks like a useful restricted shell.

24 January 2011

Raphaël Hertzog: Debian is eating its own dog food more than ever

In its Fast distributions and slow servers article, Joe Zonker Brockmeier explains the difficulties that Fedora has to use its own distribution on their infrastructure and echoes some questions they faced:
Why wouldn t a Linux distribution project wish to showcase the distribution by hosting its infrastructure using its own system?
I completely share the underlying assumption. Eating its own dog food is very important if you want to build a Linux distribution and claim with some confidence that it s of quality and usable. Debian does quite well nowadays in that respect. There were times where the mailing list server was using Qmail (non-free at that time, and thus not part of Debian) but that s long gone. We have also seen our build infrastructure relying on software that was not public and not packaged in Debian, but that also is history. The Debian System Administration team (DSA) maintains more than 140 servers running Debian. They mostly run the stable version of Debian but a few test machines are already running Squeeze since a few months (qa.debian.org for example). The Debian admins want to ensure that the next stable release of Debian still has everything they need and that the software they use still work as expected. Big kudos to the DSA team for this choice! I hope we ll be able to continue to live up to those standards for a long time to come. PS: If you want to learn more about the setup that the DSA team uses, head to dsa.debian.org. You ll find all their repositories and some of their internal documentation.
Subscribe to this blog by RSS, by email or on Facebook.

12 comments Liked this article? Click here. My blog is Flattr-enabled.

23 January 2011

Russell Coker: Autism, Food, etc

James Purser wrote Stop Using Autism to Push Your Own Brand of Lunacy about the organisations that use Autism for their own political agenda [1]. He references the ongoing vaccine issue (which didn t even stop when Andrew Wakefield was proven to have engaged in fraudulent research) and a campaign of PETA.PETA s ClaimsPETA claims that many autistic kids improve dramatically when put on a diet free of dairy foods [2]. They also speculate about the possibility of Autism being caused by dairy.One thing that confuses the issue is the impact of stress on people with Autism Spectrum Disorders (ASDs). Whatever you are trying to do you will probably do it less effectively if under an excessive amount of stress, and the background stress levels of people on the Autism Spectrum tend to be quite high at the best of times. So add in some extra stress from a food allergy and the ability to emulate an NT can decrease a lot.There are people who make credible claims about significant behavior improvements in Autistic children after changing their diet. Eliminate foods that trigger a minor allergic response and the behavior of NT kids will improve too.I m sure that there are some Autistic kids who have a minor intolerance to lactose or casein who s behavior would improve if they had no dairy products. There are probably just as many Autistic kids who have a minor allergy to peanuts who would get the same benefit from avoiding peanut butter. It s not inconceivable that some Autistic people may be allergic to soy or wheat and find that they can cope better if their consume more milk, cheese, and steak and less vegetables!Also some people on the Autism Spectrum have a better sense of smell than NTs, and it s fairly common to be more fussy about the appearance of food. Force children to eat food that smells bad and their behavior might deteriorate.The issues of diet and Autism are more difficult than those of vaccines because while people just imagine the vaccines to have a bad affect, there are people who are proven to get a bad affect from certain foods.Some Ideas for ResearchI think that the biggest problem with PETA s research in this regard is citing a study of only 20 individuals and only testing the removal of milk. A proper study would involve more subjects (maybe 100) and separate tests of removing milk, eggs, wheat, rice, seeds, nuts, soy, fish, spices, and food colorings and additives.It would also test various methods of reducing stress, having a more quiet environment etc.PETA do deserve some credit for leaving this URL in the comments [3], it s a study that disproves the link between milk and Autism. But it s still a small sample size of 14.Why They Target AutismIn spite of psychologists moving to define Asperger Syndrome and Autism as the same thing, the people who make such claims aren t talking about people like me. They want to make claims about people who can t refute them, such as people who fit the Rain Man stereotype. The Low Functioning Autistic (LFA) people have less ability to communicate and are easier to ignore, so if you want to make some stuff up about a group of people then they are an easy target.Parents of LFA kids are often desperate and will latch on to any idea that seems to help. They can imagine that things have improved in spite of a lack of evidence and also fail to realise that children can improve rapidly just due to their natural development, 1 month is not much time for an adult but for a 4yo it s 2% of their life!Also any supposed research that includes testimony of parents should be regarded as rubbish. If you talk to any child psychologist they will tell you about how difficult it is to convince parents that their child is on the Autism Spectrum, a Google search for autism + denial also brings up plenty of hits. Parental denial doesn t stop at refusing to accept a diagnosis, it includes wanting to reject it at any later time based on any shred of evidence that may be available. So when there is a diet change or some other supposed cause followed by some apparently better behavior they want to claim that Autism is cured.I ve written a document titled Some Suggestions for Parents of Aspies [4] which is largely based on the childhood memories and adult experiences of various Aspies. My document covers many issues and few kids on the Spectrum will benefit from all of them, but I think that there is something there that can benefit every child on the Spectrum and probably a lot of geeky NT kids too. It seems to me that when dealing with Autistic kids it s a lot better to seek advice from adults on the Autism Spectrum than from random Neuro-Typical people (NTs), particularly if those NTs happen to have an agenda to push. The sensible parents are the ones who seek advice from psychologists and people like me rather than from random unqualified NTs.

22 January 2011

Matthew Palmer: Debugging Debian Installer

I've been doing a bit of work recently on the netcfg portion of Debian Installer, with the largest work so far being adding IPv6 support. Unlike most of the rest of d-i, netcfg is written in C, and as such suffers from some difficulties in the realm of debugging. Worse, the whole d-i environment is rather bereft of the usual debugging tools (strace, lsof, etc) -- but most importantly, in this case, gdb. Thanks to some hints from Julien Cristau, I was reminded of the existence of gdbserver, a small part of gdb that allows you to debug a program running on a remote machine, using a copy of gdb running on your local machine. It's designed for debugging embedded systems, I think, because all of the heavy lifting (symbol lookups, etc) happens on your local machine, not where gdbserver is running. Here are the steps I took to get a useful gdbserver session inside d-i:
  1. Get the gdb sources (apt-get source gdb) and drop into the gdbserver tree (cd gdb-x.y.z/gdb/gdbserver)
  2. Configure gdbserver to build a static binary (LDFLAGS=-static ./configure). This is necessary because you don't want to have to faff around with library dependencies inside the d-i environment. Then build gdbserver (make).
  3. Copy gdbserver over to your d-i machine/VM (I like using netcat; nc -l -p 31310 >/bin/gdbserver on the d-i machine, and nc IPADDR 31310 <gdbserver where you built gdbserver), then make it executable (chmod /bin/gdbserver)
  4. Fire up gdbserver in d-i, attaching it to the process you wish to debug: gdbserver host:31337 --attach PID
  5. Now things get a little trickier... on your local machine, you need to have an unstripped copy of the binary you're debugging. For me, this is easy, because I'm building netcfg myself (to keep the symbols, I just run DEB_BUILD_OPTIONS=noopt dpkg-buildpackage -b -rfakeroot) but I'm not sure what you'd do if you were just tracing around for fun.
  6. Once you've got your binary, just run gdb BINARY on your host machine to start gdb running.
  7. For the "even trickier" part of the show, your local machine also needs copies of all the libraries your binary is linked against. If you don't have this, when you try to debug you'll get all sorts of "wrong library or version mismatch?" errors, and your backtraces will likely break (amongst other things). Again, for me, this is easy, because I'm only running the d-i initrd, but if you're debugging something later in the installation process you might need to unpack some udebs somewhere. All I have to do is keep the initrd build tree around (which lives in the installer tree, under build/tmp/netboot/tree). Once you've got your tree of libraries, you can continue.
  8. You need to tell gdb about that tree of libraries, by running the following in gdb: set solib-absolute-prefix /path/to/library/tree
  9. Now, tell gdb to connect to your gdbserver: target remote IPADDR:31337 -- gdb should tell you "Remote debugging using IPADDR:31337", and the gdbserver should say "Remote debugging from host LOCALIP:31337".
  10. Now, you should be able to interact with the program being debugged just like any other. Enjoy!

Joey Hess: command composition

Traditionally, Unix has operated by piping between commands ("find grep sort uniq"). I wonder if we are experiencing a shift to command composition being a more common model? I started thinking about this when I considered the following command line: 
chronic mr run git annex get
Unpacking, that uses chronic, which arranges for quiet running of mr (unless it fails, then it shows the full output), which in turn runs git in every repository it knows about, and git is asked to run the git-annex subcommand, which in turn has a "get" subcommand of its own. So, in other words, this is a cron job that syncs large files in a set of git repos, quietly. You might want to make it run niced also, which adds 3 more commands: 
chronic nice ionice -c3 trickle -d100 mr run git annex get
This is the most extreme example I know of (unless you tack in a "sudo"), but there does seem to have been a gradual shift to using more composition, and more subcommands. In git in particular, two-levels of subcommands is not unknown, for example "git remote add". Subcommands, especially as implemented in git (or earlier, in arch), are kind of a special case of command composition that adds a namespace. I'm sure that a count of commands with subcommands would show it exploded in the past 10-12 years. I'll bet a count of composable commands would also show a significant increase. Anyone have other examples of long compositions of commands?

Russell Coker: Continuously Usable Testing of SE Linux

Joey has proposed a new concept of Continuously Usable Testing for Debian [1], basically testing should be usable at all times and packages that aren t usable should be dropped. But to properly achieve this goal we need continual testing of usability.The Plan For SE LinuxTo do this for SE Linux I m setting up a Xen server which will have a number of different DomUs for testing a variety of server applications. The system has 1.5G of RAM and 160G of mirrored storage. An image of a typical server will take about 4G of disk space, so we could have something like 40 images online and ready for testing. I have already setup Squid on another system on the same LAN to cache Debian packages, so running apt-get dist-upgrade on a number of DomUs won t take that long. With 256M for a typical server image I could have 5 images running at the same time. If the hardware isn t enough then I can expand it, I hope to get some donations of DDR-266 or DDR-333 RAM (or maybe DDR-400) to upgrade the system to 4G of RAM, I can add more hard drives, and I could even install more servers.I want to have testing be very usable for SE Linux throughout the development cycle so that I don t have to rush things before release.At this stage I m not sure whether to track Unstable or Testing for this. I guess it might be best to track Testing most of the time and only track Unstable for daemons that are changing rapidly. It might get boring testing every version that comes through Unstable, but if people want to do this then I won t stop them.Setting up the TestsWhat I need are interested people who want to install server configurations for testing. If you have some favorite combination of daemons that you want tested for SE Linux support (even if it s daemons that have no current policy) then I can give you root access to a DomU to develop test cases. Ideally there would be automated tests used for most things for example testing a mail server by using swaks to deliver mail and a POP or IMAP client script to retrieve it. But some things can t be tested properly without human intervention.For the automated tests I want to script the creation of DomUs, upgrading the packages in the DomU, testing it, and then shutting down the DomU if it all works. If at any time the tests fail (or the upgrade fails) then it would wait for human intervention. That would be me fixing SE Linux problems and other people fixing the application problems. I think that discovering SE Linux issues will only be a part of this project.For the manual tests I will grant access to create and destroy the DomUs in question to people who can run the tests.I m thinking of having a couple of DomUs running permanently for things which are test candidates but also useful for the project, such as a MediaWiki instance. It really depends on the interest of people who might use such things.Also I m thinking of setting up some Ubuntu DomUs too, I probably should join Ubuntu and get involved with SE Linux there.Sharing the ImagesI have a web server in Germany with almost unlimited bandwidth and storage. For every image that is created I want to upload a version to the server in Germany to allow anyone in the world to test it. There are lots of possible ways of using this for software development. For example if you had a patched version of Apache that you wanted to test then you could download every image that had Apache installed and test that they all work. That would be easier than configuring Apache in different ways and also possibly provide better coverage.Also if someone can t figure out how to configure a daemon correctly then downloading a Xen image of a working configuration could be helpful (if a little bandwidth intensive). Note that deploying such an image in production would be a really bad idea, among other things there are lots of places where passwords are stored and you wouldn t want to risk missing one.I also plan to share the scripts used in running the Dom0 and anything else that seems useful along the way.What We NeedThe main thing we need is volunteers to configure virtual machines with their favorite daemons. Note that I don t plan to have only one daemon per DomU, if we can get multiple daemons running that don t conflict (EG file server and mail server) or multiple daemons that can interact (EG database server and a mail server or anything else that can be a database client) running on the same system then that s a good thing. So there will be some degree of interaction with other people.I m happy to accept contributions from people who aren t interested in SE Linux. But SE Linux will run on all DomUs.Finally I also need more RAM for a HP D530S, DU875PA (that s a Celeron 2.4GHz). I ll accept donations of complete systems too once my HP system gets full, preferably relatively low power systems as they will be housed in a location that s not as well ventilated as I would like (cost and availability of IP addresses were the main criteria). A laptop with a broken screen would be great!The system won t go live until Monday, but I think that probably people won t be ready to do much work with less than two days notice anyway.

Enrico Zini: Match package names across distributions

Match package names across distributions What would happen if we had a quick and reliable way to match package names across distributions? These ideas came up at the appinstaller2011 meeting: We thought they were good ideas, so we started hacking. To try it, you need to get the code and build the index first: 
git clone git://git.debian.org/users/enrico/distromatch.git
cd distromatch
# Careful: 90Mb
wget http://people.debian.org/~enrico/dist-info.tar.gz
tar zxf dist-info.tar.gz
# Takes a long time to do the indexing
./distromatch --reindex --verbose
Then you can query it this way: 
./distromatch $DISTRO $PKGNAME [$PKGNAME1 ...]
This would give you, for the package $PKGNAME in $DISTRO, the corresponding package names in all other distros for which we have data. If you do not provide package names, it automatically shows output for all packages in $DISTRO. For example: 
$ time ./distromatch debian libdigest-sha1-perl
debian:libdigest-sha1-perl fedora:perl-Digest-SHA1
debian:libdigest-sha1-perl mandriva:perl-Digest-SHA1
debian:libdigest-sha1-perl suse:perl-Digest-SHA1
real    0m0.073s
user    0m0.056s
sys 0m0.016s
Yes it's quick. It builds a Xapian index with the information it needs, and then it reuses it. As soon as I find a moment, I intend to deploy an instance of it on DDE. It is using a range of different heuristics: This list may get obsolete soon as more heuristics get implemented. Euristics will never cover all corner cases we surely have, but the idea is that if we can match a sizable amout of packages, the rest can be somehow fixed by hand as needed. The data it requires for a distribution should be rather straightforward to generate:
  1. a file which maps binary package names to source package names
  2. a file with the list of files in all the packages
For example: 
$ ls -l dist-debian/
total 39688
-rw-r--r--  1 enrico enrico  1688249 Jan 20 17:37 binsrc
drwxr-xr-x  2 enrico enrico     4096 Jan 21 19:12 db
-rw-r--r--  1 enrico enrico 29960406 Jan 21 10:02 files.gz
-rw-r--r--  1 enrico enrico  8914771 Jan 21 18:39 interesting-files
$ head dist-debian/binsrc 
openoffice.org-dev openoffice.org
ext4-modules-2.6.32-5-4kc-malta-di linux-kernel-di-mipsel-2.6
linux-headers-2.6.30-2-common linux-2.6
libnspr4 nspr
ipfm ipfm
libforks-perl libforks-perl
med-physics debian-med
libntfs-3g-dev ntfs-3g
libguppi16 guppi
selinux selinux
$ zcat dist-debian/files.gz   head
memstat etc/memstat.conf
memstat usr/bin/memstat
memstat usr/share/doc/memstat/changelog.gz
memstat usr/share/doc/memstat/copyright
memstat usr/share/doc/memstat/memstat-tutorial.txt.gz
memstat usr/share/man/man1/memstat.1.gz
libdirectfb-dev usr/bin/directfb-config
libdirectfb-dev usr/bin/directfb-csource
libdirectfb-dev usr/include/directfb-internal/core/clipboard.h
libdirectfb-dev usr/include/directfb-internal/core/colorhash.h
interesting-files and db are generated when indexing. To prove the usefulness of the idea (but does it need proving?), you can find in the same git repo a little example app (it took me 10 minutes to write it), that uses the distromatch engine to export Debtags tags to other distributions: 
$ ./exportdebtags fedora   head
memstat: admin::benchmarking, interface::commandline, role::program, use::monitor
libdirectfb-dev: devel::lang:c, devel::library, implemented-in::c, interface::framebuffer, role::devel-lib
libkonqsidebarplugin4a: implemented-in::c++, role::shared-lib, suite::kde, uitoolkit::qt
libemail-simple-perl: devel::lang:perl, devel::library, implemented-in::perl, role::devel-lib, role::shared-lib, works-with::mail
libpoe-component-pluggable-perl: devel::lang:perl, devel::library, implemented-in::perl, role::shared-lib
manpages-ja: culture::japanese, made-of::man, role::documentation
libhippocanvas-dev: devel::library, qa::low-popcon, role::devel-lib
libexpat-ocaml-dev: devel::lang:ocaml, devel::library, implemented-in::c, implemented-in::ocaml, role::devel-lib, works-with-format::xml
libgnutls-dev: devel::library, role::devel-lib, suite::gnu
Just in case this made you itch to play with Debtags in a non-Debian distribution, I've generated the full datasets for Fedora, Mandriva and OpenSUSE. Others have been working on the same matching problem. After we started writing code we started to become aware of existing work: I'd like to make use of those efforts, maybe to cross-validate results, maybe even better as yet another heuristics.

21 January 2011

Rapha&#235;l Hertzog: People behind Debian: Michael Vogt, synaptic and APT developer

Michael and his daughter Marie

Michael has been around for more than 10 years and has always contributed to the APT software family. He s the author of the first real graphical interface to APT synaptic. Since then he created software-center as part of his work for Ubuntu. Being the most experienced APT developer, he s naturally the coordinator of the APT team. Check out what he has to say about APT s possible evolutions. My questions are in bold, the rest is by Michael. Who are you? My name is Michael Vogt, I m married and have two little daughters. We live in Germany (near to Trier) and I work for Canonical as a software developer. I joined Debian as a developer in early 2000 and started to contribute to Ubuntu in 2004. What s your biggest achievement within Debian or Ubuntu? I can not decide on a single one so I will just be a bit verbose. From the very beginning I was interested in improving the package manager experience and the UI on top for our users. I m proud of the work I did with synaptic. It was one of the earliest UIs on top of apt. Because of my work on synaptic I got into apt development as well and fixed bugs there and added new features. I still do most of the uploads here, but nowadays David Kalnischkies is the most active developer. I also wrote a bunch of tools like gdebi, update-notifier, update-manager, unattended-upgrade and software-properties to make the update/install situation for the user easier to deal with. Most of the tools are written in python so I added a lot of improvements to python-apt along the way, including the initial high level apt interface and a bunch of missing low-level apt_pkg features. Julian Andres Klode made a big push in this area recently and thanks to his effort the bindings are fully complete now and have good documentation. My most recent project is software-center. Its aim is to provide a UI strongly targeted for end-users. The goal of this project is to make finding and installing software easy and beautiful. We have a fantastic collection of software to offer and software-center tries to present it well (including screenshots, instant search results and soon ratings&reviews). This builds on great foundations like aptdaemon by Sebastian Heinlein, screenshots.debian.net by Christoph Haas, ddtp.debian.org by Michael Bramer, apt-xapian-index by Enrico Zini and many others (this is what I love about free software, it usually adds , rarely takes away ). What are your plans for Debian Wheezy? For apt I would love to see a more plugable architecture for the acquire system. It would be nice to be able to make apt-get update (and the frontends that use this from libapt) be able to download additional data (like debtags or additional index file that contains more end-user targeted information). I also want to add some scripts so that apt (optionally) creates btrfs snapshots on upgrade and provide some easy way to rollback in case of problems. There is also some interesting work going on around making the apt problem resolver a more plugable part. This way we should be able to do much faster development. software-center will get ratings&reviews in the upstream branch, I really hope we can get that into Wheezy. If you could spend all your time on Debian, what would you work on? In that case I would start with a refactor of apt to make it more robust about ABI breaks. It would be possible to move much faster once this problem is solved (its not even hard, it just need to be done). Then I would add a more complete testsuite. Another important problem to tackle is to make maintainer scripts more declarative. I triaged a lot of upgrade bug reports (mostly in ubuntu though) and a lot of them are caused by maintainer script failures. Worse is that depending on the error its really hard for the user to solve the problem. There is also a lot of code duplication. Having a central place that contains well tested code to do these jobs would be more robust. Triggers help us a lot here already, but I think there is still more room for improvement. What s the biggest problem of Debian? That s a hard question :) I mostly like Debian the way it is. What frustrated me in the past were flamewars that could have been avoided. To me being respectful to each other is important, I don t like flames and insults because I like solving problems and fighting like this rarely helps that. The other attitude I don t like is to blame people and complain instead of trying to help and be positive (the difference between it sucks because it does not support $foo instead of it would be so helpful if we had $foo because it enables me to let me do $bar ). For a long time, I had the feeling you were mostly alone working on APT and were just ensuring that it keeps working. Did you also had this feeling and are things better nowadays ? I felt a bit alone sometimes :) That being said, there were great people like Eugene V. Lyubimkin and Otavio Salvador during my time who did do a lot of good work (especially at release crunch times) and helped me with the maintenance (but got interested in other area than apt later). And now we have the unstoppable David Kalnischkies and Julian Andres Klode. Apt is too big for a single person, so I m very happy that especially David is doing superb work on the day-to-day tasks and fixes (plus big project like multiarch and the important but not very thankful testsuite work). We talk about apt stuff almost daily, doing code reviews and discuss bugs. This makes the development process much more fun and healthy. Julian Andres Klode is doing interesting work around making the resolver more plugable and Christian Perrier is as tireless as always when it comes to the translations merging. I did a quick grep over the bzr log output (including all branch merges) and count around ~4300 total commits (including all revisions of branches merged). Of that there ~950 commits from me plus an additional ~500 merges. It was more than just ensuring that it keeps working but I can see where this feeling comes from as I was never very verbose. Apt also was never my only project, I am involved in other upstream work like synaptic or update-manager or python-apt etc). This naturally reduced the time available to hack on apt and spend time doing the important day-to-day bug triage, response to mailing list messages etc. One the python-apt side Julian Andres Klode did great work to improve the code and the documentation. It s a really nice interface and if you need to do anything related to packages and love python I encourage you to try it. Its as simple as: 
import apt
cache = apt.Cache()
cache["update-manager"].mark_install()
cache.commit()
Of course you can do much more with it (update-manager, software-center and lots of more tools use it). With pydoc apt you can get a good overview. The apt team always welcomes contributors. We have a mailing list and a irc channel and it s a great opportunity to solve real world problems. It does not matter if you want to help triage bugs or write documentation or write code, we welcome all contributors. You re also an Ubuntu developer employed by Canonical. Are you satisfied with the level of cooperation between both projects? What can we do to get Ubuntu to package new applications developed by Canonical directly in Debian? Again a tricky question :) When it comes to cooperation there is always room for improvement. I think (with my Canonical hat on) we do a lot better than we did in the past. And it s great to see the current DPL coming to Ubuntu events and talking about ways to improve the collaboration. One area that I feel that Debian would benefit is to be more positive about NMUs and shared source repositories (collab-maint and LowThresholdNmu are good steps here). The lower the cost is to push a patch/fix (e.g. via direct commit or upload) the more there will be. When it comes to getting packages into Debian I think the best solution is to have a person in Debian as a point of contact to help with that. Usually the amount of work is pretty small as the software will have a debian/* dir already with useful stuff in it. But it helps me a lot to have someone doing the Debian uploads, responding to the bugmail etc (even if the bugmail is just forwarded as upstream bugreports :) IMO it is a great opportunity especially for new packagers as they will not have to do a lot of packaging work to get those apps into Debian. This model works very well for me for e.g. gdebi (where Luca Falavigna is really helpful on the Debian side). Is there someone in Debian that you admire for his contributions? There are many people I admire. Probably too many to mention them all. I always find it hard to single out individual people because the project as a whole can be so proud of their achievements. The first name that comes to my mind is Jason Gunthorpe (the original apt author) who I ve never met. The next is Daniel Burrows who I met and was inspired by. David Kalnischkies is doing great work on apt. From contributing his first (small) patch to being able to virtually fix any problem and adding big features like multiarch support in about a year. Sebastian Heinlein for aptdaemon. Christian Perrier has always be one of my heroes because he cares so much about i18n. Christoph Haas for screenshots.debian.net, Michael Bramer for his work on debian translated package descriptions.
Thank you to Michael for the time spent answering my questions. I hope you enjoyed reading his answers as I did. Subscribe to my newsletter to get my monthly summary of the Debian/Ubuntu news and to not miss further interviews. You can also follow along on Identi.ca, Twitter and Facebook.

4 comments Liked this article? Click here. My blog is Flattr-enabled.

Cyril Brulebois: Debian XSF News #2

Time for a second Debian XSF News issue! I ll skip stuff related to the new katamari since I wrote about that already.

19 January 2011

Rapha&#235;l Hertzog: What I publish on Planet Debian and Planet Ubuntu

Up to now, Planet Debian and Planet Ubuntu were syndicating the main feed of my blog. I changed this recently (remember, I explained how I created my custom WordPress feed). Read on to learn why and subscribe to the main feed now if you want to get all my Debian/Ubuntu articles in the future. Planet Debian/Ubuntu are supposed to be windows into the lives of the contributors. Whatever they are doing. Obviously you get a lot of infos about Debian and Ubuntu since most contributors spend a lot of time working on those distributions. Due to this, those planets have also become dedicated medias to exchange between contributors. The problem I had is that my blog is more than just a place where I speak of what I do within Debian. As part of my evil plan, I set out my blog to become a great documentation resource for Debian/Ubuntu users. So when I wrote articles for users and/or beginners (like 5 reasons why a Debian package is more than a simple file archive or How to rebuild Debian packages), I got some remarks that it was odd to have those articles on Planet Debian because most contributors knew those things already. The bottom line is that from now on I ll decide on a case by case whether the article is suitable for Planet Debian&Ubuntu based on my feelings (and the feedback I get by email). But if you enjoyed my articles for users, I invite you to directly subscribe to my blog to make sure you ll get them all in the future: By the way I would like to thank you all. This blog has been created last July and I m glad to have already more than 1400 regular followers (spread over RSS/email/Twitter/Identi.ca/Facebook, not counting the thousands that read me via Planet Debian/Ubuntu). I hope you ll enjoy 2011 with me, and continue sharing the articles you like.

6 comments Liked this article? Click here. My blog is Flattr-enabled.

18 January 2011

Eddy Petri&#537;or: Back to coding and similar stuff soon

There has been a quite long hiatus in my Debian activity, first due to my involvement in OpenStreeMap and later due to some personal reasons. This has left the packages I was responsible for basically without a maintainer, even in the case of the game packages which should have been taken care by the Debian Games Team. Sadly, it seems the principle there are other people which can get involved was at work in this case.

The good news is that I foresee a period when I will be able to get involved again in Debian work, although I don't know yet how long will it take to start and how long will it last.

The first thing on the agenda should be the Womux - Warmux rename and packaging the latest version of the game.

And yes, currently I am totally clueless about the state of squeeze release and the RC bug count. If I get fast enough back in the Debian work wagon, I'll probably try to help fix a couple of RC bugs here and there.

Russell Coker: Conferences and People on the Autism Spectrum

There have been some recent discussions about issues related to the treatment of women at Free Software conferences, I ve written posts about Aspie Social Skills and Free Software [1] and Empathy, Autism, and Geeks [2] about this. But of course discussion continued on how Aspies supposedly cause problems that no-one seems to be noticing.Lisa of Chaotic Idealism wrote an interesting post about the NT social bias [3]. In summary NTs seem to think that Autism Spectrum Disorders are only about socialisation, not realising that for many people on the Spectrum it s sensory issues that are their main problem. Social problems are exacerbated by sensory issues and other causes of stress, so I think it s worth considering ways in which conferences can be planned to be less stressful for people on the Autism Spectrum, people who have SPD [4] that isn t associated with an ASD, and NTs who just get annoyed by loud noises etc.What Autism Conferences DoAutscape is one of only two conferences for people on the Autism Spectrum and the only one which clearly documents how they plan their conference [5]. The first noteworthy thing that they do is have badge colors to indicate what level of social interaction is desired by each delegate, I don t think that this is relevant to Free Software conferences as people who don t have sufficient social skills to suit at least a green badge probably won t be attending. But I think that when attending a conference about Free Software or any other equally geeky topic (if there is an equally geeky topic) it s worth keeping in mind the fact that there are probably a lot of people who would like to talk to you but lack the social skills to start a conversation.
No initiationRedPlease do not initiate any interaction with me.
Prior PermissionYellowPlease do not initiate unless I have already given you permission to approach me on a yellow badge.
Please initiateGreenI would like to socialise, but I have difficulty initiating. Please initiate with me.
NeutralWhite (or no badge)I am able to regulate my own interaction.
The next thing that they document is a black circle badge which indicates that the wearer shouldn t be photographed. Prior to reading that web site I wasn t aware of this being an Autistic issue, I was only aware of it being an issue for women who don t want zoomed-in pictures of themselves appearing on guys web sites (even pictures that aren t up-skirt or down-blouse can be unwelcome). A conference policy that prohibited photographs that zoom in on one person without that person s consent (or parental consent in the case of minors) would probably be a good idea.Another thing about badges is that it s a really good idea to have the delegate s name on both sides of the badge if the badge is attached in a way that permits it to turn. People on the Autism Spectrum tend to have some difficulty in recognising people and in remembering names. I find it inconvenient when someone expects me to recognise them but has their badge turned around so I can t see their name. Some people get really unhappy if they think that someone doesn t recognise them.Autscape has long breaks between activities and a leisure session each afternoon to allow delegates to recover from the stress of dealing with people. It seems to me that computer conferences in some cases could do with longer breaks between sessions. I find that a lot of the benefit of a conference is in what happens outside sessions and the standard practice of publishing videos of presentations makes personal meetings a more important part of the conference.Autscape has designated quiet spaces. I think that for computer conferences which have hack-labs (which seems to be most Free Software conferences nowadays even if they aren t labelled as such) it would be good to have some lab areas designated as quiet zones. I think that it is a really good thing to meet people you ve only known by email and then play some LAN based games against them and that this should be encouraged as part of a conference, but having that sort of thing separated from people who want to do some quiet coding is a really good idea. Whether people want to do quiet coding, read email, or just escape from the stress of a conference a quiet hack-lab would be a good place for it.An issue that s related to quiet spaces is the ability to escape from social situations. One of my pet hates is corporate meetings on boats, when the boat is in motion there is no escape. Corporate meetings that are only accessible by coach are also bad. Fortunately most conferences aren t like that.The Autscape web site states that they prohibit people from wearing perfume or aftershave to help people who are sensitive to smell. My observation of Free Software conferences is that encouraging everyone to have a shower every morning would be a good idea as there are stronger smells from unwashed people than from perfume.One thing that s interesting about the Autscape web site is that they have different color schemes available and have tested it in multiple web browsers including Safari and Konqueror (which apparently don t work so well). It s interesting to note that they test with such a variety of browsers including free software ones most corporations don t do that.Sensory Processing IssuesThe main SPD issues related to conferences seem to be noise and light related.For social events one difficulty that most people on the Spectrum seem to face is in listening to one person in a noisy crowded room. As well as that sudden noises and loud background noises can be very stressful. While it s sometimes impossible to avoid crowds (which can be another problem) it is often possible to select venues that have less noise problems. A venue with carpet on the floor and soft walls (anything other than brick or concrete) will be a lot quieter than one with hard surfaces that reflect noise. It seems that a quiet venue will benefit NTs as well, there seems to be a strong correlation between the price of food at restaurants and the amount of sound absorbing material on the floor and walls people who can afford a good dining experience seem to want it quiet.In terms of visual issues the relevant problems seem to be related to sudden transitions and lecture halls that are extremely dark. The only thing that can be done by conference organisers is to seek to have the lights in the lecture halls as bright as possible without preventing the viewing of the projector screen, this is usually done anyway.For social events the whole binge drinking at crowded bars thing doesn t work too well due to noise, poor lighting, crowds, and the smell of vomit. But there s no shortage of reasons to discourage binge drinking at conferences.FoodPsychology Today has a good article about preferences for food and picky eaters [6]. Lots of people on the Autism Spectrum have similar issues. The thing to do when arranging a conference meal is to have things neat and without needless combinations. Think about making food look more like what you would expect to see in a Japanese restaurant and less like Paella. Also having some very plain food on offer is a good thing, I think that bread-rolls for dinner and ice-cream for desert makes a viable meal. But any dish with a word like hash or mixed in it s name isn t a good option. Finally some combinations are really bad, I always find rare steak and potatoes on the same plate to be rather disgusting potato that s blood-stained doesn t appeal. The majority of conference meals satisfy these criteria.Autscape provides options of eating outside the main dining hall for people who can t tolerate the noise. For a Free Software conference it might be a better idea to provide seating outside the main area for people who are going to make noise. I don t mind people who want to get really drunk at the conference dinner, but I would prefer them to be in a separate room. Also when assisting a drunk friend to leave the venue before they cause more problems it would be convenient if there was a good place to take them to. The one occasion when I had to strongly encourage a friend to leave a conference dinner to reduce his embarrassment the next day (and reduce the annoyance for everyone else) there was no good place to take him so he just came back!ConclusionI don t expect that anyone will make any significant changes to conference plans after reading this post. I have tried to focus on things that would benefit NTs as well, but there are practical issues that get in the way such as the cost and availability of venues that are large enough. But there are some smaller ideas that can be implemented with relative ease.I hope that the people who are making claims about Autism and conference behavior will refer to this in future. If you think that there is a problem with the way people on the Spectrum act at conferences then the solution will more likely involve the suggestions I make here than anything else.

17 January 2011

Dirk Eddelbuettel: Keeping simple things simple

My friend Jeff deserves a sincere congratulation for finally unveiling his rebranded R consultancy Lemnica. One notable feature of the new website is a section called esoteric R which discusses less frequently-visited corners of the R world. It even boasts its own CRAN package esotericR with the example sources. esoteric R currently holds two articles. Jeff had sent me the one about introducing closures a while back, and I like it and may comment at another time. What caught me by surprise when Lemnica finally opened was the other article: R calling C. It is a fine article motivated by all the usual reasons that are e.g. mentioned in the Google Tech Talk which Romain and I gave last October about our work around Rcpp. But it is just not simple. Allow me to explain. When Jeff showed this C language file 
#include <R.h>
#include <Rinternals.h>
SEXP esoteric_rev (SEXP x)  
  SEXP res;
  int i, r, P=0;
  PROTECT(res = allocVector(REALSXP, length(x))); P++;
  for(i=length(x), r=0; i>0; i--, r++)  
     REAL(res)[r] = REAL(x)[i-1];
   
  copyMostAttrib(x, res);
  UNPROTECT(P);
  return res;
and then needs several paragraphs to explain what is going on, what is needed to compile and then how to load it --- I simply could not resist. Almost immediately, I emailed back to him something as simple as this using both our Rcpp package as well as the wonderful inline package by Oleg which Romain and I more or less adopted: 
library(inline)  ## for cxxfunction()
src <- 'Rcpp::NumericVector x = Rcpp::NumericVector(xs);
        std::reverse(x.begin(), x.end());
        return(x);'
fun <- cxxfunction(signature(xs="numeric"), body=src, plugin="Rcpp")
fun( seq(0, 1, 0.1) )
Here we load inline, and then define a three-line C++ program using facilities from our Rcpp package. All we need to revert a vector is to first access its R object in C++ by instantiating the R vector as a NumericVector. These C++ classes then provide iterators which are compatible with the Standard Template Library (STL). So we simply call the STL function reverse pointing the beginning and end of the vector, and are done! Rcpp then allows us the return the C++ vector which it turns into an R vector. Efficient in-place reversal, just like Jeff had motivated, in three lines. Best of all, we can execute this from within R itself: 
R> library(inline)  ## for cxxfunction()
R> src <- 'Rcpp::NumericVector x = Rcpp::NumericVector(xs);
+         std::reverse(x.begin(), x.end());
+         return(x);'
R> fun <- cxxfunction(signature(xs="numeric"), body=src, plugin="Rcpp")
R> fun( seq(0, 1, 0.1) )
 [1] 1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0
R>
Lastly, Jeff shows a more complete example wherein a new vector is created, and any potential attributes are copied as well. Naturally, we can do that too. First, we used clone() to make a deep copy (ie forcing creation of a new object rather than a mere proxy) and use the same R API function he accessed---but it our case both prefixed with ::Rf_ for R remapping (to protect clashed with other functions with identical names) and a global namespace identifier (as it is a global C function from R). 
R> library(inline)
R> src <- 'Rcpp::NumericVector x = Rcpp::clone<Rcpp::NumericVector>(xs);
+         std::reverse(x.begin(), x.end());
+         ::Rf_copyMostAttrib(xs, x);
+         return(x);'
R> fun <- cxxfunction(signature(xs="numeric"), body=src, plugin="Rcpp")
R> obj <- structure(seq(0, 1, 0.1), obligatory="hello, world!")
R> fun(obj)
 [1] 1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0
attr(,"obligatory")
[1] "hello, world!"
R> obj
 [1] 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
attr(,"obligatory")
[1] "hello, world!"
R>
Both the obj variable and the new copy contain the desired data attribute, the new copy is reversed, the original is untouched---and all in four lines of C++ called via one inline call. I have now been going on for over one hundred lines yet I never had to mention memory management, pointers, PROTECT or other components of the R API for C. Hopefully, this short writeup provided an idea of why Romain and I think Rcpp is the way to go for creating C/C++ functions for extending and enhancing R.

Rapha&#235;l Hertzog: Review of my Debian related goals for 2010

Last year I shared my Debian related goals for 2010 . I announced that I would not be able to complete them all and indeed I have not, but I have still done more than what I expected. Let s have a look. Translate my Debian book into English and get it published: NOT DONE Or rather not yet done. It s still an important project of mine and I will do it this year. When I wrote this last year, I expected to find a publisher that would take care of everything but we failed to find a suitable one so we re going to do it ourselves. Cleanup the dpkg perl API and create libdpkg-perl: DONE libdpkg-perl has been introduced with dpkg 1.15.6. Create dpkg-buildflags: DONE dpkg-buildflags has been introduced with dpkg 1.15.7. It s not widely used yet and it won t really be until debhelper 7 supports it (see #544844). It would be nice to see progress on this front this year. Ensure the new source formats continue to gain acceptance: DONE The adoption rate has been steady, it clearly slowed down since the freeze though. I have implemented quite a few features to satisfy the needs of users, like the possibility to unapply the patches after the build, or the possibility to fail in case of unwanted upstream changes. Design a generic vcs-buildpackage infrastructure to be integrated in dpkg-dev: NOT DONE I believe it s something important on the long term but it never made it to my short-term TODO list and it s unlikely to change any time soon. Continue fixing dpkg bugs faster than they are reported: PARTLY DONE We have dealt with many bugs over the year, but we still have 20 more bugs than at the start of last year (370 vs 350). We re not doing bad compared to many other Debian teams but we can still benefit from some help. Start here if you re interested. Enhance our infrastructure to ease interaction between contributors and to have a better view of how each package is maintained: NOT DONE I am convinced that we need something to have a clearer idea of the commitments made by each contributor. I don t put the same amount of care in maintaining smarty-gettext that I do on dpkg. If we had a database of the stuff that we know we don t do well enough, it s easier to point new contributors towards those. Anyway, this project is still unlikely to come to the top of my priorities any time soon. Work on the developers-reference: NOT DONE We have switched the Maintainer field to debian-policy@lists.debian.org to have more review of the changes suggested through the bug tracking system but that has not changed much on the global situation. I still hope to become more active on it sometimes this year. Maybe by trying to make it more fun and creating the text for some of the wishlist bugs as blog articles first. Rewrite in C the last Perl scripts provided by the dpkg binary package: DONE Dpkg 1.15.8 was the first version working without Perl, I announced it in July. Integrate the 3-way merge tool for Debian changelogs in dpkg-dev: DONE dpkg-mergechangelogs is part of dpkg-dev since 1.15.7. I enjoy it regularly. Unfortunately it doesn t work well for cherry-picks. Would be nice to see this fixed, anyone up to the task? :-)
Click here to subscribe to my free newsletter and get my monthly analysis on what s going on in Debian and Ubuntu. Or just follow along via the RSS feed, Identi.ca, Twitter or Facebook.

One comment Liked this article? Click here. My blog is Flattr-enabled.

16 January 2011

Dirk Eddelbuettel: Plotting overbought / oversold regions in R

The good folks at Bespoke Investment Group frequently show charts of so-called overbought or oversold levels; see e.g. here for the most recent global markets snapshot. Classifying markets as overbought or oversold is a popular heuristic. It starts from computing a rolling smoothed estimate of the prices, usually via a (exponential or standard) moving average over a suitable number of days (where Bespoke uses 50 days, see here). This is typically coupled with a (simple) rolling standard deviation. Overbought and oversold regions are then constructed by taking the smoothed mean plus/minus one and two standard deviations. Doing this is in R is pretty easy thanks to the combination of R's rich base functions and its add-on packages from CRAN. Below is a simply function I wrote a couple of months ago---and I figured I might as well release. It relies on the powerful packages quantmod and TTR by my pals Jeff Ryan and Josh Ulrich, respectively. 
## plotOBOS -- displaying overbough/oversold as eg in Bespoke's plots
##
## Copyright (C) 2010 - 2011  Dirk Eddelbuettel
##
## This is free software: you can redistribute it and/or modify it
## under the terms of the GNU General Public License as published by
## the Free Software Foundation, either version 2 of the License, or
## (at your option) any later version.
suppressMessages(library(quantmod))     # for getSymbols(), brings in xts too
suppressMessages(library(TTR))          # for various moving averages
plotOBOS <- function(symbol, n=50, type=c("sma", "ema", "zlema"), years=1, blue=TRUE)  
    today <- Sys.Date()
    X <- getSymbols(symbol, src="yahoo", from=format(today-365*years-2*n), auto.assign=FALSE)
    x <- X[,6]                          # use Adjusted
    type <- match.arg(type)
    xd <- switch(type,                  # compute xd as the central location via selected MA smoother
                 sma = SMA(x,n),
                 ema = EMA(x,n),
                 zlema = ZLEMA(x,n))
    xv <- runSD(x, n)                   # compute xv as the rolling volatility
    strt <- paste(format(today-365*years), "::", sep="")
    x  <- x[strt]                       # subset plotting range using xts' nice functionality
    xd <- xd[strt]
    xv <- xv[strt]
    xyd <- xy.coords(.index(xd),xd[,1]) # xy coordinates for direct plot commands
    xyv <- xy.coords(.index(xv),xv[,1])
    n <- length(xyd$x)
    xx <- xyd$x[c(1,1:n,n:1)]           # for polygon(): from first point to last and back
    if (blue)  
        blues5 <- c("#EFF3FF", "#BDD7E7", "#6BAED6", "#3182BD", "#08519C") # cf brewer.pal(5, "Blues")
        fairlylight <- rgb(189/255, 215/255, 231/255, alpha=0.625) # aka blues5[2]
        verylight <- rgb(239/255, 243/255, 255/255, alpha=0.625) # aka blues5[1]
        dark <- rgb(8/255, 81/255, 156/255, alpha=0.625) # aka blues5[5]
      else  
        fairlylight <- rgb(204/255, 204/255, 204/255, alpha=0.5)         # grays with alpha-blending at 50%
        verylight <- rgb(242/255, 242/255, 242/255, alpha=0.5)
        dark <- 'black'
     
    plot(x, ylim=range(range(xd+2*xv, xd-2*xv, na.rm=TRUE)), main=symbol, col=fairlylight) 		# basic xts plot
    polygon(x=xx, y=c(xyd$y[1]+xyv$y[1], xyd$y+2*xyv$y, rev(xyd$y+xyv$y)), border=NA, col=fairlylight) 	# upper
    polygon(x=xx, y=c(xyd$y[1]-1*xyv$y[1], xyd$y+1*xyv$y, rev(xyd$y-1*xyv$y)), border=NA, col=verylight)# center
    polygon(x=xx, y=c(xyd$y[1]-xyv$y[1], xyd$y-2*xyv$y, rev(xyd$y-xyv$y)), border=NA, col=fairlylight) 	# lower
    lines(xd, lwd=2, col=fairlylight)   # central smooted location
    lines(x, lwd=3, col=dark)           # actual price, thicker
    invisible(NULL)
After downloading data and computing the rolling smoothed mean and standard deviation, it really is just a matter of plotting (appropriate) filled polygons. Here I used colors from the neat RColorBrewer package with some alpha blending. Colors can be turned off via an option to the function; ranges, data length and type of smoother can also be picked. To call this in R, simply source the file and the call, say, plotOBOS("^GSPC", years=2) which creates a two-year plot of the SP500 as shown here: Example chart of overbought/oversold levels from plotOBOS() function This shows the market did indeed bounce off the oversold lows nicely on a few occassions in 2009 and 2010 --- but also continued to slide after hitting the condition. Nothing is foolproof, and certainly nothing as simple as this is, so buyer beware. But it may prove useful in conjunction with other tools. The code for the script is here and of course available under GPL 2 or later. I'd be happy to help incorporate it into some other finance package. Lastly, if you read this post this far, also consider our R / Finance conference coming at the end of April.

Edit: Corrected several typos with thanks to Josh.

Daniel Kahn Gillmor: resetting passphrases for mapped LUKS volumes

Let's say you set up a machine using an encrypted disk with LUKS (debian-installer's partman makes this wonderfully easy!). You choose an initial passphrase, get the machine working, and it's working great. Then, you need to restart it, and realize that (for whatever reason) you've forgotten or lost the passphrase for the volume. oops! (i'm sure this has never happened to you -- let's just pretend it's your less-fortunate friend). If your system is still running, and you have superuser access to it, you can actually set a new passphrase for the LUKS volume using information that the dm-crypt kernel module has about the in-use mapping. In my examples, i'll imagine that the source volume is /dev/XXX2 and the exported cleartext volume is known by the device-mapper as XXX2_crypt In the bigger picture, this should serve as a reminder that even though your disk is encrypted, if someone gets live access to the superuser account on a system with the encryption keys loaded, your data is no longer secret. Before you do any tweaking, you might want to back up your LUKS header, just in case: 
0 root@example:~# umask 077
0 root@example:~# cryptsetup luksHeaderBackup /dev/XXX2 --header-backup-file XXX2.luksheader.backup
0 root@example:~#
Maybe also copy that off the machine, since a copy of the LUKS header stored within its own volume isn't terribly useful for a backup-recovery situation. You might also be interested in looking at the contents of the LUKS header: 
0 root@example:~# cryptsetup luksDump /dev/XXX2
LUKS header information for /dev/XXX2
Version:       	1
Cipher name:   	aes
Cipher mode:   	cbc-essiv:sha256
Hash spec:     	sha1
Payload offset:	2056
MK bits:       	256
MK digest:      93 51 6c 66 ec ce 32 54 6f 6b 52 d1 27 9b 5a 62 6f 6b 52 d1
MK salt:       	b2 ca 20 49 9f 78 49 c2 fe 15 b4 0f 74 11 23 49
               	64 9e 61 bb f2 82 60 47 a5 76 fa a4 24 0e 5a 7e
MK iterations: 	10
UUID:          	052f1da0-21a1-11e0-ac64-0800200c9a66
Key Slot 0: ENABLED
	Iterations:         	218733
	Salt:               	f2 ae 8c 53 48 a5 f0 bf e1 2c 06 5f 5a bd ff d9
	                      	9a 2e d1 49 3a 63 f8 49 82 ed ae 86 7b 7b 7e 76 
	Key material offset:	8
	AF stripes:            	4000
Key Slot 1: DISABLED
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: DISABLED
0 root@example:~#
Now, the fix: We pull the live "master key" from the running device map, and fill a new luksKeySlot from it (this example uses bash's <() syntax for process substitution -- if you use a different shell, i'm sure you can find a different way to do it): 
0 root@example:~# cryptsetup --master-key-file <( 
> dmsetup --showkeys table   awk '/^XXX2_crypt: /  print $6  '   tr -d '\n'   perl -e 'print pack("H*", <STDIN>);' 
> ) luksAddKey /dev/XXX2
Enter new passphrase for key slot: abc123
Verify passphrase: abc123
2 root@example:~#
(note that the luksAddKey invocation above returned an error code of 2 even though it succeeded. I think this is a bug in cryptsetup's return code, not a bug in the password resetting -- it should have returned 0 instead of 2). You can check to see that a new key slot was enabled by re-running cryptsetup luksDump And if you really want to double-check before you reboot, you can try enabling a third keyslot using the passphrase you just added, since this would not succeed if your new passphrase failed to unlock any of the existing keyslots: 
0 root@example:~# cryptsetup luksAddKey /dev/XXX2
Enter any passphrase: abc123
Enter new passphrase for key slot: anotherpassphrase
Verify passphrase: anotherpassphrase
0 root@example:~#
You can also get rid of the original keyslot (which you don't know the passphrase to) like this: 
0 root@example:~# cryptsetup luksKillSlot /dev/XXX2 0
Enter any remaining LUKS passphrase: abc123
0 root@example:~#
(the above commands were all demonstrated using debian testing, with cryptsetup 2:1.1.3-4 and dmsetup 2:1.02.48-4) Tags: cryptsetup, dmsetup, luks, tip

Russell Coker: Links January 2011

Halla Tomasdottir gave an interesting TED talk about her financial services company which uses feminine values and thus survived the Icelandic financial crash [1]. She also has a lot of other interesting ideas about how to run a corporation or a country.Hillary Clinton gave an interesting TED talk about national security and women s rights [2]. The US State Department is now making women s rights a priority due to the fact that places where women are mistreated tend to pose a security threat to the US. Unfortunately this talk can t be viewed without flash.Richard Dawkins reads some of his hate-mail while sitting by the fire [3], it s interesting to know that a loving God wants his minions to send hate-mail to scientists. There are a few cheap laughs here.Emily Pilloton gave an interesting TED talk about fixing some of the problems with the education system in rural USA by incorporating design [4]. Instead of having useless projects in shop class the students will complete real projects for their community which have to be designed, planned, and made. The students who end up going to university will learn some generally useful skills through this and the ones who don t go to university will probably have an immediate opportunity to apply the skills that they learn.Elizabeth Pisani gave an interesting TED talk about rational behavior and HIV [5]. She explains why certain behavior that carries great risk can be rational for the people who do it. One of her interesting points is the fact that the poor health system in the US makes it economically viable for the government to refrain from providing free needles to junkies as they don t pay to treat them when they get sick. She makes a good case for political change.Tony Porter gave an inspiring TED talk titled A Call to Men about the way the macho culture socialises men towards violence against women [6].Fraud Magazine has an interesting Interview with Dr. Robert D. Hare and Dr. Paul Babiak about psychopathic fraudsters [7]. Most of the interview is generally about psychopaths and the cases of psychopaths in business, there isn t much that is specific to financial fraud. It has a good summary of what some psychopaths are like.Barbara Fister wrote a Liberation Bibliography a call for knowledge to be free [8]. Currently universities pay researchers who then pay for corporations to accept copyright for their work and then resell it back to the universities. This is sucking money from the university system and denying access to knowledge to the people who can t pay.Charles Limb gave an interesting TED talk about using an MRI to study the brains of Jazz musicians doing Improv and Rappers doing Freestyle [9].Deborah Rhodes gave an interesting TED talk about her new method of screening for breast cancer for women with dense breast tissue [10].Cory Doctorow wrote an interesting article about Net Neutrality and it s importance in helping the less famous writers (the majority of writers) earn money [11]. He also makes some interesting points about Google and about the corporate welfare given to telephone companies.Beyond Zero Emissions has an interesting summary of the current situation regarding Carbon Capture and Sequestration (CCS), among other things the only system that s currently operational is leaking CO2 and killing plants and animals [12].The Brisbane council knew that the flood risks were greater than suggested by published information, they kept this news to themselves to assist property developers in selling low-lying land, and this led to increased problems now that the predicted floods have occurred [13]. Sounds like corruption, we could probably do with a Royal Commission into this.Psychology Today has an interesting article titled Wise Words From Fellow Introverts which is a collection of blog comments from introverts [14].

15 January 2011

Matthew Garrett: Joojoo, once more

Having complained about the GPL compliance of the Fusion Garage Joojoo tablet in the past, I'm thrilled to say that I spoke to their new CTO earlier in the week and they now have full source availability here[1]. Previous engineering practices involved editing binary Debian packages rather than rebuilding from the Debian source packages, so there may be a small number of cases where it's awkward to rebuild an identical binary, but I think it's pretty clear at this point that they've done everything practical to make up for past mistakes. I've been assured that all further product development will take license compliance into account from the start, so while things may have taken a little longer than I'd have liked I think everything has worked out for the best. I'd like to say thanks to Fusion Garage and Megan Alpers at McGrath Power for their help in getting all of this resolved.

[1] I've taken a brief look at their new downloadable tarballs. If anyone does find anything missing, let me know and I'll be sure to pass it onto the right people.

Next.

Previous.