I ve just installed BTRFS on some systems that matter to me. It is still regarded as experimental but Oracle supports it with their kernel so it can t be too bad and it s almost guaranteed that anything other than BTRFS or ZFS will lose data if you run as many systems as I do and I run lots of systems that don t have enough RAM for ZFS (4G isn t enough in my tests). So I have to use BTRFS. BTRFS and Virtual Machines I m running BTRFS for the DomUs on a virtual server which has 4G of RAM (and thus can t run ZFS). The way I have done this is to use ext4 on Linux software RAID-1 for the root filesystem on the Dom0 and use BTRFS for the rest. For BTRFS and virtual machines there seem to be two good options given that I want BTRFS to use it s own RAID-1 so that it can correct errors from a corrupted disk. One is to use a single BTRFS filesystem with RAID-1 for all the storage and then have each VM use a file on that big BTRFS filesystem for all it s storage. The other option is to have each virtual machine run BTRFS RAID-1. I ve created two LVM Volume Groups (VGs) named diska and diskb, each DomU has a Logical Volume (LV) from each VG and runs BTRFS. So if a disk becomes corrupt the DomU will have to figure out what the problem is and fix it. #!/bin/bash
for n in $(xm list cut -f1 -d\ egrep -v ^Name\ ^Domain-0) ; do
echo $n
ssh $n "btrfs scrub start -B -d /"
done I use the above script in a cron job from the Dom0 to scrub the BTRFS filesystems in the DomUs. I use the -B option so that I will receive email about any errors and so that there won t be multiple DomUs scrubbing at the same time (which would be really bad for performance). BTRFS and Workstations The first workstation installs of BTRFS that I did were similar to installations of Ext3/4 in that I had multiple filesystems on LVM block devices. This caused all the usual problems of filesystem sizes and also significantly hurt performance (sync seems to perform very badly on a BTRFS filesystem and it gets really bad with lots of BTRFS filesystems). BTRFS allows using subvolumes for snapshots and it s designed to handle large filesystems so there s no reason to have more than one filesystem IMHO. It seems to me that the only benefit in using multiple BTRFS filesystems on a system is if you want to use different RAID options. I presume that eventually the BTRFS developers will support different RAID options on a per-subvolume basis (they seem to want to copy all ZFS features). I would like to be able to configure /home to use 3 copies of all data and metadata on a workstation that only has a single disk. Currently I have some workstations using BTRFS with a BTRFS RAID-1 configuration for /home and a regular non-RAID configuration for everything else. But now it seems that this is a bad idea, I would be better off just using a single copy of all data on workstations (as I did for everything on workstations for the previous 15 years of running Linux desktop systems) and make backups that are frequent enough to not have a great risk. BTRFS and Servers One server that I run is primarily used as an NFS server and as a workstation. I have a pair of 3TB SATA disks in a BTRFS RAID-1 configuration mounted as /big and with subvolumes under /big for the various NFS exports. The system also has a 120G Intel SSD for /boot (Ext4) and the root filesystem which is BTRFS and also includes /home. The SSD gives really good read performance which is largely independent of what is done with the disks so booting and workstation use are very fast even when cron jobs are hitting the file server hard. The system has used a RAID-1 array of 1TB SATA disks for all it s storage ever since 1TB disks were big. So moving to a single storage device for /home is a decrease in theoretical reliability (in addition to the fact that a SSD might be less reliable than a traditional disk). The next thing that I am going to do is to install cron jobs that backup the root filesystem to something under /big. The server in question isn t used for anything that requires high uptime, so if the SSD dies entirely and I need to replace it with another boot device then it will be really annoying but it won t be a great problem. Snapshot Backups One of the most important uses of backups is to recover from basic user mistakes such as deleting the wrong file. To deal with this I wrote some scripts to create backups from a cron job. I put the snapshots of a subvolume under a subvolume named backup . A common use is to have everything on the root filesystem, /home as a subvolume, /home/backup as another subvolume, and then subvolumes for backups such as /home/backup/2012-12-17, /home/backup/2012-12-17:00:15, and /home/backup/2012-12-17:00:30. I make /home/backup world readable so every user can access their own backups without involving me, of course this means that if they make a mistake related to security then I would have to help them correct it but I don t expect my users to deal with security issues, if they accidentally grant inappropriate access to their files then I will be the one to notice and correct it. Here is a script I name btrfs-make-snapshot which has an optional first parameter -d to cause it do just display the btrfs commands it would run and not actually do anything. The second parameter is either minutes or days depending on whether you want to create a snapshot on a short interval (I use 15 minutes) or a daily snapshot. All other parameters are paths for subvolumes that are to be backed up: #!/bin/bash
set -e

# usage:
# btrfs-make-snapshot [-d] minutes days paths
# example:
# btrfs-make-snapshot minutes /home /mail

if [ "$1" == "-d" ]; then
BTRFS="echo btrfs"
shift
else
BTRFS=/sbin/btrfs
fi

if [ "$1" == "minutes" ]; then
DATE=$(date +%Y-%m-%d:%H:%M)
else
DATE=$(date +%Y-%m-%d)
fi
shift

for n in $* ; do
$BTRFS subvol snapshot -r $n $n/backup/$DATE
done Here is a script I name btrfs-remove-snapshots which removes old snapshots to free space. It has an optional first parameter -d to cause it do just display the btrfs commands it would run and not actually do anything. The next parameters are the number of minute based and day based snapshots to keep (I am currently experimenting with 100 100 for /home to keep 15 minute snapshots for 25 hours and daily snapshots for 100 days). After that is a list of filesystems to remove snapshots from. The removal will be from under the backup subvolume of the path in question. #!/bin/bash
set -e

# usage:
# btrfs-remove-snapshots [-d] MINSNAPS DAYSNAPS paths
# example:
# btrfs-remove-snapshots 100 100 /home /mail

if [ "$1" == "-d" ]; then
BTRFS="echo btrfs"
shift
else
BTRFS=/sbin/btrfs
fi

MINSNAPS=$1
shift
DAYSNAPS=$1
shift

for DIR in $* ; do
BASE=$(echo $DIR cut -c 2-200)
for n in $(btrfs subvol list $DIR grep $BASE/backup/.*: head -n -$MINSNAPS sed -e "s/^.* //"); do
$BTRFS subvol delete /$n
done
for n in $(btrfs subvol list $DIR grep $BASE/backup/ grep -v : head -n -$MINSNAPS sed -e "s/^.* //"); do
$BTRFS subvol delete /$n
done
done A Warning The Debian/Wheezy kernel (based on the upstream kernel 3.2.32) doesn t seem to cope well when you run out of space by making snapshots. I have a filesystem that I am still trying to recover after doing that. I ve just been buying larger storage devices for systems while migrating them to BTRFS, so I should be able to avoid running out of disk space again until I can upgrade to a kernel that fixes such bugs.

I haven't used my yurt much at all lately. It was an excellent getaway when I needed to get away; these days I don't. Or, perhaps, some would say, I've gotten away... So yesterday, since I happened to be over there, and after we'd played board games to our hearts content, we decided to take it down. This had been on the todo list since last fall, so it was not entirely spur of the moment. Still, the days are short this near to solstice and this one only had a few hours of light left. It'd been up for well over 4 years, in a wet environment. The roof is pretty well shot and will need to be replaced, but the canvas walls are still in good shape. There was a surprise; it seems a mouse or something started nibbling on all the wood of the walls recently, and even the center ring is pitted and worn by rodent teeth. It all seems still structurally sound. And looking back at my setup pictures, I'm not bothered by the yurt having aged, it just makes it seem more real. We left the bones of the yurt standing on its hill, seeming very appropriate under the bare winter trees. Will return soon and finish taking it down. And perhaps in some years I'll find a need for it again. (Pictures by my sister, who will be annoyed at me linking to this not-yet-formally published post about the yurt from her POV: Decline and fall of the yurt)

This week, I accepted a job on the faculty of at the University of Washington Department of Communication. I've arranged for a post-doc during the 2013-2014 academic year which I will spend at UW as an Acting Assistant Professor. I'll start the tenure-track Assistant Professor position in September 2014. The hire is part of a "big data" push across UW. I will be setting up a lab and research projects, as well as easing into a teaching program, over the next couple years. I'm not going to try to list all the great people in the department, but UW Communication has an incredible faculty with a strong background in studying the effect of communication technology on society, looking at political communication, enagement, and collective action, and tracing out the implications of new communication technologies in addition to very strong work in other areas. Years ago, I nearly joined the department as a graduate student. I am unbelievably happy that their faculty has invited me to join as a colleague. Outside of my new department, the University of Washington has a superb group of folks working across the school on issues of quantitative and computational social science, human-computer interaction, and computer-supported cooperative work. They are hiring a whole bunch of folks, across the university, who specialize in data-driven social science. I already have a bunch of relationships with UW faculty and students and am looking forward to expanding and deepening those. On a personal level, Mika and I are also very excited to return to Seattle. I grew up in the city and I've missed it, deeply, since I left now nearly half my lifetime ago! It will be wonderful to be much closer to many of my family members. But I know that I will miss the community of friends and colleagues that I've built in Boston over the last 7+ years just as deeply. I'm going to miss the intellectual resources, and the intellectual community, that folks in Cambridge get to take for granted. That said, I plan to maintain affiliations and collaborations with folks at Harvard and MIT and will have resources that let me spend time in Boston doing that. If you are curious what I'm going to be up to and what the future is likely to hold in terms of my research you should check the material I've put online as part of the job market this year. I've posted just about everything on my academic website. This includes a little four page research statement which describes the work I've done and the directions I've been thinking about taking it. The academic job market is challenging and confusing. But it's given me a lot of opportunity to reflect, at length, on both the substance of my research and the academy and its structures and processes. I've got a list of blog topics queued up based on that thinking. I'll be posting them here on my blog over the next few months.

A long time ago, I spent many many hours playing Frontier (Elite 2) on the family Atari ST. It was nice-looking (oh, those sunrises on distant planets!), playable, varied, had ample room for progression, very different possible roles, a huge universe of which nobody could ever hope of exploring more than a tiny fraction, and it was basically the first game I encountered with no set goals. You are a spaceship pilot in a galaxy colonised left and right, and it's up to you to decide what you want the game to be. Pirate, headhunter, mercenary, trader, taxi, explorer, there's no end goal to achieve except what you fix for yourself. That was, for me as a youngster, a mind-opening experience, which I never felt as strongly since (with the possible exception of the Creatures game, but I barely tried it). Then came modernity, the PC, and the First Encounters sequel (Elite 3). It added moderate complexity to the gameplay, some sort of political evolution among the factions and a kind of plot so the galaxy was no longer quite so static, but the end results were still a bit disappointing. It felt like Frontier with a few not-exactly-overwhelming textures stuck onto the ships, a loose plot arc, and a third faction, the Alliance of Independent Systems, beyond the Federation and the Empire. Not that exciting, although the name of the Alliance's capital star system stayed in my mind. So, like many others, I came to wait for Elite 4. That wait started in 1995, more than seventeen years ago. During that time, Duke Nukem Forever was announced, promised, developed, postponed, abandoned, found again, developed again, and it was even actually released in 2011, while Elite 4 went into the same kind of hell only without the actual release (yet). At some point I started trying various substitutes to help pass time. There was Parsec which was a nice-looking space combat simulator, multiplayer and all, but never felt like finished. Development slowed down to a thin trickle, and it seems like it only recently started again as Open Parsec. Hopefully it won't wither off again, but since it's only focused on combat, it's not exactly the same spirit as Elite. There's Oolite, but it looks and feels very much like a lightly-modernised Frontier. Better textures, sure, but the UI still looks like it got taken from the times where games mentioned they required an EGA/VGA video card. There's Vega Strike. It's very promising too, and I did try it at some point, but it was crashing too often for my tastes. Development seems to have slowed down, but not stopped; maybe it's time I gave it another go. I just recently found out about Pioneer, and I admit I'm excited. Really excited. It feels like like Frontier felt at the time. It's not finished yet (the on-planet cities in particular look a bit weird), but from the alpha I tried and the videos on the website it's really impressive. I'm going to keep an eye on that, knowing I could spend quite some time in there even before it's quite finished, if only to fly a spaceship in the canyons of Europa with Jupiter rising in the distance. I'll let you know if I find a black monolith. But the main event triggering this blog post is that apparently Elite 4 is not dead (yet). Much better, it's under heavy development, under the Elite Dangerous name, and the current state seems to be rather good already. And Frontier Developments, the company behind it, opened a Kickstarter campaign to fund the project. The screenshots and the videos look impressive, and the interviews of the lead developer imply that the gameplay will be awesome. The galaxy will apparently be really dynamic, and the player's actions could really influence its geopolitical (galactopolitical?) evolution, the profitability of trade routes, the prevalence and repartition of space pirates in various sectors, and so on. Combine with a multi-player mode, and this could become the greatest game ever made (for my personal taste at least). So I'm very much hoping the funding campaign reaches its target. The release date is set to March 2014. It's going to be a long wait, fraught with incertitudes. But then again, Duke Nukem Forever was eventually released, wasn't it? [No, I never played the original Elite. I'm not that old.]

Have you ever heard someone talking extensively about Cloud Computing or generally Clouds? and have you ever noticed the fact many people (even the ones who present themselves as experts) don t really understand what a Cloud is at all? That happened to me multiple times and one of the most common misunderstandings is many see the Cloud as something being on the internet. Many companies add a little logo representing a cloud on their frontpage and without a single change on their infrastructure (but surely with a price increment) they start calling their products as being on the Cloud. Given the lack of knowledge about this specific topic people tend to buy the product presented as being on the Cloud without understanding what they really bought. But what Cloud Computing really means? it took several years and more than fifteen drafts to the National Institute of Standards and Technology s (NIST) to find a definition. The final accepted proposal:

Cloud computing is a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction.

The above definition requires a few more clarifications specifically when it comes to understand where should we focus on while checking for a Cloud Computing solution. A few key points:

1. On-demand self-service: every consumer will be able to unilaterally provision multiple computing capabilities like server time, storage, bandwidth, dedicated RAM or CPU without requiring any sort of human interaction from their respective Cloud providers.
2. Rapid elasticity and scalability: all the computing capabilities outlined above can be elastically provisioned and released depending on how much demand my company will have in a specific period of time. Suppose the X company is launching a new product today and it expects a very large number of customers. The X company will add more resources to their Cloud for the very first days (where they suppose the load to be very high) and then it ll scale the resources back as they were before. Elasticity and scalability permit the X company to improve and enhance their infrastructure when they need it with an huge saving in monetary terms.
3. Broad network access: capabilities are available over the network and accessed through standard mechanisms that promote use by heterogeneous thin or thick client platforms (e.g., mobile phones, tablets, laptops, and workstations).
4. Measured service: Cloud systems allow maximum transparency between the provider and the consumer, the usage of all the resources is monitored, controlled, and reported. The consumer knows how much will spend, when and in how long.
5. Resource pooling: each provider s computing resources are pooled to serve multiple consumers at the same time. The consumer has no control or knownledge over the exact location of the provided resources but may be able to specify location at a higher level of abstraction (e.g., country, state, or datacenter).
6. Resources price: when buying a Cloud service make sure the cost for two units of RAM, storage, CPU, bandwidth, server time is exactly the double of the price of one unit of the same capability. An example, if a provider offers you one hour of bandwitdh for 1 Euro, the price of two hours will have to be 2 Euros.

The Cloud Computing technology is reasonably the future but can we trust Cloud providers? Are we sure that no one will ever have access to our files except us? and what about governments interested in acquiring a specific customer data hosted on the Cloud? I always suggest to read deeply both the Privacy Policy and Terms of Use of a certain service before signing in especially when it comes to choose a Cloud storage provider. Many providers have the same aspect, they seem to provide the same resources, the same amount of storage for the same price but legally they may present different problems, and that is the case of Spideroak vs Dropbox. Quoting from the Dropbox s Privacy Policy:

Compliance with Laws and Law Enforcement Requests; Protection of DropBox s Rights. We may disclose to parties outside Dropbox files stored in your Dropbox and information about you that we collect when we have a good faith belief that disclosure is reasonably necessary to (a) comply with a law, regulation or compulsory legal request; (b) protect the safety of any person from death or serious bodily injury; (c) prevent fraud or abuse of DropBox or its users; or (d) to protect Dropbox s property rights. If we provide your Dropbox files to a law enforcement agency as set forth above, we will remove Dropbox s encryption from the files before providing them to law enforcement. However, Dropbox will not be able to decrypt any files that you encrypted prior to storing them on Dropbox.

It s evident that Dropbox employees can access your data or be forced by legal process to turn over your data unencrypted. On the other side, Spideroak on its latest update to its Privacy Policy states that data stored on their Cloud is encrypted and inaccessible without user s key, which is stored locally on user s computers. And what about the latest research paper, titled Cloud Computing in Higher Education and Research Institutions and the USA Patriot Act written by the legal experts of the University of Amsterdam s Institute for Information Law stating the anti-terror Patriot Act could be theoretically used by U.S. law enforcement to bypass strict European privacy laws to acquire citizen data within the European Union without their consensus? The only requirement for the data acquisition is the provider being an U.S company or an European company conducting systematic business in the U.S. For example an Italian company storing their documents (protected by the European privacy laws and under the general Italian jurisdiction) on a provider based in Europe but conducting systematic business in the United States, could be forced by U.S. law enforcement to transfer data to the U.S. territory for inspection by law enforcement agencies. Does someone really care about the privacy of companies, consumers and users at all? or better does privacy exists at all for the millions of the people that connect to the internet every day?

Commonly used to sleep a random amount of time or to create unique temporary file names, $RANDOM is one of those bashisms that you are best avoiding it altogether.

It is not uncommon to see scripts generating a "unique" temporary file name with code that goes like: tempf="/tmp/foo.$RANDOM", or tempf="/tmp/foo.$$.$RANDOM".

Under some shells the "unique" temporary file name will be "/tmp/foo." for the first example code. So much for randomness, right?

Even if you go around it by defining $RANDOM to the output of cksum after reading some bytes from /dev/urandom, please: don't do that. Use the mktemp command instead.
When creating temporary files there's more than just generating a file name. Just don't do it on your own: use mktemp. Really, use it, the list of those who weren't using mktemp (or similar) is large enough as it is.

Don't even dare to mention the linux kernel-based protection against symlink attacks. There's no excuse for not using mktemp.

Tip: If you are going to use multiple temporary files, create a temporary directory instead. Use mktemp -d.
Tip: Don't reuse a temporary file's name, even if you unlink/remove it. Generate a new one with mktemp.
Tip: Reusing also means doing things like tmp="$(mktemp)"; some_command > "$tmp.stdout" 2> "$tmp.stderr"
Tip: Even if $RANDOM is not empty, don't use it. It could have been exported as an environment variable. Again, just use mktemp.

For the remaining cases where you may want a pseudo random number such as for sleeping a random number of seconds: you can use something as simple as $$. Use shell arithmetic to adjust it as needed: use the modulo operator, multiply it, etc.

If you think you need something more "random" than the process' id, then you should probably not be using $RANDOM in the first place.

Kasbah, my old laptop, is almost dead. It was my first laptop, an Asus X50C, which worked like a charm and was robust like a tank. But then the screen hinges broke. Time for a new one. So, ladies and gentlemen, I'm glad to present you malatesta. Malatesta is a Lenovo Thinkpad E335. Sadly, while kasbah worked with free drivers, malatesta requires proprietary drivers both for the video card and the wifi. Which sounds a bit ironic for a laptop named after a famous anarchist. The only two problems I encountered with it were related to the video card (00:01.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Device 9808) and the special keys. Malatesta worked well with the kernel of Squeeze, but freezed during the boot with Wheezy. Thankfully, I stumbled upon this blogpost by Blars Blarson suggesting to use the fglrx-driver with Sid. And that solved the problem. Another little odissey was the brightness of the monitor. With kernels < 3.2.0-4-amd64 the thinkpad_acpi didn't recognize my model and I was stuck with a brightness level probably visible from the outer space. Neither xrandr nor xbacklight could help, and I was seriously considering to use sunglasses to work (a slight unusual approach to the brightness problem, I know, but I suffer of migraines and couldn't bear an hyper-bright monitor), when the upgrade to Sid solved the problem. On the bright side (pun intended) this laptop was born without Windows. (Both malatesta and kasbah were shipped with FreeDOS). Now, if you're wondering about the names: my first namescheme was "references to songs by the Clash" (and kasbah is obviously for "Rock the Casbah", with a typo). But it wasn't really scalable. Since last year, I've decided to name my systems after famous anarchists. The server is named after one of my all time favourite anarchists, Buenaventura Durruti.

madamezou@durruti:~$ cat /etc/motd
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Llevamos un mundo nuevo en nuestros corazones;
y ese mundo est  creciendo en este instante
                        Buenaventura Durruti
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

While the new laptop's name comes from Errico Malatesta.

madamezou@malatesta:~$ cat /etc/motd 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
We anarchists do not want to emancipate the people;
We want the people to emancipate themselves.
            Errico Malatesta, L'Agitazione (18 June 1897)
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Steven Johnson gave an interesting TED talk about where good ideas come from [1]. He starts by attributing coffee replacing alcohol as a standard drink for some good ideas and then moves on to how ideas develop. Erez Lieberman Aiden and Jean-Baptiste Michel gave an interesting and amusing TED talk about the ngram analysis of books that Google scanned [2]. Here is the link for the Google Books ngram search [3]. Clay Shirky gave an insightful TED talk about how the Internet is changing the world [4]. He cites Open Source programmers as the modern day equivalent to the Invisible College based on our supposed great ability to get along with other people on the Internet. If we really are so much better than the rest of the Internet then things must be bad out there. He ends with ways of using Git to draft legislation. Hans Rosling gave an interesting TED talk about religion and the number of babies that women have [5]. His conclusion is that it s more about income and social stability and that the world s population can stabilise at 10 billion if we provide family planning to everyone. Alexis C. Madrigal wrote an interesting interview with Genevieve Bell about her work at Intel and the way people use technology [6]. Indigogo is raising funds for the Cuddle Mattress , it s a mattress with foam slats and a special fitted sheet to allow your arm to slide between the slats [7]. So you could have your arm underneath your SO for an extended period of time without risking nerve damage. They also show that when sleeping on your side your shoulder can go between the slats to avoid back problems. Nate Silver (who is famous for predicting US elections gave an interesting TED talk about racism and politics [8]. One of his main points is to show the correlation between racism and lack of contact of members of other races. Sociological Images has an interesting article by Lisa Wade about whether marriage is a universal human value [9]. In regard to historical marriage she says women were human property, equivalent to children, slaves, servants, and employees . The general trend in the comments seems to be that there are so many types of marriage that it s difficult to make any specific claims to traditional marriage unless you count a tradition of a short period in a single geographic region. Plurality is an excellent sci-fi short movie on youtube [10]. TED has an interesting interview with Hakeem Oluseyi about his research about astrophysics and how he achieved a successful career after being a gangster as a teenager [11]. He has some good ideas about helping other children from disadvantaged environments become successful. Paul Dwerryhouse wrote an interesting blog post about his work in designing and implementing a filesystem based on a Cassandra data store with FUSE [12]. Paul also wrote a post about using Apache Zookeeper to lock metadata to permit atomic operations [13]. The documentary Monumental Myths provides an interesting and insightful analysis of the processes of creating, maintaining, and explaining monuments [14]. It focusses on some significant monuments in the US and explains both sides to each story. Howard Zinn makes the insightful point that when people present a certain point of view of history it s not controversial, as soon as you present the other side they call it controversial . That happens even in debates about current issues. Howard also says to criticise whatever the government does is not anti-America, it s anti-government, it s pro-America, it s pro the people, it s pro the country . The song that plays during the closing credits is interesting too. The music video Same Love is one of the best presentations of the argument for marriage equality [15]. Chris Samuel wrote an interesting post about systems locked down for Windows 8 and options for purchasing PCs for running Linux [16]. His solution is to buy from ZaReason. I saw his laptop in action at the last LUV meeting and it looks really nice. Unfortunately a byproduct of the extremely thin form factor is the fact that it lacks a VGA port, this meant that Chris had to use my Thinkpad T61 (which is rather clunky by comparison) for his presentation.

• [1] http://www.ted.com/talks/steven_johnson_where_good_ideas_come_from.html
• [2] http://www.ted.com/talks/what_we_learned_from_5_million_books.html
• [3] http://books.google.com/ngrams/
• [4] http://www.ted.com/talks/clay_shirky_how_the_internet_will_one_day_transform_government.html
• [5] http://www.ted.com/talks/hans_rosling_religions_and_babies.html
• [6] http://tinyurl.com/czsmbhh
• [7] http://www.indiegogo.com/cuddle-mattress
• [8] http://www.ted.com/talks/nate_silver_on_race_and_politics.html
• [9] http://thesocietypages.org/socimages/2012/10/03/is-marriage-a-universal-human-value/
• [10] http://www.youtube.com/watch?v=IzryBRPwsog
• [11] http://blog.ted.com/2012/10/05/rise-of-a-gangsta-nerd-fellows-friday-with-hakeem-oluseyi/
• [12] http://tinyurl.com/co32js5
• [13] http://tinyurl.com/c9bfh3q
• [14] http://www.youtube.com/watch?v=cQIMrw8gSVQ
• [15] http://www.youtube.com/watch?v=hlVBg7_08n0
• [16] http://www.csamuel.org/2012/10/18/paying-for-freedom

Russ makes a good point about a nuance of practical jokes that makes them special, the characteristic of involving a third party which is the target of the joke, and how that relates to consent. He is right, consent is an important fact, I acknowledge that. Russ also says at the end of his post that being an asshole doesn't make you responsible for somebody else's decisions, I ackowledge that, too. I even somewhat agree to the position that the employer might fire you on assholery reasons, but in this particular case, I think that's a hypocrite position. If assholery was an issue, it should have been before, too, and I don't think the employer thought of it before this sad even, as long as it meant more money for them. I think this only shows how cynical they are, the cost analysis for them showed them they now might lose more money by not firing, than by keeping them. It's just cynical.

In spite of that, my initial post, just as I wrote, was not about the DJs, the people involved, the assholery, the taste or lack thereof of the joke. That's why I was intentionally leaving out names or specific context. It was about the bigger picture, irrational reactions and how in the face of a tragedy we surrender our thinking capabilities and react in an irrational manner to the least significant things.

I have had quite recent unfortunate events happening to people close to me, and, invariably, I observe this loss of rational faculties in the face of sad events. Even when this kind of disproportionate response is shown for what it is and how it is harmful in itself, people still refuse to give up on being irrational or they do after a long struggle with the facts.

I'll finish my article with a similar question: can we, as a species, go beyond our condition and make a conscious effort to keep calm and think things through before making decisions in the face of tragedy, or are we incapable to do that?

And no, I am not advocating to drop feelings and affection, I find it sad that I have to say that again, but I am almost sure that in spite of it, I will be wrongly understood by somebody.

Review: Peacekeeper, by Laura E. Reeve

Series:	Major Ariane Kedros #1
Publisher:	Roc
Copyright:	December 2008
ISBN:	0-451-46245-9
Format:	Mass market
Pages:	324

Ariane Kedros is a pilot of a second-wave prospecting ship. Second-wave prospectors are the first explorers into new systems after slower-than-light generation ships haul in the beacons that add them to the FTL network. She's also a reserve major in the Consortium military, which is a point of ongoing tension with her business partner and employer. She disappears on missions for the intelligence branch that she can't talk about, and usually ends up drinking herself into a stupor when she returns. In actuality, Kedros is not her real name. It's a fake identity given her after the last war, a war in which humans detonated a temporal distortion warhead for the first time and possibly destroyed an entire solar system. Kedros was part of the team that launched the bomb and is, according to the Terran Expansion League (the other side of the all-human conflict), a war criminal. She's in the equivalent of a witness protection program, as are all the other people involved in the act. But someone has broken their cover and is picking them off, one by one, and her latest secret mission from the Consortium is to protect another member of the group. Peacekeeper is Reeve's first novel, and unfortunately this shows badly at the start of the book. The writing is choppy and awkward, the paragraph breaks are a bit off, and the sentence structure kept making my teeth itch. I'm not a good enough technical critic to put a finger on what's wrong, but the writing doesn't flow. I also found myself anticipating and completing sentences in my head, which is a sign of too many low-level cliches. A few dozen pages into the book, the writing was bothering me, I hadn't warmed to any of the characters, and I wasn't sure I was going to like it. Ariane grew on me, though, particularly once she got into her mission and started balancing a cover assignment and her actual work. She's hurt, suffering from what's probably post-traumatic stress disorder and self-medicating with alcohol, but she's smart and competent and the plot is sufficiently tricky to provide a lot of meat. Not only is someone after her old team, but she and her partner discovered something potentially major on their last trip, an old friend of her partner has been murdered, the Consortium and the Terrans are negotiating a peace treaty, and all this is happening under the watchful eye of a superior alien race. The Minoans are probably Reeve's best creation here (if a little too similar to Vorlons at times). They're absolutely neutral, but they do not want humanity messing with temporal distortion weapons and are trying to nudge (or push) the humans into peace and disarmament. They are sticklers for not imposing their cultural practices on others, attempting to judge humanity solely by its own rules, but they're utterly literal-minded about rules. They expect everyone to follow any stated rules exactly and to the letter, making them excellent but frustrating overseers of disarmament treaties. They also have a fascinating tendency to interact with people in roles rather than as individuals. This is clearly the first book in an ongoing series, and a lot of the setup is not resolved here. We get only glimpses of what Ariane and her partner may have found, and only a few interactions with the Minoans, while Ariane tracks down who is responsible for killing her former team. There's a bit too much torture for my taste, and the subplot of Ariane's struggle with alcoholism wasn't really what I wanted to read about, but the writing seemed to get substantially better and the plot kept me turning the pages. There's a quality to military SF written by people who have actually been in the military that I find hard to describe but that helps a book tremendously. It didn't surprise me at all to learn that Reeve served in the US Air Force, since Peacekeeper has that quality. It's less about technical accuracy and more about a feel, a sense of what it's like to be inside a military routine and interacting with other military personnel. The culture feels coherent, despite not being described in detail. I think it makes for good SF in general, since it conveys to someone like me (who has never been in the military) a semi-alien but completely coherent culture. It certainly improves this book; in general, I thought Reeve's military characters were more interesting and more believable than her non-military characters. With the rocky beginning, the unpolished writing, and a villain who wasn't entirely satisfactory, I can't quite recommend Peacekeeper, but it's not a bad read. I'm not sure if I'll pick up the sequels, but I did leave it curious about the unresolved plot threads and willing to learn more about Reeve's world. Followed by Vigilante. Rating: 6 out of 10

I want to start this post by saying two things:

1. It is sad when a good person decides to commit suicide
2. It is even sadder anyone writing on a subject like this MUST add this kind of disclaimers in the hope the message of the writing isn't distorted because we are unable to listen to each other without suspecting/blaming the other of pure evilness/lack of compassion.

Imagine you are working in an office with other colleagues and one of them makes a print screen of your desktop screen and then displays it full-screen on your computer. You struggle a little and get frustrated when nothing works, the colleagues laugh at your expense. Pretty childish, but no harm, right? You laugh it off.

Imagine a similar scenario, but your shoelaces are tied to the desk without your notice, and the colleagues tied your laces so you can't stand, so the risk of injury is null. You stumble a little when you want to leave the desk. Again, childish, funny, but no harm, right? Again, you laugh it off.

Imagine even a scenario where you are the target of a prank that doesn't injure you physically, but it happens in front of an audience of 100 people. You might feel ashamed, embarrassed, depending on the prank. But that would go away with time, with your laughs or by leaving that audience, right?

So now imagine you are just a person that unknowingly contributed to the success of a harmless prank in which somebody was duped into thinking they were talking to a very public figure about some unimportant and inconsequential stuff related to their job. You'd brush it off, wouldn't you? Most likely, you'd probably laugh at the prank and you wouldn't think the pranksters should be prosecuted or fired. Even if the famous public figure would be upset, taking into account the fact that the things discussed were inconsequential, you wouldn't worry about it. I wouldn't worry even if that famous person was my direct boss and I was the one talking about my boss.

At least, that's how I am thinking, but for some reason, many people don't think this way. And yes, I am talking about the recent sad news about the apparent suicide of a nurse who was involved in the care of the person called Kate Middleton.

I don't want to talk about names, or man-made titles, man-made concepts, or how some people forget that even the most influential people in the world are still a member of this same species that all of us are. I don't even want to talk about this particular sad situation, I just want to talk about the totally irrational reaction many people had to it.


Everybody knows that jokes are a matter of taste. What I might consider funny, some people would take offence at, but, generally speaking, there's an area of jokes that even kids would consider funny. Also, if we are honest to ourselves, a joke is not funnier or less funnier if from the total of people in the room instead of 2 people not liking it, 5 people don't. Not even if some of them are so offended by the joke they leave the room. The joke is equally funny or in poor taste with 2 or with 100 people in the room.

And there's the issue of confusing correlation with causation. If I tell somebody some good news about my aunt and the next day that somebody happens to had bought a car, it doesn't mean my good news made that person decide they want to buy a car. It could be, but it's not necessary, it could be a coincidence.

The same happens in the sad case of the nurse. She decided to take her life, as far as it seems. But that's the extent to which we know anything about the situation. We don't know if the suicide had anything to do with the prank or not. Just the fact that it happened the next day doesn't mean AT ALL the prank call is the cause of the suicide. It doesn't even mean that is "the straw that broke the camel's back", it might be totally unrelated. Or it might not. Taking into account the inconsequential and harmless nature of the call, it seems even less likely. Yet, I might be mistaken.

What I am sure of now is that jumping to the conclusion that the prank call had anything to do with the death, is wrong, as it is jumping to the opposing conclusion. I am also sure that in the absence of other information, the latter, the lack of connection, is more likely, because we know that such drastic sad decisions are not generally caused by trivial stuff.

Time proximity doesn't prove causation. Geographical proximity doesn't either. There needs to be a lot more into it than simple temporal sequence.



Which gets me to the second part of irrational reactions: terminating the show and twitter accounts of the two pranksters. What the hell? Talk about hypersensitivity and paranoid reactions! Are we this blind and willing to blame the least insignificant thing for the gravest consequences that we forget how we all make decisions and how to use our brains, as soon as some tragedy happens?



Imagine some person who thinks the 21st of December 2012 will bring the end of the earth. Imagine that person taking as a serious declaration the obvious joke from the Australian prime-minister Julia Gillard, and would end their lives to avoid the mayhem, panic and violence they think would follow the 21st of December. I wonder, would as many people blame Julia Gillard for the suicide? I doubt it. And the irony is that for such an hypothetical case, there would be a stronger case for causation than what we now know about the previous case.


I still wonder, when will we learn to think a little more rationally than the people walking the Earth 100.000 years ago?

I ve just bought a 7 Onix tablet from Aldi. It runs Android 4.0.4, has a 1GHz Cortex A8 CPU, 512M of RAM, 16G of flash storage, and a 800*480 display. They are selling rapidly and I don t know how long they will last probably you could get a returned one next week if you can t get one today. But if you like pink then you may be able to get one (the black ones are selling out first). The tablet seems like a nice piece of hardware, solid construction and it feels nice to hold. Mine has a minor screen defect, but that s the sort of
thing you expect from a cheap device, apart from that the display is good. The Wifi doesn t seem to have as good a range as some other devices (such as my phones and the more expensive 10 tablet I got from Kogan). This isn t a
problem for me (the data intensive uses for this device will be in the same room as the AP) but could be a killer for some people. If you have your phone or a dedicated 3G Wifi AP in your pocket while using the tablet then it should be fine, but if you have an AP at the wrong end of your house then you could be in trouble. I found Youtube unusable due to slow downloading even when sitting next to my AP but I can play videos downloaded from iView that are on my local web server (which is more important to me). I expect that I will be able to play local copies of TED talks too. The camera is bad by phone camera standards, fortunately I have no interest in using a tablet as a camera. I had no real problems with the Google Play store (something that caused problems for some users of an earlier Aldi Android tablet). Generally the tablet works well. The people who build Android for modern devices seem remarkably stupid when it comes to partitioning, every device I ve seen has only a small fraction of the storage usable for apps. This tablet is the worst I ve ever seen, it has 16G of storage of which there is 512M partitioned for apps of which only 400M is free when you first get the device! It comes pre-installed with outdated versions of the Facebook client and Google Maps (which isn t very useful on a Wifi device) and some other useless things. If you upgrade them to the latest versions then you ll probably lose another 100M of the 512M! Fortunately the Android feature to run apps from the VFAT partition works so I haven t been prevented from doing anything by this problem yet. In conclusion, it s not the greatest Android tablet. But you don t expect a great tablet for $100. What I hoped for was a somewhat low spec tablet that works reasonably well and that s what I got. I m happy.

I've recently replaced my laptop's HDD as the S.M.A.R.T. data suggested a potential failure. I took the opportunity to try out beta4 of debian-installer and to install a pure wheezy system. Prior to this I've been running a mostly wheezy system with some bits from sid or experimental. I've got quite a few notes about bugs I've hit in the installer, most of which I still need to file, but perhaps the most problematic appears to be known as #661789. Unless I'm reading this wrong, it would appear this show-stopper bug makes booting on an EFI system impossible without some workarounds, it's been a known problem for 9 months, and d-i beta4's images opt for EFI booting where possible. After the 8th install run I must admit I started getting angry that someone was wasting my time. Perhaps this is covered in some install notes somewhere. Anyway, I got an install working eventually (with legacy BIOS boot mode). and have been running wheezy for the last week. I've just now had a string of problems which were a fairly typical experience for me running a mixed system. Whenever I hit a problem, particularly in the GNOME3 stack, my first response was to check "am I running the latest version of the packages?" inevitably not and so I got around my problems via a reboot or something similarly drastic; updated the packages and left things. Until the next time. Just now, the following things happened:

• I resumed the laptop and typed my password into GNOME's screensaver prompt, which then hung. I switched to a text console and tried to log in, but also hung, before timing out. I got in as root (glad I set up a root password) and had a look around, it seems kerberos made it's way into my PAM configuration and all users with uids 1000 were failing. I have libpam-krb5 installed but Kerberos unconfigured. I'm not sure why this has only bitten now. I'm also fairly sure kerberos is an optional PAM module and pam_unix should still work.
• Killing gnome-screensaver gets my to my "desktop" but gnome-shell appears to have hung: it isn't drawing anything and all I can see is my wallpaper. Killing the shell process sadly doesn't cure this: I can see the (undecorated) edge of an open window now, but the shell isn't coming back. It seems the session manager doesn't respawn such subprocesses when they die.
• Give up, restart gdm3, lose my open windows/data, log in afresh. For some reason, there's no networking panel applet visible. Wifi is not up. If I go to the GNOME control panel and visit "Networking", I'm told "The system network services are not compatible with this version". (They were earlier today, with no package changes ). Poking around in a terminal, I see the wifi controller is software disabled. I don't have rfkill installed and can't immediately see how to unblock it via poking around in /sys (probably possible) so I'm left with no choice but to reboot.

I've experienced all of the above (except for the PAM thing) more than once over the last 12 months or so since I blogged about moving to GNOME3. Now I'm running a pure wheezy system, at this point in the freeze the relevant packages are likely to be more stable. So, not running the latest version is no longer a reason to avoid fixing the problem. Time for me to start proper analysis! Debian is now in deep freeze. I've heard it said a few times that we are coming to the end of that process. Based on my experiences, in my humble opinion, we've got some work to do before we're release worthy.

My parents will soon be moving to a smaller home, so they are digging up a lot of stuff of my sister and mine past. Among my stuff, they just sent me my first computer (which was, of course, the first computer of my father s company I was using in the spare time). I couldn t believe they kept that. It was an Unitron Apple ][ 64K!! I just had it cleaned and took this picture: Due to the closed informatics market Brazilians were subject to at the time, it came with a full set of manuals in Portuguese which taught me how to code in Basic (I was too young to learn English at the time) Interesting how a bad policy like that can result in a Good Thing sometimes. Is it just me or does this picture made you nostalgic also?

Update Dec 2012: ResearchGate still keeps on sending me their spam. Most of the colleagues I had that tried out RG now deleted their account there, apparently, so the invitation mails become fewer. Please do not try to push this link on Wikipedia just because you are also annoyed by their emails. My blog is not a "reliable source" by Wikipedia standards. It solely reflects my personal view of that web site, not journalistic or scientific research. The reason why I call ResearchGate spam is the weasel words they use to trick authors into sending the invitation spam. Here's the text coming with the checkbox you need to uncheck (from the ResearchGate "blog")

Add my co-authors that are already using ResearchGate as contacts and invite those who are not yet members.

See how it is worded so it sounds much more like "link my colleagues that are already on researchgate" instead of "send invitation emails to my colleagues"? It deliberately avoids the mentioning of "email", too. And according to the researchgate news post, this is hidden in "Edit Settings", too (I never bothered to try it -- I do not see any benefit to me in their offers, so why should I?). Original post below:

---

If you are in science, you probably already received a couple of copies of the ResearchGate spam. They are trying to build a "Facebook for scienctists", and so far, their main strategy seems to be aggressive inivitation spam. So far, I've received around 5 of their "inivitations", which essentially sound like "Claim your papers now!" (without actually getting any benefit). When I asked my colleagues about these invitations none actually meant to invite me! This is why I consider this behaviour of ResearchGate to be spam. Plus, at least one of these messages was a reminder, not triggered by user interaction. Right now, they claim to have 1.9 million users. They also claim "20% interact at least once a month". However, they have around 4000 Twitter followers and Facebook fans, and their top topics on their network are at like 10000-50000 users. That is probably a much more real user count estimation: 4k-40k. And these "20%" that interact, might just be those 20% the site grew in this timeframe and that happened to click on the sign up link. For a "social networking" site, these numbers are pointless anyway. That is probably even less than MySpace. Because I do not see any benefit in their offers! Before going on an extremely aggressive marketing campaign like this, they really should consider to actually have something to offer... And the science community is a lot about not wasting their time. It is a dangerous game that ResearchGate is playing here. It may appeal to their techies and investors to artificially inflate their user numbers in the millions. But if you pay for the user numbers with your reputation, that is a bad deal! Once you have the reputation as being a spammer (and mind it, every scientist I've talked to so far complained about the spam and "I clicked on it only to make it stop sending me emails") it's hard to be taken serious again. The scientific community is a lot about reputation, and ResearchGate is screwing up badly on this. In particular, according to researchgate founder on quora, the invitations are opt-out on "claiming" a paper. Sorry, this is wrong. Don't make users annoy other users by sending them unwanted invitations to a worthless service! And after all, there are alternatives such as Academia and Mendeley that do offer much more benefit. (I do not use these either, though. In my opinion, they also do not offer enough benefit to bother going to their website. I've mentioned the inaccuracy of Mendeleys data - and the lack of an option to get them corrected - before in an earlier blog post. Don't rely on Mendeley as citation manager! Their citation data is unreviewed. I'm considering to send ResearchGate (they're Berlin based, but there maybe also is a US office you could direct this to) a cease and desist letter, denying them to store personal information on me, and to use my name on their websites to promote their "services". They may have visions of a more connected and more collaborative science, but they actually don't have new solutions. You can't solve everything by creating yet another web forum and "web2.0izing" everything. Although many of the web 2.0 bubble boys don't want to hear it: you won't solve world hunger and AIDS by doing another website. And there is a life outside the web.

It is possible to put Debian on smartphones like the Samsung Galaxy S:

• Find a friendly Android distro that supports your device, install it with fastboot or heimdall and play around.
• Be sad that the proprietary bootloader doesn't support dual boot.
• Reformat the "sdcard" as ext3 and install Debian in a chroot on it.
• "Tether" your device via USB using the Android settings menu & add a default route with adb: adb shell ip add route default via 192.168.XXX.XXX. Then use one or other USB network script on your PC.
• Install OpenSSH, copy some commands from the usb.rc to /etc/rc.local so that you will have a way to access your Debian system over USB.
• Hack up the initramfs to mount the sdcard partition and then run the Debian init in the chroot instead of the Replicant init.
• Install some sort of graphical environment such as Enlightenment and nodm.
• Realise that the touchscreen driver is buggy, which means that udev and thus Xorg do not recognise it as an input device. Find out that the same device seems to also be supported by a different driver that is in Linux mainline. Boggle.
• Work around the buggy touchscreen driver by using x2x (orphaned) to send your laptop input devices over SSH to Xorg running on the phone.
• Click through the Enlightenment startup wizard, work around desktop-base not supporting E17 and install a touchscreen-friendly app like intone.
• Marvel at the speed and flashyness of Enlightenment even though it does not use OpenGL because there are no open drivers for PowerVR devices.
• Take a photo and blog about it to create some bzzt.
• Get depressed about the rediculousness of all that and proceed to drinking over 9000 litres of beer.
• Think about just using Replicant plus f-droid.

Linux mainline doesn't run on any of the mobile devices I have. It probably doesn't run on any of the mobile devices you have either. There has been some effort by the OpenMoko community to merge the gta02 kernel patches but it is not yet complete. I doubt Samsung will spend money on merging support for old devices obsoleted by more recent devices. There is Linaro but they are focused on things the hardware vendors pay them for and probably would not have the resources anyway. Therefore I would guess the timeframe for supporting the OpenMoko FreeRunner and the Samsung Galaxy S in Linux mainline is between many years and never. For better or worse, the Debian Linux kernel maintainers prefer not to include non-mainline stuff and Debian as a whole generally prefers to include one copy of each package instead of 9. The procedures I documented above are not a great way to support mobile devices at all and could break at any moment anyway. So everyone, please become a kernel developer and help merge all of the many many versions of Android Linux into Linux mainline so that you can have your favourite distribution on your devices. You can comment on this post at identi.ca.

A minor feature release RQuantLib 0.3.9 is now on CRAN and in Debian. RQuantLib combines (some of) the quantitative analytics of QuantLib with the R statistical computing environment and language. Bryan Lewis had suggested to enable another pricing engine for American Options in order to get (at least some) Greeks. This is now supported by picking engine="CrankNicolson" as shown in the default example for the AmericanOption function:

R> library(RQuantLib)
R> example(AmericanOption)
AmrcnOR> # simple call with unnamed parameters
AmrcnOR> AmericanOption("call", 100, 100, 0.02, 0.03, 0.5, 0.4)
Concise summary of valuation for AmericanOption 
  value   delta   gamma    vega   theta     rho  divRho 
11.3648      NA      NA      NA      NA      NA      NA 
AmrcnOR> # simple call with some explicit parameters
AmrcnOR> AmericanOption("put", strike=100, volatility=0.4, 100, 0.02, 0.03, 0.5)
Concise summary of valuation for AmericanOption 
  value   delta   gamma    vega   theta     rho  divRho 
10.9174      NA      NA      NA      NA      NA      NA 
AmrcnOR> # simple call with unnamed parameters, using Crank-Nicolons
AmrcnOR> AmericanOption("put", strike=100, volatility=0.4, 100, 0.02, 0.03, 0.5, engine="CrankNicolson")
Concise summary of valuation for AmericanOption 
  value   delta   gamma    vega   theta     rho  divRho 
10.9173 -0.4358  0.0140      NA      NA      NA      NA 
R> 

Thanks to CRANberries, there is also a diff to the previous release 0.3.8. Full changelog details, examples and more details about this package are at my RQuantLib page.

The following proposal extends the JSON specification, with the idea of using JSON as an information interchange format, rather than just a way of writing certain ECMAscript values. They do not add anything but only restrict valid JSON content and encoders with some rationale. First of, I d like to remind everyone, including JSON s author, that JSON is case-sensitive, except in the four hexdigits after a backslash-u sequence in a String. Second, I d like to remind everyone that JSON is not binary-safe. No way around that, it implements Unicode (actually, 16-bit UCS-2, and it doesn t guarantee that UTF-16 surrogates are correctly paired) text. I also consider only UTF- 8, 16,32 B,L E valid encodings for JSON. (No PDP endian, either. Sorry, guys.) For my first proposal, I d like to point out CVE-2011-4815 which was about overflowing hashtables. The obvious fix is to randomise the hash per hashtable; to ensure this doesn t leak, we sort ASCIIbetically the keys of an Object in the encoder. (Using Unicode is good here we can just sort the keys as UTF-8 strings by their uint8_t value or as Unicode (UCS-2 or even UCS-4 or UTF-16) strings by the codepoints.) JSON was never preserving the order of elements in an Object anyway so we make it standardised (we still accept any order, and, when parsing, in collision cases, the later value wins). This also helps diffs. For my second proposal, I d like to forbid \u0000, \uFFFE, \uFFFF in strings. The first because many implementations use C strings, and for an information interchange format this is better; it also has security implications to allow NUL in a string. The other two, but not unpaired UTF-16 surrogates (as ECMAscript uses UCS-2 and got UTF-16 only later) because they re not valid Unicode; JSON was not binary-safe already so why bother. Among other benefits, this also helps implementations. For my third proposal, I d like to agree that implementations should impose a nesting depth limit that may be user-defined, and in the face of which, cyclic checking may be ignored by an encoder. I emit nesting depth overflows as literalnull; might also throw an error. Since I was asked, the common standard value is to restrict nesting depth is 32, unless the user specified one. (I also saw 8, but 32 WFM pretty well.) Most seem to use it even if it may seem low at first. Only specialised applications probably need more, and they can always pass a value. For my forth proposal, backslash-escape U+007F U+009F always. It may upset humans, editors, databases, etc. (This paragraph is newly added, after some IRC discussion.) All these do not permit anything that wasn t accepted to be accepted afterwards. I ve got a fifth proposal which changes acceptance rules but only for a subset of parsers: formally JSON is defined in ECMA-262 as industry standard that, in contrast to RFC 4627, always allowed any Value as top-level element of a JSON text. I d like to make it so, and ignore the RFC s requirement for it to be an Object or Array. Even so, the first two characters (after the BOM, if any) of a JSON text always are in the non-NUL 7-bit ASCII range, allowing for encoding detection. (This is done by the NUL octet pattern in the first four octets.) JSON has only taken off because it s a tightly defined simple format that can be used everywhere and isn t too awful for humans (escaping not needed for U+0020 U+D7FF and U+E000 U+FFFD after all, although I d also take the C1 control characters out, see my forth proposal above). I ve started to use a trailing comma in indexed and associative arrays in code I write at work, when the array values are one a line, to help version control systems to do their diffs, but refrain from asking for a JSON extension to permit that in order to not endanger compatibility any (no comment needed, it s just not worth it), but I d like my above proposals to be followed by implementators (and I m one of them). Some more discussion with Jonathan pointed out that JSON5 allows for trailing commata in Object and Array; IMHO the only feature of it that is not bad or outright harmful. I ll probably keep from accepting them because, on their own, they re not that useful, and I usually would run JSON texts, even configs, through a parser/encoder roundtrip to pretty-print them which would lose them anyway. As for binary-safeness: probably best to just use base64 and let the outer layers worry about compression. The data is usually unrelated to the JSON-encoded structure, and even if it s related to other data the base64 representation is usually similar (unless misaligned). Update 02.12.2012 Wrong I was about the first two characters: " " is a valid JSON text. Still possible to peek at four octets and determine the encoding by ordering the tests; updated my notes.

Review: American Gods, by Neil Gaiman

Publisher:	HarperTorch
Copyright:	July 2001
Printing:	May 2002
ISBN:	0-380-78903-5
Format:	Mass market
Pages:	588

At the start of American Gods, Shadow is in prison. He's done three years of a sentence of six and is about to get released on parole. He has a wife and a job waiting for him and every hope of getting back to a good life. None of that happens. The day he's released, he finds out his wife died in a car accident. The job is gone too; it was an offer from his best friend, who died in the same accident. Numb, an ex-con, with no job prospects, he's recruited by a strange man named Wednesday, with the help of a leprechaun named Mad Sweeney, to drive Wednesday around the country, run errands for him, hurt people if absolutely necessary, and hold his vigil if he dies. And that's how Shadow is drawn into the conflict between old gods and young, between the gods that immigrants brought to America from the old world and the flashy new gods of technology and media that America is creating for itself. With native American gods and cultural legends watching around the edges. I first read American Gods shortly after it came out and before I started writing reviews, so this is a re-read. It's a book I'm glad I re-read before reviewing. The first time I read it, I was thoroughly engrossed and thought it was excellent. This time, I was entertained, but surprised at how slowly it moved, how disconnected it sometimes felt, and how little substance there was to the plot. Reading American Gods involves spending lots of time with Shadow, and I think one's reactions to this book will vary a lot based on how much one likes him as a character. Shadow is very withdrawn, very detached from the world, and quietly matter-of-fact in how he goes through his life. He's been through some major shocks and has been severed fairly thoroughly from society by three years in prison, and he doesn't reconnect easily. To some readers, such as myself, this comes across as taciturn competence and quiet observation, two things that I greatly enjoy reading about. I liked Shadow, liked reading between the lines for hints at his feelings, and sympathized a lot with his emotional state. But I can easily see an alternative reading that makes him a cipher or, worse, a blank spot in the book into which the reader has to insert themselves. One has to work at seeing Shadow's character and finding some sympathy for the degree to which he stays separate from the world. American Gods does have a plot, but it's the sort of plot that allows for endless digressions, background stories, and side meetings. In that, it reminded me less of Gaiman's other novels and more of several arcs of Sandman. I think it's worthwhile to think of this less as a novel and more as an extended ramble through the world as Gaiman sees it, full of mythology, stories, quirky people understanding the world in their own ways, and upwellings of the sacred and significant in the most unexpected of places. For example, the holy places of America in American Gods, analogous to the multiple small sacred places and old ruins in Europe, are roadside attractions, via a line of argument that makes much more sense than it has any right to and demonstrates a canny understanding of the expression of American entrepreneurial culture. As with much Gaiman, reading with Wikipedia or another mythological resource at hand is a good idea. Even if you have a good general knowledge of the most common mythology studied in the US, you probably won't recognize many of the figures Gaiman draws on: Czernobog, for example, or Wisakedjak. Part of the fun of this book is seeing Gaiman humanize them and embed them in modern culture and modern concerns while retaining their mythological edge and strangeness. As any reader of Sandman can tell you, this is one of Gaiman's strengths as a writer, and he does it well here. The bits with the undertakers Mr. Ibis and Mr. Jacquel (Thoth and Anubis) are absolutely brilliant and probably the best sections of the book (although Sam's exchange with the Men in Black is a strong contender).

"Back in my day, we had it all set up. You lined up when you died, and you'd answer for your evil deeds and for your good deeds, and if your evil deeds outweighed a feather, we'd feed your soul and your heart to Ammet, the Eater of Souls." "He must have eaten a lot of people." "Not as many as you'd think. It was a really heavy feather. We had it made special. You had to be pretty damn evil to tip the scales on that baby."

Like so much of Gaiman's work, this is the sort of book that contributed a half-dozen quotes to my quote file during my first reading. Gaiman always impresses me by not only managing to portray gods as human without losing the numinous edge, but sustaining that portrayal through very extended contact with the same god. And despite the interlude with the undertakers, the long interlude in small-town life in Lakeside, and the numerous other digressions and meanderings of a book that's not in much of a hurry to get anywhere, there is a plot and a definite conclusion, and Gaiman has some interesting twists in store at the end of that road. There's a lot of philosophical currents going on under the surface, not all of which cohere, but Gaiman does pull a lot of them together. The resolution can be a bit of an anticlimax, but I think it's a fitting one that's true to the nature of the story. I didn't enjoy this as much on the second reading as I did on the first. It didn't feel quite as coherent as I'd remembered it, and the meandering pace occasionally made the story feel stretched. But I do like Shadow, I love spending time in Gaiman's head and with his portrayals of deities, and American Gods is a treasure of great quotes. If you enjoy Gaiman's other work and aren't looking for a lot of plot and forward-moving energy, this is definitely recommended. Rating: 7 out of 10

Good news, everyone! So as of last Sunday, this works on all Linux archs in Debian unstable and gives you a modern version of upstart:

echo 'Yes, do as I say!'   apt-get -o DPkg::options=--force-remove-essential -y --force-yes install upstart

Thanks to the ifupdown, sysvinit, and udev maintainers for their cooperation in getting upstart support in place; to the Debian release team for accomodating the late changes needed for upstart to be supported in wheezy; and to Scott for his past maintenance of upstart in Debian. Benchmarking One of the consequences is that it's now possible to do meaningful head-to-head comparisons of boot speed between sysvinit (with startpar), upstart, and systemd. At one time or another people have tested systemd vs. sysvinit when using bash as /bin/sh, and upstart vs. sysvinit, and systemd vs. sysvinit+startpar, and there are plenty of bootcharts floating around showing results of one init system or another on one distro or another, but I'm not aware of anyone having done a real, fair comparison of the three solutions, changing nothing but the init system. I've done some initial comparisons in a barebones sid VM, and the results are definitely interesting. Sysvinit with startpar (the default in Debian) can boot a stock sid install, with no added services, in somewhere between 3.37 and 3.42 seconds (three runs). That's not a whole lot, but on the other hand this is a system with a single filesystem and no interesting services yet. Is this really as fast as we can boot? No, even this minimal system can boot faster. Testing with upstart shows that upstart can do the same job in between 3.03 and 3.19 seconds (n=3, mean=3.09). This confirms what we'd already seen in Ubuntu, that it makes a difference to boot speed to have filesystem mounting handled by an integrated process that understands the whole system instead of as a group of serialized shell scripts. What about systemd? The same test gives a boot time between 2.32 and 2.85 seconds (n=4, mean=2.48). Interesting; what would make systemd faster than upstart in sid? Well, a quick look at the system shows one possible contributing factor: the rsyslog package in Debian has a systemd unit file, but not an upstart job file. Dropping in the /etc/init/rsyslog.conf from Ubuntu has a noticeable impact, and brings the upstart boot time down nearer to that of systemd (2.78-3.03s, n=5, mean=2.92). Besides telling me that it's time to start spamming Debian maintainers with wishlist bugs asking them to include upstart jobs in their packages, this suggests that the remaining difference in boot time may be due to the outstanding init scripts in rcS.d that are made built-in no-ops by systemd but not (yet) by upstart in Debian (e.g., hwclock, hostname, udev-mtab). (In Ubuntu, /etc/rcS.d has long since been emptied out in favor of upstart jobs in the common case, since the time it takes to get to runlevel=2 is definitely a major issue for boot speed and boot parallelization.) It also gives the lie to the claim that's been made in various places that spawning shells is a major bottleneck for upstart vs. systemd. More study is certainly needed to confirm this, but at least this naive first test suggests that in spite of the purported benefits of hard-coding boot-time policies in C code, upstart with its default degree of runtime configurability is at least in the ballpark of systemd. Indeed, when OpenSuSE switched from upstart to systemd, it seems that something else in the stack managed to nullify any benefit from improving the boot-time performance of apparmor. Contrary to what some would have you believe, systemd is not some kind of silver bullet for boot speed. Upstart, with its boot-time flexibility and its long history of real-world testing in Ubuntu, is a formidable competitor to systemd in the boot speed department - and a solid solution to the many longstanding boot-time ordering bugs in Debian, which still affect users of sysvinit. I've published the bootcharts for the above tests here. Between the fact that Debian's bootchart package logs by default to /var/log/bootchart.tgz (thus overwriting on every boot) and the fact that these tests are in a VM, I haven't bothered to include the raw data, just the bootcharts themselves. The interested reader can probably generate more interesting boot charts of their own anyway - in particular, it will be interesting to see how the different init systems perform with more complicated filesystem layouts, or when booting a less trivial set of services. Other musings The boot charts have been created with the bootchart package rather than with bootchart2. For one thing, it turns out that bootchart2 includes systemd units, not init scripts; so when replacing bootchart with bootchart2, the non-matching init script is left behind and systemd in particular gets terribly confused. This is now reported as Bug #694403. In an amusing twist, while I was experimenting with bootchart2, I also noticed that having systemd installed would slow down booting with other init systems, because systemd installs udev rules which take a noticeable amount of time to run a helper command at boot even though the helper should be a no-op. So if you're doing boot speed testing of other init systems, be sure you don't have systemd on the system at the time!