I ve just installed BTRFS on some systems that matter to me. It is still regarded as experimental but Oracle supports it with their kernel so it can t be too bad and it s almost guaranteed that anything other than BTRFS or ZFS will lose data if you run as many systems as I do and I run lots of systems that don t have enough RAM for ZFS (4G isn t enough in my tests). So I have to use BTRFS. BTRFS and Virtual Machines I m running BTRFS for the DomUs on a virtual server which has 4G of RAM (and thus can t run ZFS). The way I have done this is to use ext4 on Linux software RAID-1 for the root filesystem on the Dom0 and use BTRFS for the rest. For BTRFS and virtual machines there seem to be two good options given that I want BTRFS to use it s own RAID-1 so that it can correct errors from a corrupted disk. One is to use a single BTRFS filesystem with RAID-1 for all the storage and then have each VM use a file on that big BTRFS filesystem for all it s storage. The other option is to have each virtual machine run BTRFS RAID-1. I ve created two LVM Volume Groups (VGs) named diska and diskb, each DomU has a Logical Volume (LV) from each VG and runs BTRFS. So if a disk becomes corrupt the DomU will have to figure out what the problem is and fix it. #!/bin/bash
for n in $(xm list cut -f1 -d\ egrep -v ^Name\ ^Domain-0) ; do
echo $n
ssh $n "btrfs scrub start -B -d /"
done I use the above script in a cron job from the Dom0 to scrub the BTRFS filesystems in the DomUs. I use the -B option so that I will receive email about any errors and so that there won t be multiple DomUs scrubbing at the same time (which would be really bad for performance). BTRFS and Workstations The first workstation installs of BTRFS that I did were similar to installations of Ext3/4 in that I had multiple filesystems on LVM block devices. This caused all the usual problems of filesystem sizes and also significantly hurt performance (sync seems to perform very badly on a BTRFS filesystem and it gets really bad with lots of BTRFS filesystems). BTRFS allows using subvolumes for snapshots and it s designed to handle large filesystems so there s no reason to have more than one filesystem IMHO. It seems to me that the only benefit in using multiple BTRFS filesystems on a system is if you want to use different RAID options. I presume that eventually the BTRFS developers will support different RAID options on a per-subvolume basis (they seem to want to copy all ZFS features). I would like to be able to configure /home to use 3 copies of all data and metadata on a workstation that only has a single disk. Currently I have some workstations using BTRFS with a BTRFS RAID-1 configuration for /home and a regular non-RAID configuration for everything else. But now it seems that this is a bad idea, I would be better off just using a single copy of all data on workstations (as I did for everything on workstations for the previous 15 years of running Linux desktop systems) and make backups that are frequent enough to not have a great risk. BTRFS and Servers One server that I run is primarily used as an NFS server and as a workstation. I have a pair of 3TB SATA disks in a BTRFS RAID-1 configuration mounted as /big and with subvolumes under /big for the various NFS exports. The system also has a 120G Intel SSD for /boot (Ext4) and the root filesystem which is BTRFS and also includes /home. The SSD gives really good read performance which is largely independent of what is done with the disks so booting and workstation use are very fast even when cron jobs are hitting the file server hard. The system has used a RAID-1 array of 1TB SATA disks for all it s storage ever since 1TB disks were big. So moving to a single storage device for /home is a decrease in theoretical reliability (in addition to the fact that a SSD might be less reliable than a traditional disk). The next thing that I am going to do is to install cron jobs that backup the root filesystem to something under /big. The server in question isn t used for anything that requires high uptime, so if the SSD dies entirely and I need to replace it with another boot device then it will be really annoying but it won t be a great problem. Snapshot Backups One of the most important uses of backups is to recover from basic user mistakes such as deleting the wrong file. To deal with this I wrote some scripts to create backups from a cron job. I put the snapshots of a subvolume under a subvolume named backup . A common use is to have everything on the root filesystem, /home as a subvolume, /home/backup as another subvolume, and then subvolumes for backups such as /home/backup/2012-12-17, /home/backup/2012-12-17:00:15, and /home/backup/2012-12-17:00:30. I make /home/backup world readable so every user can access their own backups without involving me, of course this means that if they make a mistake related to security then I would have to help them correct it but I don t expect my users to deal with security issues, if they accidentally grant inappropriate access to their files then I will be the one to notice and correct it. Here is a script I name btrfs-make-snapshot which has an optional first parameter -d to cause it do just display the btrfs commands it would run and not actually do anything. The second parameter is either minutes or days depending on whether you want to create a snapshot on a short interval (I use 15 minutes) or a daily snapshot. All other parameters are paths for subvolumes that are to be backed up: #!/bin/bash
set -e

# usage:
# btrfs-make-snapshot [-d] minutes days paths
# example:
# btrfs-make-snapshot minutes /home /mail

if [ "$1" == "-d" ]; then
BTRFS="echo btrfs"
shift
else
BTRFS=/sbin/btrfs
fi

if [ "$1" == "minutes" ]; then
DATE=$(date +%Y-%m-%d:%H:%M)
else
DATE=$(date +%Y-%m-%d)
fi
shift

for n in $* ; do
$BTRFS subvol snapshot -r $n $n/backup/$DATE
done Here is a script I name btrfs-remove-snapshots which removes old snapshots to free space. It has an optional first parameter -d to cause it do just display the btrfs commands it would run and not actually do anything. The next parameters are the number of minute based and day based snapshots to keep (I am currently experimenting with 100 100 for /home to keep 15 minute snapshots for 25 hours and daily snapshots for 100 days). After that is a list of filesystems to remove snapshots from. The removal will be from under the backup subvolume of the path in question. #!/bin/bash
set -e

# usage:
# btrfs-remove-snapshots [-d] MINSNAPS DAYSNAPS paths
# example:
# btrfs-remove-snapshots 100 100 /home /mail

if [ "$1" == "-d" ]; then
BTRFS="echo btrfs"
shift
else
BTRFS=/sbin/btrfs
fi

MINSNAPS=$1
shift
DAYSNAPS=$1
shift

for DIR in $* ; do
BASE=$(echo $DIR cut -c 2-200)
for n in $(btrfs subvol list $DIR grep $BASE/backup/.*: head -n -$MINSNAPS sed -e "s/^.* //"); do
$BTRFS subvol delete /$n
done
for n in $(btrfs subvol list $DIR grep $BASE/backup/ grep -v : head -n -$MINSNAPS sed -e "s/^.* //"); do
$BTRFS subvol delete /$n
done
done A Warning The Debian/Wheezy kernel (based on the upstream kernel 3.2.32) doesn t seem to cope well when you run out of space by making snapshots. I have a filesystem that I am still trying to recover after doing that. I ve just been buying larger storage devices for systems while migrating them to BTRFS, so I should be able to avoid running out of disk space again until I can upgrade to a kernel that fixes such bugs.

have I said before that we are running out of easy RC bugs? the list is a bit short this week:

• #677861 src:lftp: "lftp: FTBFS[kfreebsd-i386]: error: conflicting declaration 'typedef __int32_t gl_intptr_t'"
  sponsor upload to t-p-u from Ivo De Decker
• #680100 libatomic-ops: "libatomic-ops/powerpc: FTBFS: eats all disk space in the known universe (almost, anyway)"
  backport fix for t-p-u, upload with maintainer's and release team's permission
• #691015 conserver-client: "conserver-client: fails to upgrade from testing: Can't locate Tie/Hash/NamedCapture.pm in @INC"
  update regexp to use numbered capture groups, upload to DELAYED/2

to make up for this poor performance, I've uploaded ~14 new lib.*-perl packages to the NEW queue yesterday & today :)

Why? I switched to Jekyll but it wasn't a fast decision. I really tried to give WordPress a chance, again, but I'm not getting along pretty well with PHP since like 2 years ago. In fact, is a headache to maintain on my server. In the course of the years I have had several downtimes on my PHP-based sites because of changes like PHP-FPM. Some sites just dropped working for me while others kept doing OK. This is in the same server, with the same database and the same codebase (WordPress). After a lot of time programming on Node.js and been learning more Ruby (CoffeeScript made me love the syntax) I just got tired of using more resources of what I really needed, and with a regular CMS that means a database. Why would I need a database if I just update like once a day? For a comment-system? There's Disqus already, and people can just comment you on Twitter or make a Branch for it, or just write you an email! Plain simple. Jekyll So there was Jekyll, a pretty nice tool creating static-sites very fast and easy to learn. It didn't needed a database and because all is static is pretty fast. Even the hosting requirements got lowered to the point I could (and I did) just upload this to GitHub and served it from there as a GitHub Page! One more thing One thing made me decided to upload to GitHub as well was Prose.io, a simple text-editor for GitHub with support for MarkDown made by the guys @ developmentseed.com. So while it was made for being updated on my terminal I could just do it as well from the web, from anywhere with just my GitHub account and having previews on-the-fly. The downside Since I don't control this server I couldn't setup redirections for my old feeds. Of course, I could have just copied the same pattern in a folder and put there an XML feeding the category X, but I think is easier to update the planets being feeded to the new URIs. On the bright-side I even could put pretty-permalinks to work! So the structure for my old links kept the same.

A long time ago, I spent many many hours playing Frontier (Elite 2) on the family Atari ST. It was nice-looking (oh, those sunrises on distant planets!), playable, varied, had ample room for progression, very different possible roles, a huge universe of which nobody could ever hope of exploring more than a tiny fraction, and it was basically the first game I encountered with no set goals. You are a spaceship pilot in a galaxy colonised left and right, and it's up to you to decide what you want the game to be. Pirate, headhunter, mercenary, trader, taxi, explorer, there's no end goal to achieve except what you fix for yourself. That was, for me as a youngster, a mind-opening experience, which I never felt as strongly since (with the possible exception of the Creatures game, but I barely tried it). Then came modernity, the PC, and the First Encounters sequel (Elite 3). It added moderate complexity to the gameplay, some sort of political evolution among the factions and a kind of plot so the galaxy was no longer quite so static, but the end results were still a bit disappointing. It felt like Frontier with a few not-exactly-overwhelming textures stuck onto the ships, a loose plot arc, and a third faction, the Alliance of Independent Systems, beyond the Federation and the Empire. Not that exciting, although the name of the Alliance's capital star system stayed in my mind. So, like many others, I came to wait for Elite 4. That wait started in 1995, more than seventeen years ago. During that time, Duke Nukem Forever was announced, promised, developed, postponed, abandoned, found again, developed again, and it was even actually released in 2011, while Elite 4 went into the same kind of hell only without the actual release (yet). At some point I started trying various substitutes to help pass time. There was Parsec which was a nice-looking space combat simulator, multiplayer and all, but never felt like finished. Development slowed down to a thin trickle, and it seems like it only recently started again as Open Parsec. Hopefully it won't wither off again, but since it's only focused on combat, it's not exactly the same spirit as Elite. There's Oolite, but it looks and feels very much like a lightly-modernised Frontier. Better textures, sure, but the UI still looks like it got taken from the times where games mentioned they required an EGA/VGA video card. There's Vega Strike. It's very promising too, and I did try it at some point, but it was crashing too often for my tastes. Development seems to have slowed down, but not stopped; maybe it's time I gave it another go. I just recently found out about Pioneer, and I admit I'm excited. Really excited. It feels like like Frontier felt at the time. It's not finished yet (the on-planet cities in particular look a bit weird), but from the alpha I tried and the videos on the website it's really impressive. I'm going to keep an eye on that, knowing I could spend quite some time in there even before it's quite finished, if only to fly a spaceship in the canyons of Europa with Jupiter rising in the distance. I'll let you know if I find a black monolith. But the main event triggering this blog post is that apparently Elite 4 is not dead (yet). Much better, it's under heavy development, under the Elite Dangerous name, and the current state seems to be rather good already. And Frontier Developments, the company behind it, opened a Kickstarter campaign to fund the project. The screenshots and the videos look impressive, and the interviews of the lead developer imply that the gameplay will be awesome. The galaxy will apparently be really dynamic, and the player's actions could really influence its geopolitical (galactopolitical?) evolution, the profitability of trade routes, the prevalence and repartition of space pirates in various sectors, and so on. Combine with a multi-player mode, and this could become the greatest game ever made (for my personal taste at least). So I'm very much hoping the funding campaign reaches its target. The release date is set to March 2014. It's going to be a long wait, fraught with incertitudes. But then again, Duke Nukem Forever was eventually released, wasn't it? [No, I never played the original Elite. I'm not that old.]

I just watched an interesting TED talk by Pankaj Ghemawat (of which the most important points are summarised in a TED blog post) about the world not being as globalised as people expect [1]. One point is that only 2% of traditional voice phone calling minutes (and ~6% when you include VOIP) are for international calls which is less than most people expect. After reading that it occurred to me that most of the included value in my mobile phone contract goes unused, I pay for extra data transfer and it includes voice calling credit that I don t use. So out of $450 of included calls I typically use much less than $50. $400 of calls to even the most expensive countries is about 100 minutes of talking. So the logical thing to do is to find people in other countries to call. If you are involved in the FOSS community and would like to speak to me then send me an email with your phone number, time zone, and a range of times that are convenient. I won t make any promises about calling you soon (I could use up my monthly credit on a single call), but I will call you eventually. I will also send email to some people I know by email and suggest a chat. Some years ago I did this with people who were involved in SE Linux development and it seemed to help the development of the SE Linux community. Also if anyone in Australia wants to speak to me then that s OK too. While Pankaj s talk inspired me to call people I m not dedicated to calling other countries.

• [1] http://tinyurl.com/czmmwxj

Have you ever heard someone talking extensively about Cloud Computing or generally Clouds? and have you ever noticed the fact many people (even the ones who present themselves as experts) don t really understand what a Cloud is at all? That happened to me multiple times and one of the most common misunderstandings is many see the Cloud as something being on the internet. Many companies add a little logo representing a cloud on their frontpage and without a single change on their infrastructure (but surely with a price increment) they start calling their products as being on the Cloud. Given the lack of knowledge about this specific topic people tend to buy the product presented as being on the Cloud without understanding what they really bought. But what Cloud Computing really means? it took several years and more than fifteen drafts to the National Institute of Standards and Technology s (NIST) to find a definition. The final accepted proposal:

Cloud computing is a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction.

The above definition requires a few more clarifications specifically when it comes to understand where should we focus on while checking for a Cloud Computing solution. A few key points:

1. On-demand self-service: every consumer will be able to unilaterally provision multiple computing capabilities like server time, storage, bandwidth, dedicated RAM or CPU without requiring any sort of human interaction from their respective Cloud providers.
2. Rapid elasticity and scalability: all the computing capabilities outlined above can be elastically provisioned and released depending on how much demand my company will have in a specific period of time. Suppose the X company is launching a new product today and it expects a very large number of customers. The X company will add more resources to their Cloud for the very first days (where they suppose the load to be very high) and then it ll scale the resources back as they were before. Elasticity and scalability permit the X company to improve and enhance their infrastructure when they need it with an huge saving in monetary terms.
3. Broad network access: capabilities are available over the network and accessed through standard mechanisms that promote use by heterogeneous thin or thick client platforms (e.g., mobile phones, tablets, laptops, and workstations).
4. Measured service: Cloud systems allow maximum transparency between the provider and the consumer, the usage of all the resources is monitored, controlled, and reported. The consumer knows how much will spend, when and in how long.
5. Resource pooling: each provider s computing resources are pooled to serve multiple consumers at the same time. The consumer has no control or knownledge over the exact location of the provided resources but may be able to specify location at a higher level of abstraction (e.g., country, state, or datacenter).
6. Resources price: when buying a Cloud service make sure the cost for two units of RAM, storage, CPU, bandwidth, server time is exactly the double of the price of one unit of the same capability. An example, if a provider offers you one hour of bandwitdh for 1 Euro, the price of two hours will have to be 2 Euros.

The Cloud Computing technology is reasonably the future but can we trust Cloud providers? Are we sure that no one will ever have access to our files except us? and what about governments interested in acquiring a specific customer data hosted on the Cloud? I always suggest to read deeply both the Privacy Policy and Terms of Use of a certain service before signing in especially when it comes to choose a Cloud storage provider. Many providers have the same aspect, they seem to provide the same resources, the same amount of storage for the same price but legally they may present different problems, and that is the case of Spideroak vs Dropbox. Quoting from the Dropbox s Privacy Policy:

Compliance with Laws and Law Enforcement Requests; Protection of DropBox s Rights. We may disclose to parties outside Dropbox files stored in your Dropbox and information about you that we collect when we have a good faith belief that disclosure is reasonably necessary to (a) comply with a law, regulation or compulsory legal request; (b) protect the safety of any person from death or serious bodily injury; (c) prevent fraud or abuse of DropBox or its users; or (d) to protect Dropbox s property rights. If we provide your Dropbox files to a law enforcement agency as set forth above, we will remove Dropbox s encryption from the files before providing them to law enforcement. However, Dropbox will not be able to decrypt any files that you encrypted prior to storing them on Dropbox.

It s evident that Dropbox employees can access your data or be forced by legal process to turn over your data unencrypted. On the other side, Spideroak on its latest update to its Privacy Policy states that data stored on their Cloud is encrypted and inaccessible without user s key, which is stored locally on user s computers. And what about the latest research paper, titled Cloud Computing in Higher Education and Research Institutions and the USA Patriot Act written by the legal experts of the University of Amsterdam s Institute for Information Law stating the anti-terror Patriot Act could be theoretically used by U.S. law enforcement to bypass strict European privacy laws to acquire citizen data within the European Union without their consensus? The only requirement for the data acquisition is the provider being an U.S company or an European company conducting systematic business in the U.S. For example an Italian company storing their documents (protected by the European privacy laws and under the general Italian jurisdiction) on a provider based in Europe but conducting systematic business in the United States, could be forced by U.S. law enforcement to transfer data to the U.S. territory for inspection by law enforcement agencies. Does someone really care about the privacy of companies, consumers and users at all? or better does privacy exists at all for the millions of the people that connect to the internet every day?

Working with open hardware Some weeks ago at LinuxCon EU in Barcelona I talked about how to use QEMU to improve the reliability of device drivers. At Igalia we have been using this for some projects. One of them is the Linux IndustryPack driver. For this project I virtualized two boards: the TEWS TPCI200 PCI carrier and the GE IP-Octal 232 module. This work helped us find some bugs in the device driver and improve its quality. Now, those two boards are examples of products available in the market. But fortunately we can use the same approach to develop for hardware that doesn t exist yet, or is still in a prototype phase. Such is the case of a project we are working on: adding Linux support for this FMC Time-to-digital converter. This piece of hardware is designed by CERN and is published under the CERN Open Hardware Licence, which, in their own words is to hardware what the General Public Licence (GPL) is to software . The Open Hardware repository hosts a number of projects that have been published under this license. Why we use QEMU So we are developing the device driver for this hardware, as my colleague Samuel explains in his blog. I m the responsible of virtualizing it using QEMU. There are two main reasons why we want to do this:

1. Limited availability of the hardware: although the specification is pretty much ready, this is still a prototype. The board is not (yet) commercially available. With virtual hardware, the whole development team can have as many boards as it needs.
2. Testing: we can test the software against the virtual driver, force all kinds of conditions and scenarios, including the ones that would probably require us to physically damage the board.

While the first point might be the most obvious one, testing the software is actually the one we re more interested in. My colleague Miguel wrote a detailed blog post on how we have been using QEMU to do testing. Writing the virtual hardware Writing a virtual version of a particular piece of hardware for this purpose is not as hard as it might look. First, the point is not to reproduce accurately how the hardware works, but rather how it behaves from the operating system point of view: the hardware is a black box that the OS talks to. Second, it s not necessary to have a complete emulation of the hardware, there s no need to support every single feature, particularly if your software is not going to use it. The emulation can start with the basic functionality and then grow as needed. The FMC TDC, for example, is an FMC card which is in our case connected to a PCIe bridge called SPEC (also available in the Open Hardware repository). We need to emulate both cards in order to have a working system, but the emulation is, at the moment, treating both as if they were just one, which makes it a bit easier to have a prototype and from the device driver point of view doesn t really make a difference. Later the emulation can be split in two as I did with with TPCI200 and IP-Octal 232. This would allow us to support more FMC hardware without having to rewrite the bridging code. There s also code in the emulation to force different kind of scenarios that we are using to test if the driver behaves as expected and handles errors correctly. Those tests include the simulation of input in the any of the lines, simulation of noise, DMA errors, etc. And we have written a set of test cases and a continuous integration system, so the driver is automatically tested every time the code is updated. If you want details on this I recommend you again to read Miguel s post.

A the new Rcpp release 0.10.1 arrived this morning on CRAN (as already has Windows binaries) and in Debian. This is a follow-up to the recent 0.10.0 release which extends the exciting new Rcpp-attributes and Rcpp-sugar work further, and as in a number of other areas as detailed below in the NEWS sections. This release brings an change to some of the binary interfaces. If you have packages using Rcpp, you will most likely have to reinstall them from source. Some change were made to const correctness as well as other aspects, and it seems that we have temporarily broken the excellent RcppEigen and RcppOctave packages. We are looking into this, and are sorry about the bug. The complete NEWS entry for 0.10.1 is below; more details are in the ChangeLog file in the package and on the Rcpp Changelog page.

Changes in Rcpp version 0.10.1 (2012-11-26)

• Changes in Rcpp sugar:
  • New functions: setdiff, union_, intersect setequal, in, min, max, range, match, table, duplicated
  • New function: clamp which combines pmin and pmax, e.g. clamp( a, x, b) is the same as pmax( b, pmin(x, a) )
  • New function: self_match which implements something similar to match( x, unique( x ) )
• Changes in Rcpp API:
  • The Vector template class (hence NumericVector ...) get the is_na and the get_na static methods.
  • New helper class no_init that can be used to create a vector without initializing its data, e.g. : IntegerVector out = no_init(n) ;
  • New exception constructor requiring only a message; stop function to throw an exception
  • DataFrame gains a nrows method
• Changes in Rcpp attributes:
  • Ability to embed R code chunks (via specially formatted block comments) in C++ source files.
  • Allow specification of argument defaults for exported functions.
  • New scheme for more flexible mixing of generated and user composed C++ headers.
  • Print warning if no export attributes are found in source file.
  • Updated vignette with additional documentation on exposing C++ interfaces from packages and signaling errors.
• Changes in Rcpp modules:
  • Enclose .External invocations in BEGIN_RCPP/END_RCPP
• Changes in R code :
  • New function areMacrosDefined
  • Additions to Rcpp.package.skeleton:
    • attributes parameter to generate a version of rcpp_hello_world that uses Rcpp::export.
    • cpp_files parameter to provide a list of C++ files to include the in the src directory of the package.
• Miscellaneous changes:
  • New example 'pi simulation' using R and C++ via Rcpp attributes

Thanks to CRANberries, you can also look at a diff to the previous release 0.10.0. As always, even fuller details are on the Rcpp Changelog page and the Rcpp page which also leads to the downloads, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page

Julian Treasure gave an informative TED talk about The 4 Ways Sound Affects US [1]. Among other things he claims that open plan offices reduce productivity by 66%! He suggests that people who work in such offices wear headphones and play bird-songs. Naked Capitalism has an interesting interview between John Cusack and Jonathan Turley about how the US government policy of killing US citizens without trial demonstrates the failure of their political system [2]. Washington s blog has an interesting article on the economy in Iceland [3]. Allowing the insolvent banks to go bankrupt was the best thing that they have ever done for their economy. Clay Shirky wrote an insightful article about the social environment of mailing lists and ways to limit flame-wars [4]. ZRep is an interesting program that mirrors ZFS filesystems via regular snapshots and send/recv operations [5]. It seems that it could offer similar benefits to DRBD but at the file level and with greater reliability. James Lockyer gave a movingTEDx talk about his work in providing a legal defence for the wrongly convicted [6]. This has included overturning convictions after as much as half a century in which the falsely accused had already served a life sentence. Nathan Myers wrote an epic polemic about US government policy since 9-11 [7]. It s good to see that some Americans realise it s wrong. There is an insightful TED blog post about TED Fellow Salvatore Iaconesi who has brain cancer [8]. Apparently he had some problems with medical records in proprietary formats which made it difficult to get experts to properly assess his condition. Open document standards can be a matter of life and death and should be mandated by federal law. Paul Wayper wrote an interesting and amusing post about Emotional Computing which compares the strategies of Apple, MS, and the FOSS community among other things [9]. Kevin Allocca of Youtube gave an insightful TED talk about why videos go viral [10]. Jason Fried gave an interesting TED talk Why Work Doesn t Happen at Work [11]. His main issues are distraction and wasted time in meetings. He gives some good ideas for how to improve productivity. But they can also be used for sabotage. If someone doesn t like their employer then they could call for meetings, incite managers to call meetings, and book meetings so that they don t follow each other and thus waste more of the day (EG meetings at 1PM and 3PM instead of having the second meeting when the first finishes). Shyam Sankar gave an interesting TED talk about human computer cooperation [12]. He describes the success of human-computer partnerships in winning chess tournaments, protein folding, and other computational challenges. It seems that the limit for many types of computation will be the ability to get people and computers to work together efficiently. Cory Doctorow wrote an interesting and amusing article for Locus Magazine about some of the failings of modern sci-fi movies [13]. He is mainly concerned with pointless movies that get the science and technology aspects wrong and the way that the blockbuster budget process drives the development of such movies. Of course there are many other things wrong with sci-fi movies such as the fact that most of them are totally implausible (EG aliens who look like humans). The TED blog has an interesting interview with Catarina Mota about hacker spaces and open hardware [14]. Sociological Images has an interesting article about sporting behaviour [15]. They link to a very funny youtube video of a US high school football team who make the other team believe that they aren t playing until they win [16]

• [1] http://www.ted.com/talks/lang/en/julian_treasure_the_4_ways_sound_affects_us.html
• [2] http://www.nakedcapitalism.com/2012/08/john-cusack-jonathan-turley-on-obamas-constitution.html
• [3] http://www.washingtonsblog.com/2012/08/top-economists-iceland-did-it-right-everyone-else-is-doing-it-wrong.html
• [4] http://www.shirky.com/writings/group_user.html
• [5] http://www.bolthole.com/solaris/zrep/
• [6] http://www.youtube.com/watch?v=k22MJpgTUiM
• [7] http://www.advogato.org/person/ncm/diary.html?start=434
• [8] http://tinyurl.com/9zzsg4s
• [9] http://www.mabula.net/tbfw/blosxom.cgi/2012/09/17
• [10] http://www.ted.com/talks/lang/en/kevin_allocca_why_videos_go_viral.html
• [11] http://www.ted.com/talks/lang/en/jason_fried_why_work_doesn_t_happen_at_work.html
• [12] http://www.ted.com/talks/shyam_sankar_the_rise_of_human_computer_cooperation.html
• [13] http://www.locusmag.com/Perspectives/2012/09/cory-doctorow-why-science-fiction-movies-drive-me-nuts/
• [14] http://blog.ted.com/2012/09/21/tinker-tailor-solderer-fellows-friday-with-catarina-mota/
• [15] http://thesocietypages.org/socimages/2010/12/28/context-how-do-you-play-football/
• [16] http://www.youtube.com/watch?v=0UIdI8khMkw

Related posts:

1. Links April 2012 Karen Tse gave an interesting TED talk about how to...
2. Links March 2012 Washington s Blog has an informative summary of recent articles about...
3. Links November 2011 Forbes has an interesting article about crowd-sourcing by criminals and...

The Free and Open successor to the Clonk computer game series OpenClonk is now available in Debian sid and Ubuntu raring: just use sudo apt-get install openclonk to install it (backports to Ubuntu precise and quantal can be found in the OpenClonk Releases PPA).


It's still under development, but as you can see it's also already quite functional. The engine was made available freely under the ISC license after the original author of Clonk was no longer interested in creating new successors. Many of the old volunteer contributors then picked it up. Be aware that same computer multiplayer, which was available in the older parts of the series, is gone. If you play on the network, you need to ensure that both of you have the same version of the game.

Another decent article on online privacy was published by the NYT this morning. The article documents the collection and centralized aggregation of consumer data online. It ends with a strong message about consumers becoming the "product" - which echoes what Noam Chomsky and Ben Bagdikian have been saying for decades. The problem with this critique is that it doesn't go far enough. Chomsky and Bagdikian didn't say readers are the product of modern journalism and that's bad because nobody wants to be a product. They critiqued the model because it produces terrible journalism and democracy suffers as a result. Similarly, the online trading of audiences and the invasion of our privacy that results has more important implications than "don't be a sucker" which seems to be the gist of the privacy argument these days. Honestly, I don't stay awake at night worrying that Nike knows I'm looking for a pair of sneakers, or even that they know my income. I do stay awake at night worrying that people are tagging my photo on Facebook, which could allow the New York Police Dept to submit a photo of protesters to Facebook and get a list of names and addresses of the people in the photo. Or it could allow the police to track my movements via existing networks of surveillance cameras by matching my image to my name. Would that require a search warrant? How would that impact my trust in my government to know that my movements are being tracked? Or worse, to know they might be tracked but I'll never know if they are or aren't? Many governments (including the US) have sordid histories of infiltrating political organizations and intentionally sowing dissent. Having access to activists' purchasing habits provides a treasure trove of material for this purpose that previously was labor intensive for cops to collect. It also provides leverage during interrogation, a time of heightened emotional vulnerability where the mere mention of a private detail of one's life can have a powerful impact. How can we prevent law enforcement from having access to these details of our lives? And then there's the dragnet. A central database of consumer habits provides a tantalizing collection of information that could be searched for people who fit a profile of a crime. Is solving the crime at hand worth the invasion of privacy that results from people being investigating only based on their consumer habits? How would we measure the impact an investigation like this would have on our confidence in democracy and government? The corporate advertisers promoting this kind of data collection argue that these problems are the price of progress. However, the ideology behind these technology developments is capitalism, not some intrinsic aspect of the Internet or network communications. Historically, the Internet has developed based on a drive toward de-centralization. Email and web sites, the bedrock of Internet activity, are based on these principles: anyone can add their own email server or web server and everyone can seamlessly access it. In contrast, the logic of capitalism as it relates to the Internet, is one of centralization and aggregating data. Facebook and Google want you to use their platforms for all your online activities so they have as complete a picture as possible of what you like and do. As the New York Times article describes, companies like Rubicon want to track all your movements and aggregate that data for profit. We have a choice. We can carefully evaluate our use of corporate services and consider the implications it has not only on our lives but on society as a whole. And, we can choose to use and support services that promote open standards and interoperability that will respect our personal privacy values. Want a practical example? Friendica is software designed to allow you to post status alerts, photos, videos and other information about yourself in a way similar to Facebook. However, it has one big difference: you can host your account on any server running the software anywhere in the world. You can still "friend" people on different servers, but we don't have to all trust the same organization to host it. Furthermore, Friendica allows you to post once and automatically cross-post to your Twitter, Facebook or other similar services. Sound good? You are welcome to try out an account on the May First/People Link install: https://friends.mayfirst.org/.

A short announcement to mark the release of the first stable OpenID provider for Drupal. Started by James Walker around 4 years ago, I have since then got commit access and taken care of the module for the last 2 years. And while it was not an easy task, with the help of the community, this module was able to mature to a fully featured, flexible and standard implementation during that time. One has to wonder however, is it too late for OpenID?

A bit of history Drupal was one of the pioneers in OpenID support. Back in 2006 when the protocol was still in its infancy (less than a year old!), Drupal was adding OpenID support to its 4.7 release through a contrib module and two years later, the Drupal 6.0 release was sporting OpenID in core, although not without significant interoperability issues. Still, we could say that rapid adoption by such a popular CMS as Drupal really boosted OpenID's adoption. At about the same time, Yahoo, Sourceforge, Myspace, Microsoft and Google would all join the OpenID bandwagon. On the OpenID "provider" side of things, things were not so great however. Back when I started experimenting with this, it wasn't actually that easy to host your own OpenID provider, to allow your users to login to other OpenID-enabled sites. While there are a lot of different libraries to provide OpenID support in your application there was not (and still isn't) any software that gives you an OpenID provider and is dedicated to that. No official reference implementation either, other than the code from the one of the founding companies of OpenID, JanRain. There was an OpenID provider module in Drupal, but there was no official release done until a two beta releases in 2010.

Koumbit's community involvement It is about that time I started getting involved in the project. Koumbit needed an easy way for our workers to log into multiple sites without too much trouble, so I started looking at OpenID provider implementations. The lack of dedicated software somehow made the Drupal implementation a natural fit, and I started experimenting with the module, filing bugs and reviewing patches, as there were a lot of interoperability and design issues with the provider. After a while, walkah was kind enough to give me full access to the project and I started doing regular releases (every few months), culminating in today's official 1.0 release for both Drupal 6 and Drupal 7, which I am very proud and happy about. While I am here, I want to send many thanks to the people behind the Drupal 7 port: paranojik, mikecar, xamanu and testing from neizod. A lot of issues were also patched, reviewed and tested by numerous contributors that are unfortunately difficult to track down other than reading the commit log. So we at Koumbit have been running our own OpenID provider for a good while now, and it has the key advantage of being connected to our LDAP directory (with the D6 LDAP integration module so it is, in a sense, a central point for authentication at Koumbit. Very often people don't clearly understand what their LDAP password does, and I can easily remind them "it's the OpenID thing", or "try your password in the OpenID site" if they think they have forgotten it. So much goes on in the web browser these days that it makes everything much easier.

The problem with Drupal's OpenID provider However, it is kind of embarrassing that it took so long. More than 6 years after the OpenID protocol was written, we have a stable OpenID provider. Now, this may be because I am overly conservative about stable releases, but I don't think so: the OpenID provider module had, for a long time, serious interoperability issues with OpenID sites. There was also a clear lack of features on the OpenID provider, a gap that various third party modules, now documented on the OpenID provider homepage are now trying to fill. There is also serious security issues with the protocol itself. The complexity and lack of clarity of the OpenID specification have rendered my work quite difficult at time, as the protocol has significant ambiguities and no clear reference implementation. OpenID is unfortunately renowned to be vulnerable to phishing and while Koumbit's use of the HTTPS protocol alleviates some of those issues (especially the redirect session hijack), tools like sslstrip can bypass those protections easily. The privacy aspect is also problematic, as the user is usually not free to choose which information from his or her profile is divulged to the site they are logging into. Of course those issues may be overlooked if the security trade off is not worth it. In the case of Koumbit, we use OpenID on a lot of client sites, but usually nothing critical, and we train our users to pay attention to the HTTPS icon when login in their OpenID account.

Are there alternatives? But unfortuantely, the reality is that there are not really any alternatives to OpenID out there. Mozilla's new BrowserID initiative (now rebranded Persona, boy those guys sure love rebranding..) is interesting, but actually has even more flaws than OpenID itself. Not only is it not resolving the phishing problem, it actually creates new problems, including my personal favorite, trusting server-side Javascript code, which OpenID is not using that much. I should also mention the Shibboleth module which has a surprising number of installs recorded on Drupal.org, but that depends on the much more complicated Shibboleth federation, for which there is no server implementation for Drupal, from what I understand. Oauth, on the other end, is even more widely used, and has server-side components like Oauth Login Provider and Services 3.x. We could also simply give up and use our souls stored at Facebook, Google and Twitter for authentication everywhere. Maybe OpenID opened the door for those corporations to see that there is a real opportunity in leveraging user's profile to track their online activity. It is not something that OpenID was necessarily designed for but it's an idea that certainly permeates Facebook connect and "like" buttons or Twitter and Google online business strategies, where you are the product sold to advertisers. I personally believe we should keep the struggle for distributed and standard identity mechanisms for the web that respect users' privacy. With Drupal's OpenID Provider, it is possible to create a secure, privacy-aware identity provider, although I still have to perform that damn security audit. Experiments with those third party modules should also be encouraged and I welcome any feedback on those modules. While I do not plan to attend OpenID conferences or do significant development on the OpenID provider right now, I will certainly make sure Koumbit keeps its commitment to maintaining stewardship on the OpenID provider module in Drupal. I encourage people to try it out, submit issues and patches to make it even better. Long life to OpenID! PS: if people are interested in funding that security audit, I would be happy to perform this through my regular work at Koumbit.org, just file a request at services@koumbit.org.

Another month, another update to http.debian.net. This time around most of the work was done outside the redirector's code base, as strange as it may sound.

The redirector heavily relies on the mirrors doing at least a couple of things right, for the rest it can and does compensate. When it needs to compensate, certain requests are redirected to automatically-detected good mirrors, thus avoiding mirrors that might work fine for some parts of the day but cause headaches during the rest.

So, part of the work done since the last update was to prod more mirror administrators to upgrade to the latest version of ftpsync. This reduces the number of mirrors for which compensation is needed in order to avoid errors during installations and upgrades. Hopefully, no additional work is needed for the redirector to notice the upgrades. This results in immediate improvements.

However, not all mirrors comply with the bare minimum requirements. As stated in my previous blog post, running rsync once is not enough. When mirrors break these assumptions they lead to the "bad apple" effect. The effects in this case are temporary errors, as experienced by some people. The interesting part of those issues is that the affected population may quickly change given the redirector's use of geo location and the way it creates mirror subsets.

As interesting as the distribution of the effects may be, they are not really welcome. So I put together some code to attempt to detect the bad apples. This resulted in a list of mirrors that have now been disabled in the redirector and whose administrators are going to be contacted so that they comply with the minimum requirements. Given that detection is time-sensitive, there's no 100% guarantee that all of them have been identified so far. The code to detect them will have to be adapted and integrated into the redirector's code base to be proactive on avoiding this kind of issues.

Last but not least, the redirector is now using a database of AS peers for better (re-)routing. This is the next move towards a decision making based more on network location/topology than in geographic location. This first use of a peers database is limited to IPv4 and is based on a recent routing table dump and on feedback provided by interested people. If you are a mirror or network administrator, or you are familiar with the topology of your network please drop me an email so that the redirector can make a better use of your peering agreements.

N.b. in the case of the database, the term peer may also include transit providers. It is used to refer to and establish a relationship between two AS(N)s.


Feedback is, as always, welcome. I read each and every email but it may take me some time to get to it, or reply.

Since converting the phpMyAdmin documentation to new format and updating it's content, I was quite frightened how this will affect translated versions of it. Fortunately giving it more attractive format and more advertisement lead to increased contributions. Since Friday, we already have one language which has reached 100% - Spanish. Mat as Bellone simply did there awesome job and went through all fuzzy and new strings in just few days. Now I'm looking forward to other contributors as well, I hope we will be able to deliver translated manual for phpMyAdmin 4.0 in many languages. I'll definitely try to translate to Czech, but any help with that is welcome.

Filed under: English Phpmyadmin 2 comments Flattr this!

Quite on updated schedule (ie. one month later than originally planned), Weblate 1.3 has been released today. It comes with lot of improvements, especially in support of non gettext files, new quality checks and improved performance. Full list of changes for 1.3:

• Compatibility with PostgreSQL database backend.
• Removes languages removed in upstream git repository.
• Improved consistency checks processing.
• Added new checks (BB code, XML markup and newlines).
• Support for optional rebasing instead of merge.
• Possibility to relocate Weblate (eg. to run it under /weblate path).
• Support for manually choosing file type in case autodetection fails.
• Better support for Android resources.
• Support for generating SSH key from web interface.
• More visible data exports.
• New buttons to enter some special characters.
• Support for exporting dictionary.
• Support for locking down whole Weblate installation.
• Checks for source strings and support for source strings review.
• Support for user comments for both translations and source strings.
• Better changes log tracking.
• Changes can now be monitored using RSS.
• Improved support for RTL languages.

You can find more information about Weblate on it's website, the code is hosted on Github. If you are curious how it looks, you can try it out on demo server. You can login there with demo account using demo password or register your own user. Ready to run appliances will be soon available in SUSE Studio Gallery. Weblate is also being used https://l10n.cihar.com/ as official translating service for phpMyAdmin, Gammu, Weblate itself and others. If you are free software project which would like to use Weblate, I'm happy to help you with set up or even host Weblate for you (this will be decided case by case as my hosting space is limited).

Filed under: English Phpmyadmin Weblate 0 comments Flattr this!

As systems developer I like just being able to pick the most useful Perl modules from CPAN and preferably use quite recent versions of the modules I need. On the other hand our systems administrators prefer running Debian Stable and deploy software using .deb packages. I do not want to maintain hundred small packages by hand. I just want a single package with all needed the Perl dependencies for a given project. After looking at different possibilities for bundling Perl dependencies I finally found Carton written by Tatsuhiko Miyagawa. Given a list of your direct Perl dependencies, Carton can maintain the full dependency chain and manage which versions of each module you deploy. One of the nice features is that you get a reproducible set of versions each time and not just the latest, greatest, and recently broken versions. The initial step is to set Carton up to install the dependencies for my project. This part is done just like the Carton documentation describes. To ensure independence from whatever is available on CPAN I end by calling carton bundle. The next step is to write a small Perl module making sure you load modules from the right place. I end up with installing modules in /usr/lib/projectname. My module looks like this:

package My::Bundle::ProjectName;
=head1 NAME
My::Bundle::ProjectName - Bundle of CPAN modules used by ProjectName
=cut
our $VERSION = 1.0;
use Config;
use Carp;
use Carp::Heavy;  # Ensure that all parts of Carp are loaded before @INC mangling
use File::Spec;
our $BASE = '/usr/lib/projectname';
sub import  
    carp (__PACKAGE__ . " bundle not installed")
        unless -d $BASE;
    unshift @INC, 
        File::Spec->catdir( $BASE, 'lib', 'perl5' ),
        File::Spec->catdir( $BASE, 'lib', 'perl5', $Config archname );
 
1;

Now I need to install this module and all the modules maintained by Carton. I do this by a enhanced Module::Build script:

use v5.10.0;
use strict;
use warnings;
use Module::Build;
my $class = Module::Build->subclass(
    class => "My::Builder",
    code  => q 
        sub ACTION_build  
            my $self = shift;
            $self->SUPER::ACTION_build;
            !system 'carton', 'install', '--cached';
         
        sub ACTION_install  
            my $self = shift;
            $self->SUPER::ACTION_install;
            
            my $installdir = $self->destdir . "/usr/lib/projectname";
            system 'install', '-d', $installdir;
            !system 'cp', '-a', 'local/lib', $installdir;
         
     ,
);
$class->new(
    module_name       => 'My::Bundle::ProjectName',
    dist_author       => 'Peter Makholm <peter>',
    dist_version_from => 'lib/My/Bundle/ProjectName.pm',
    add_to_cleanup    => [ 'local/lib', 'local/bin' ],
)->create_build_script;

The last step is to create a .deb package. This is mostly just a standard debian/ directory using debhelper. You just need to override the dh_perl to pick up the correct binary dependencies. So this is my debian/rules:

#!/usr/bin/make -f
%:
        dh $@
override_dh_perl:
        dh_perl /usr/lib/projectname/lib

That s it. Then all the scripts and applications making up my project just needes to use My::Bundle::ProjectName and the rest is automatic. I am thinking about enhancing Carton to provide individual .deb packages instead of just installing the modules. This would make it quite easy to maintain the packages in a more The Debian Perl Group-compatible way. That could be cool.

TL;DR: There s a summary at the end of the article. Today I wanted to see why a dependency in a .deb-package from an external APT repository changed so that it became uninstallable. While dpkg-deb --info foobar.deb easily shows the control information, the changelog is in the filesystem part of the package. I could extract that one dpkg-deb, too, but I d have to extract either to some temporary directory or pipe it into tar which then can extract a single file from the archive and sent it to STDOUT:

dpkg-deb --fsys-tarfile foobar.deb   tar xOf - ./usr/share/doc/foobar/changelog.Debian.gz   zless

But that s tedious to type. The following command is clearly less to type and way easier to remember:

acat foobar.deb ./usr/share/doc/foobar/changelog.Debian.gz   zless

acat stands for archive cat is part of the atool suite of commands:

als

lists files in an archive.

$ als foobar.tgz
drwxr-xr-x abe/abe           0 2012-11-15 00:19 foobar/
-rw-r--r-- abe/abe          13 2012-11-15 00:20 foobar/bar
-rw-r--r-- abe/abe          13 2012-11-15 00:20 foobar/foo

acat

extracts files in an archive to standard out.

$ acat foobar.tgz foobar/foo foobar/bar
foobar/bar
bar contents
foobar/foo
foo contents

adiff

generates a diff between two archives using diff(1).

$ als quux.zip
Archive:  quux.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  2012-11-15 00:23   quux/
       16  2012-11-15 00:22   quux/foo
       13  2012-11-15 00:20   quux/bar
---------                     -------
       29                     3 files
$ adiff foobar.tgz quux.zip
diff -ru Unpack-3594/foobar/foo Unpack-7862/quux/foo
--- Unpack-3594/foobar/foo      2012-11-15 00:20:46.000000000 +0100
+++ Unpack-7862/quux/foo        2012-11-15 00:22:56.000000000 +0100
@@ -1 +1 @@
-foo contents
+foobar contents

arepack

repacks archives to a different format. It does this by first extracting all files of the old archive into a temporary directory, then packing all files extracted to that directory to the new archive. Use the --each (-e) option in combination with --format (-F) to repack multiple archives using a single invocation of atool. Note that arepack will not remove the old archive.

$ arepack foobar.tgz foobar.txz
foobar.tgz: extracted to  Unpack-7121/foobar'
foobar.txz: grew 36 bytes

apack

creates archives (or compresses files). If no file arguments are specified, filenames to add are read from standard in.

aunpack

extracts files from an archive. Often one wants to extract all files in an archive to a single subdirectory. However, some archives contain multiple files in their root directories. The aunpack program overcomes this problem by first extracting files to a unique (temporary) directory, and then moving its contents back if possible. This also prevents local files from being overwritten by mistake.

(atool subcommand descriptions from the atool man page which is licensed under GPLv3+. Examples by me.) I though miss the existence of an agrep subcommand. Guess why? atool supports a wealth of archive types: tar (gzip-, bzip-, bzip2-, compress-/Z-, lzip-, lzop-, xz-, and 7zip-compressed), zip, jar/war, rar, lha/lzh, 7zip, alzip/alz, ace, ar, arj, arc, rpm, deb, cab, gzip, bzip, bzip2, compress/Z, lzip, lzop, xz, rzip, lrzip and cpio. (Not all subcommands support all archive types.) Similar Utilities There are some utilities which cover parts of what atool does, too: Tools from the mtools package Yes, they come from the handle MS-DOS floppy disks tool package, don t ask me why. :-)

uz

gunzips and extracts a gzip d tar d archives

Advantage over aunpack: Less to type. :-)

Disadvantage compared to aunpack: Supports only one archive format.

lz

gunzips and shows a listing of a gzip d tar d archive

Advantage over als: One character less to type. :-)

Disadvantage compared to als: Supports only one archive format.

unp unp extracts one or more files given as arguments on the command line.

$ unp -s
Known archive formats and tools:
7z:           p7zip or p7zip-full
ace:          unace
ar,deb:       binutils
arj:          arj
bz2:          bzip2
cab:          cabextract
chm:          libchm-bin or archmage
cpio,afio:    cpio or afio
dat:          tnef
dms:          xdms
exe:          maybe orange or unzip or unrar or unarj or lha 
gz:           gzip
hqx:          macutils
lha,lzh:      lha
lz:           lzip
lzma:         xz-utils or lzma
lzo:          lzop
lzx:          unlzx
mbox:         formail and mpack
pmd:          ppmd
rar:          rar or unrar or unrar-free
rpm:          rpm2cpio and cpio
sea,sea.bin:  macutils
shar:         sharutils
tar:          tar
tar.bz2,tbz2: tar with bzip2
tar.lzip:     tar with lzip
tar.lzop,tzo: tar with lzop
tar.xz,txz:   tar with xz-utils
tar.z:        tar with compress
tgz,tar.gz:   tar with gzip
uu:           sharutils
xz:           xz-utils
zip,cbz,cbr,jar,war,ear,xpi,adf: unzip
zoo:          zoo

So it s very similar to aunpack, just a shorter command and it supports some more exotic archive formats which atool doesn t support. Also part of the unp package is ucat which does more or less the same as acat, just with unp as backend. dtrx From the man page of dtrx:

In addition to providing one command to extract many different archive types, dtrx also aids the user by extracting contents consistently. By default, everything will be written to a dedicated directory that s named after the archive. dtrx will also change the permissions to ensure that the owner can read and write all those files. Supported archive formats: tar, zip (including self-extracting .exe files), cpio, rpm, deb, gem, 7z, cab, rar, and InstallShield. It can also decompress files compressed with gzip, bzip2, lzma, or compress.

dtrx -l lists the contents of an archive, i.e. works like als or lz. dtrx has two features not present in the other tools mentioned so far:

• It can extract metadata instead of the normal contents from .deb and .gem files.
• It can extract archives recursively, i.e. can extract archives inside of archives.

Unfortunately you can t mix those two features. But you can use the following tool for that purpose: deepfind deepfind is a command from the package strigi-utils and recursively lists files in archives, including archives in archives. I ve already written a detailed blog-posting about deepfind and its friend deepgrep. tardiff tardiff was written to check what changed in source code tarballs from one release to another. By default it just lists the differences in the file lists, not in the files contents and hence works different than adiff. Summary atool and friends are probably the first choice when it comes to DWIM archive handling, also because they have an easy to remember subcommand scheme. uz and lz and the shortest way to extract or list the contents of a .tar.gz file. But nothing more. And you have to install mtools even if you don t have a floppy drive. unp comes in handy for exotic archive formats atool doesn t support. And it s way easier to remember and type than aunpack. dtrx is neat if you want to extract archives in archives or if you want to extract metadata from some package files with just a few keystrokes. For listing all files in recursive archives, use deepfind.

Rcpp release 0.10.0 is now on CRAN and being uploaded to Debian. This is a new feature release, and we are very exciting about the changes, notably Rcpp attributes which make using C++ from R even easier than inline (see below as well as the new vignette for details and first examples), the extensions to Rcpp modules (see below) and more as for example new Rcpp sugar functions, a new error output device syncing to R, and a new namespace R> for the statistical functions from Rmath.h. The complete NEWS entry for 0.10.0 is below; more details are in the ChangeLog file in the package and on the Rcpp Changelog page.

Changes in Rcpp version 0.10.0 (2012-11-13)

• Support for C++11 style attributes (embedded in comments) to enable use of C++ within interactive sessions and to automatically generate module declarations for packages:
  • Rcpp::export attribute to export a C++ function to R
  • sourceCpp() function to source exported functions from a file
  • cppFunction() and evalCpp() functions for inline declarations and execution
  • compileAttribtes() function to generate Rcpp modules from exported functions within a package
  • Rcpp::depends attribute for specifying additional build dependencies for sourceCpp()
  • Rcpp::interfaces attribute to specify the external bindings compileAttributes() should generate (defaults to R-only but a C++ include file using R_GetCCallable can also be generated)
  • New vignette "Rcpp-attribute"
• Rcpp modules feature set has been expanded:
  • Functions and methods can now return objects from classes that are exposed through modules. This uses the make_new_object template internally. This feature requires that some class traits are declared to indicate Rcpp's wrap/as system that these classes are covered by modules. The macro RCPP_EXPOSED_CLASS and RCPP_EXPOSED_CLASS_NODECL can be used to declared these type traits.
  • Classes exposed through modules can also be used as parameters of exposed functions or methods.
  • Exposed classes can declare factories with ".factory". A factory is a c++ function that returns a pointer to the target class. It is assumed that these objects are allocated with new on the factory. On the R side, factories are called just like other constructors, with the "new" function. This feature allows an alternative way to construct objects.
  • "converter" can be used to declare a way to convert an object of a type to another type. This gets translated to the appropriate "as" method on the R side.
  • Inheritance. A class can now declare that it inherits from another class with the .derives<Parent>( "Parent" ) notation. As a result the exposed class gains methods and properties (fields) from its parent class.
• New sugar functions:
  • which_min implements which.min. Traversing the sugar expression and returning the index of the first time the minimum value is found.
  • which_max idem
  • unique uses unordered_set to find unique values. In particular, the version for CharacterVector is found to be more efficient than R's version
  • sort_unique calculates unique values and then sorts them.
• Improvements to output facilities:
  • Implemented sync() so that flushing output streams works
  • Added Rcerr output stream (forwarding to REprintf)
• Provide a namespace 'R' for the standalone Rmath library so that Rcpp users can access those functions too; also added unit tests
• Development releases sets variable RunAllRcppTests to yes to run all tests (unless it was alredy set to 'no'); CRAN releases do not and still require setting which helps with the desired CRAN default of less testing at the CRAN server farm.

Thanks to CRANberries, you can also look at a diff to the previous release 0.9.15. As always, even fuller details are on the Rcpp Changelog page and the Rcpp page which also leads to the downloads, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page

Update: One link corrected.

Weblate is close to 1.3 release (should happen on Friday if nothing urgent appears) and it's quite last chance for translators to catch up. Weblate is of course translated using Weblate, but you can of course also translate Gettext po files directly and either upload them into Weblate or use Github issue tracker for that. If you don't know Weblate yet, it is web based tool for translating with Git integration. You can also call it crowdsourcing platform if you wish :-).

Filed under: English Phpmyadmin Weblate 2 comments Flattr this!

I do like the way programmers think in terms of DRY, for instance, among other forms of optimization. A couple of days ago I wanted to test different implementations of the same algorithm in C++. The usual approach would be to implement those algorithms, call them in main() and see the results. A more interesting approach would be to write a validator for each function and print whether the result is correct or not. Since both are boring and break the rule of DRY, I ve decided to go complicated but productive, so I ve implemented a function testing method using function pointers that will allow me to write a single function and use it to test the algorithms using different input and expected results. Let s say I have 3 algorithms that implement Modular Exponentiation.
uint64 modexp1new(const uint64 &a, const uint64 &_p, const uint64 &n)
...


uint64 modexp2(const uint64 &_a, const uint64 &_p, const uint64 &_n)
...


uint64 modexp3(const uint64 &a, const uint64 &p, const uint64 &n)
...

If you are not familiar with the syntax, const means that the parameter is constant, thus it will not be modified inside the function. The & means that the parameter is passed as a reference, read the memory address of the calling variable, so we don t overload RAM with new variables. Now, the testing part. For my very own purposes I just want to test those particular functions and know whether if the result they output conforms what I do expect. Some people call this unit testing.
I do also want to test it in different scenarios, meaning input and output. So, I m creating a function that will get the three parameters required to call the functions, and a fourth parameter that is the value I do expect to be the correct answer. Now, since I don t want to repeat myself writing a call for each case, I m going to create a function pointer that is an array and has the function s address as their value. That way I can call them in a loop, and voila! we are done. Finally, after calling the function I check the result with the expected value and print an OK or FAIL. Tricky part for this could be understanding the function pointer. Two things to consider: the returning value from the referenced functions has to be the same for all. Second: the input definition for each function has to be similar too. This is important because function pointers are just pointers and they need to know the size of the data type in order to navigate the memory. For this sample code the uint64 is a typedef for long long, of course. Full code is below.
void testAlgos(const uint64 &a, const uint64 &e, const uint64 &n, const uint64 &r)
uint64 (*fP[3])(const uint64&, const uint64&, const uint64&) = &modexp1new, &modexp2, &modexp3 ;
uint64 t; for(int i=0; i<3; i++) t = fP[i](a, e, n); std::cout < < "F(" << i << ") " << t << "\t";
if (t == r)
std::cout << "OK";
else
std::cout << "FAIL";

std::cout << std::endl;

Now that I have this, I can use it in main this way.
int main() uint64 a = 3740332;
uint64 e = 44383;
uint64 n = 3130164467;
uint64 r = 1976425102; testAlgos(a, e, n, r); a = 404137263;
r = 2520752541; testAlgos(a, e, n, r); a = 21;
e = 3;
n = 1003;
r = 234;
testAlgos(a, e, n, r); return 0;
Resulting in:
F(0) 657519405 FAIL
F(1) 1976425102 OK
F(2) 657519405 FAIL
F(0) -2752082808 FAIL
F(1) 2520752541 OK
F(2) -2752082808 FAIL
F(0) 234 OK
F(1) 234 OK
F(2) 234 OK
I m happy about not having to write tests like this for all use cases:
std::cout < < "f1: " << modexp1new(a, e, n) << std::endl;
std::cout << "f2: " << modexp2(a, e, n) << std::endl;
std::cout << "f3: " << modexp3(a, e, n) << std::endl;
Now, happiness can be more exciting if I do use templates so I can test any function independent of the data type for the returning and input values. You know, more abstraction. Homework if time allows! Have fun!
Related posts:

1. Compute Clusters Integration for Debian Development and Building Report 4 Hello, this is the fourth report for the project. First,...
2. Compute Clusters Integration for Debian Development and Testing Report 3 Hello this is the third report regarding my project. As...
3. Compute clusters for Debian development and building final report Compute clusters for Debian development and building final report...