The vp8 video codec and webm container format (which are alternates for theora/ogg and h264/flv) are getting lots of attention, especially as more web browsers are supporting html5 - which includes a new video tag that plays video directly in a web page, rather than requiring a plugin. With Google fully behind these new formats (and converting YouTube videos) and even skype using vp8, it seems as though it could supplant h264/flash as the default video codec and container. After years under the boot of the proprietary flash video, this change could usher in a lot of exciting developments. While the shift in formats is also significant for on-demand video, I'm focusing on live streaming in this blog. There are a lot of tools needed to successfully stream a live video. Enough of them support vp8 and webm to create a live video stream, however, not all of the right versions of these tools have landed in Debian. Without any patches or changes, if you try to send a live video stream to icecast using vp8 encoding and webm, you'll get something like this:

0 jamie@animal:~$ gst-launch v4l2src ! video/x-raw,width=320,height=240 ! vp8enc ! webmmux ! shout2send ip=icecast.server port=8000 password=secret mount=/test.webm
WARNING: erroneous pipeline: could not link webmmux0 to shout2send0
1 jamie@animal:~$

I've documented the steps for patching/upgrading gstreamer and icecast2 so you can get a live video stream using vp8 encoding and the webmc container that originates on a Debian wheezy machine using gstreamer and streams to a Debian squeeze server running icecast2. On the workstation, you'll need libshout3 version 2.3.0 or higher, which is at the moment available in experimental and installable without bringing in any dependencies. In addition, you will need to bring in gstreamer1.0 from sid and you will need to rebuild gstreamer1.0-plugins-good after libshout-dev has been installed from experimental. So, begin by adding experimental and sid to your sources list by creating the files /etc/apt/sources.list.d/experimental.list and /etc/apt/sources/lists.d/sid.list with the respective contents:

deb http://mirror.cc.columbia.edu/debian experimental main

and

deb http://mirror.cc.columbia.edu/debian sid main 
deb-src http://mirror.cc.columbia.edu/debian sid main

In order to avoid upgrading your entire machine to sid (experimental packages won't upgrade by default), you'll also need to add the file /etc/apt/preferences.d/sid:

Package: *
Pin: release n=sid
Pin-Priority: 200

Then update and install:

apt-get update
apt-get install -t experimental libshout3
apt-get install gir1.2-clutter-gst-1.0 gir1.2-gst-plugins-base-1.0 gir1.2-gstreamer-1.0 gstreamer0.10-ffmpeg  gstreamer1.0-libav:amd64 gstreamer1.0-plugins-bad:amd64 gstreamer1.0-plugins-base:amd64 gstreamer1.0-plugins-base-apps gstreamer1.0-plugins-good:amd64 gstreamer1.0-pulseaudio:amd64 gstreamer1.0-tools gstreamer1.0-x:amd64 libclutter-gst-1.0-0:amd64 libgstreamer-plugins-bad1.0-0:amd64 libgstreamer-plugins-base1.0-0:amd64 libgstreamer-plugins-base1.0 libgstreamer1.0-0:amd64 libgstreamer1.0

Fortunately, the version of gstreamer1.0-plugins-good in sid includes a fix for a bug patched on November 31, 2012 that allows the use of the webm container. However, for it to work properly gstreamer1.0-plugins-good has to be built against libshout version 2.3.0 or higher (which is in experimental, not sid). Therefore, you have install libshout-dev from experimental and then rebuild gstreamer1.0-plugins-good.

sudo apt-get build-dep gstreamer1.0-plugins-good
sudo apt-get -t experimental install libshout-dev
apt-get source gstreamer1.0-plugins-good
cd gst-plugins-good1.0-1.0.4
dch -v 1.0.4-1+webmmux
fakeroot debian/rules binary
sudo dpkg -i ../gstreamer1.0-plugins-good_1.0.4-1+webmmux_amd64.deb

Now if you try the command, things should work on the client side. When streaming to an icecast2 server running squeeze, I don't get any errors, however, when opening the streaming page via the icecast URL, there is a play button, which can be clicked, but no picture is shown. That's because the version of Icecast2 in squeeze doesn't support either vp8 or webm. On the server side, you will need icecast2 version 2.4beta or higher. This version hasn't landed anywhere in Debian as of this writing. However, you can build it from source by following these steps (the patches I'm removing have all been applied to the new upstream version).

apt-get source icecast2
wget http://downloads.xiph.org/releases/icecast/icecast-2.4-beta.tar.gz
tar -xzvf icecast-2.4-beta.tar.gz
cp -r icecast2-2.3.2/debian icecast-2.3.99.0/
cd icecast-2.3.99.0/
rm debian/patches/ 1001_autotools_avoid_debian-subdir.patch,1002_add_missing_xspf_file,1003_fix_memory_leak.patch,series 
dch -v 2.3.99.0-1
fakeroot debian/rules binary

When you install it, be sure to replace the icecast2.xml configuration file and update it with your server-specific customizations. Hopefully this version of icecast2 will make it to experimental soon. And now... you can successfully stream via:

gst-launch-1.0 v4l2src ! video/x-raw,width=320,height=240 ! vp8enc ! webmmux !  shout2send ip=icecast.server port=8000 password=secret mount=/test.webm

If you want full audio and video, here's a more complete pipeline: gst-launch-1.0 v4l2src ! queue ! video/x-raw,width=320,height=240 ! tee name=tscreen ! queue ! autovideosink tscreen. ! queue ! videorate ! video/x-raw,framerate=25/2 ! queue ! vp8enc ! queue ! webmmux name=mux pulsesrc ! queue ! audioconvert ! vorbisenc ! queue ! queue ! mux. mux. ! queue ! shout2send ip=hobo port=8000 mount=jamie.webm password=secret Unfortunately, since I'm not sure how to deal with the python2.7 vs python3 dependecies, my elegant mobi python script that creates this pipeline for you using python-gtk as been reduced to a crude bash script. Another item on the todo list is opus audio encoding. gstreamer doesn't seem able to use opus in the webm container (although it seems to work with ogg). In the course of this research I found directions for getting gstreamer-1.0 installed on debian independent of packages installed on the system. If you follow the directions above, these steps should not be necessary. I'm including in case you need the latest sources for another reason. Extra steps I needed were:

• Edit gst-rebuild, comment out gst-plugins-ugly from plugins var (line 14) and sources arg (line 14) - these caused errors for me.
• manually install the following packages from experimental (apt-get build-dep gets confused about how to find them):
  • apt-get install -t experimental libavcodec-dev=6:9~beta3-1
  • apt-get install -t experimental libavformat-dev=6:9~beta3-1
  • apt-get install -t experimental libswscale-dev=6:9~beta3-1 libswscale2=6:9~beta3-1
• Then follow directions as advertised

Review: Code Name Verity, by Elizabeth Wein

Publisher:	Hyperion
Copyright:	2012
ISBN:	1-4231-5219-0
Format:	Hardcover
Pages:	339

I don't read very much historical fiction, and I mostly avoid historical fiction from the 20th century. I'm particularly uninterested in reading about Nazis after the saturation programming in places like the History Channel and the number of alternative histories that involve them. I've read enough about World War II to last me a lifetime, and I also avoid books about people being tortured. When Code Name Verity showed up in reviews I read, I was therefore rather dubious: a book set during World War II featuring someone being tortured by Nazis seemed unlikely to appeal to me. But after two steller reviews from reviewers who don't give out top marks easily, I bought a copy with some trepidation. That was one of the best decisions that I've ever made. This is the sort of book that is severely hurt by spoilers. In a moment, I'm going to attempt the impossible dance of giving you a feeling for the book and why I love it so much without spoiling anything. But if you trust my taste in books, let me give you a short warning, and then I suggest you stop reading this review and just go read the book. I avoided as much information about it as possible before reading it and never regretted it. The short warning: This is a book that hurts. lightreads describes it as searing. I think that's a very good term. It's not that it's gruesome; there is torture, but most of it happens off-camera and the focus is on emotional reactions rather than physical description. But the story sneaks up on you with sudden flashes of insight, and parts of it are a gut punch. It's the kind of book that had me in tears for a quarter of the book. It's the kind of book that nearly has me in tears remembering it to write this review. So, you may want to save this one until you are in the mood for that kind of book. But, that warning aside, Code Name Verity is unspeakably brilliant. This is one of the best books I have ever read in my life. I will try to explain more, but if your tastes mostly align with mine, just stop here and go read it.

I AM A COWARD.

That's how Code Name Verity starts: the words of a captured female British agent in Nazi-occupied France. The book opens near the end of her interrogation (and torture). The important information has already been discussed, and now she's likely destined to be executed. For the last set of wireless codes, she's bought herself paper and two weeks to write, and is expected to record everything that she remembers about the British war effort. The pages of this book are that writing (quite literally in the frontpiece and part of the cover, which are beautifully designed). The way she chooses to approach that task is to tell a story about the person whose ID she had when she was captured, a story that becomes the recounting of how she ended up where she is. That story is the heart and soul of this book. It's about the war, yes, and about British operations during the war. But it's a story about airplanes, too, and about the Air Transport Auxiliary, and about women finding a place in the war (with just a bit of a fictional push, detailed by the author in an afterword, although not quite as much of one as you might expect). And it's a story of a friendship, an amazing friendship of two women who would have never met each other if it weren't for the war and its leveling and transcending of normal social separations. It's a story about a friendship of opposites, about two people who find ways to support each other, about two people who see the best in each other. This is quite possibly the best novel about a friendship that I've ever read. This story, by itself, is wonderful. The tone starts light and irreverant, with an undertone of complex despair and defiance that's hard to analyze. One wonders where it's going and whether any of this is relevant, and then it slowly draws in the reader and becomes utterly compelling. But that's only the start of what this book has in store. Wein does some amazing structural things here that turn and broaden and transform, things that absolutely should not be spoiled because experiencing them without advance knowledge is a breathtaking reading experience. And all the parts play off against each other with a confidence that lets Wein understate and even hint at deeper emotions and be completely confident that the reader is right where she wants them to be. It's not a one-trick book, nor is it one that's going to lose you in the structure, but it's a book that opens up layers as you read it and tempts you re-read it immediately with new eyes once you finish it. Except it hurts so much that re-reading it immediately might be more than your heart can take. Perhaps the most astonishing part of this book is the way it hits its emotional notes. It is a masterpiece of showing rather than telling. Wein undersells the emotions in exactly the right way to make them stronger and more effective, avoiding, in the process, multiple traps that I've rarely seen avoided. The ending is absolutely note-perfect to everything that's happened before in a way that I thought, five pages from the end, was going to be impossible to pull off, and gets even better the more that one thinks about it. And the whole book is like that. It's been a month and a half since I read this book, in part because I wasn't sure how to write this review, and I did not, in fact, manage to finish writing it without crying. If I didn't have to be on a train in eight hours from when I'm writing this (a few days before I post it), I'd start reading it again right now. I can't talk about most of the things that make it amazing with anyone who hasn't read it, but it really is that good. Rating: 10 out of 10

I have been dealing with an interesting forking issue at work. It happens to involve Perl, but don't let that put you off. So, suppose you need to perform an I/O-bound task that is eminently parallelizable (in our case, generating and sending lots of emails). You have learnt from previous such attempts, and broken out Parallel::Iterator from CPAN to give you easy fork()ing goodness. Forking can be very memory-efficient, at least under the Linux kernel, because pages are shared between the parent and the children via a copy-on-write system. Further suppose that you want to generate and share a large data structure between the children, so that you can iterate over it. Copy-on-write pages, should be cheap, right?

my $large_array_ref = get_data();
my $iter = iterate( sub  
    my $i = $_[1];
    my $element = $large_array_ref->[$i];
    ...
 , [0..1000000] );

Sadly, when you run your program, it gobbles up memory until the OOM killer steps in. Our first problem was that the system malloc implementation was less good for this particular task than Perl's built-in malloc. Not a problem, we were using perlbrew anyway, so a quick few experimental rebuilds later and this was solved. More interesting was the slow, 60MB/s leak that we saw after that. There were no circular references, and everything was going out of scope at the end of the function, so what was happening? Recall that Perl uses reference counting to track memory allocation. In the children, because we took a reference to an element of the large shared data structure, we were effectively writing to the relevant page in memory, so it would get copied. Over time, as we iterated through the entire structure, the children would end up copying almost every page! This would double our memory costs. (We confirmed the diagnosis using 'smem', incidentally. Very useful.) The copy-on-write semantics of fork() do not play well with reference-counted interpreted languages such as Perl or CPython. Apparently a similar issue occurs with some mark-and-sweep garbage-collection implementations - but Ruby 2.0 is reputed to be COW-friendly. All was not lost, however - we just needed to avoid taking any references! Implement a deep copy that does not involve saving any intermediate variables along the way. This can be a bit long-winded, but it works.

my $large_array_ref = get_data();
my $iter = iterate( sub  
    my $i = $_[1];
    my %clone;
    $clone id   = $large_array_ref->[$i] id ;
    $clone foo  = $large_array_ref->[$i] foo ;
    ...
 , [0..1000000] );

This could be improved if we wrote an XS CPAN module that cloned data structures without incrementing any reference counts - I presume this is possible. We tried the most common deep-copy modules from CPAN, but have not yet found one that avoids reference counting. This same problem almost certainly shows up when using the Apache prefork MPM and mod_perl - even read-only global variables can become unshared. I would be very interested to learn of any other approaches people have found to solve this sort of problem - do email me.

Relase 0.10.2 of Rcpp provides the second update to the 0.10.* series, and has arrived on CRAN and in Debian. It brings another great set of enhancements and extensions, building on the recent 0.10.0 and 0.10.1 releases. The new Rcpp attributes were rewritten to not require Rcpp modules (as we encountered on issue with exceptions on Windows when built this way), code was reorganized to significantly accelerate compilation and a couple of new things such as more Rcpp sugar goodies, a new timer class, and a new string class were added. See below for full details. We also tested this fairly rigorously by checking about two thirds of the over 90 CRAN packages depending on Rcpp (and the remainder required even more package installs which we did not do as this was already taking about 12 total cpu hours to test). We are quite confident that no changes are required (besides one in our own RcppClassic package which we will update. The complete NEWS entry for 0.10.2 is below; more details are in the ChangeLog file in the package and on the Rcpp Changelog page.

Changes in Rcpp version 0.10.2 (2012-12-21)

• Changes in Rcpp API:
  • Source and header files were reorganized and consolidated so that compile time are now significantly lower
  • Added additional check in Rstreambuf deletetion
  • Added support for clang++ when using libc++, and for anc icpc in std=c++11 mode, thanks to a patch by Yan Zhou
  • New class Rcpp::String to facilitate working with a single element of a character vector
  • New utility class sugar::IndexHash inspired from Simon Urbanek's fastmatch package
  • Implementation of the equality operator between two Rcomplex
  • RNGScope now has an internal counter that enables it to be safely used multiple times in the same stack frame.
  • New class Rcpp::Timer for benchmarking
• Changes in Rcpp sugar:
  • More efficient version of match based on IndexHash
  • More efficient version of unique base on IndexHash
  • More efficient version of in base on IndexHash
  • More efficient version of duplicated base on IndexHash
  • More efficient version of self_match base on IndexHash
  • New function collapse that implements paste(., collapse= "" )
• Changes in Rcpp attributes:
  • Use code generation rather than modules to implement sourceCpp and compileAttributes (eliminates problem with exceptions not being able to cross shared library boundaries on Windows)
  • Exported functions now automatically establish an RNGScope
  • Functions exported by sourceCpp now directly reference the external function pointer rather than rely on dynlib lookup
  • On Windows, Rtools is automatically added to the PATH during sourceCpp compilations
  • Diagnostics are printed to the console if sourceCpp fails and C++ development tools are not installed
  • A warning is printed if when compileAttributes detects Rcpp::depends attributes in source files that are not matched by Depends/LinkingTo entries in the package DESCRIPTION

Thanks to CRANberries, you can also look at a diff to the previous release 0.10.1. As always, even fuller details are on the Rcpp Changelog page and the Rcpp page which also leads to the downloads, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page

I run mpd on debian on "igor", an NSLU2 -- a very low-power ~266MHz armel machine, with no FPU and a scanty 32MiB of RAM. This serves nicely to feed my stereo with music that is controllable from anywhere on my LAN. When playing music and talking to a single mpd client, the machine is about 50% idle. However, during a recent upgrade, something wanted to pull in pulseaudio, which in turn wanted to pull in libasound2-plugins, and i distractedly (foolishly) let it. With that package installed, after an mpd restart, the CPU was completely thrashed (100% utilization) and music only played in stutters of 1 second interrupted by a couple seconds of silence. igor was unusable for its intended purpose. Getting rid of pulseaudio was my first attempt to fix the stuttering, but the problem remained even after pulse was all gone and mpd was restarted. Then i did a little search of which packages had been freshly installed in the recent run:

grep ' install .* <none> ' /var/log/dpkg.log

and used that to pick out the offending package. After purging libasound2-plugins and restarting mpd, the igor is back in action. Lesson learned: on low-overhead machines, don't allow apt to install recommends!

echo 'APT::Install-Recommends "0";' >> /etc/apt/apt.conf

And it should go without saying, but sometimes i get sloppy: i need to pay closer attention during an "apt-get dist-upgrade" Tags: alsa, apt, low-power, mpd

I wanted to be able to use the GMail web interface on my work machine, but for privacy reasons, I prefer not to be logged into my Google Account on my main browser. Here's how I make use of a somewhat hidden Firefox feature to move GMail to a separate browser profile. Creating a separate profile The idea behing browser profiles is simple: each profile has separate history, settings, bookmarks, cookies, etc. To create a new one, simply start Firefox with this option:

firefox -ProfileManager

to display a dialog which allows you to create new profiles: Once you've created a new "GMail" profile, you can start it up from the profile manager or directly from the command-line:

firefox -no-remote -P GMail

(The -no-remote option ensures that a new browser process is created for it.) To make this easier, I put the command above in a tiny gmail shell script that lives in my ~/bin/ directory. I can use it to start my "GMail browser" by simply typing gmail. Tuning privacy settings for 2-step authentication While I initially kept that browser profile in private browsing mode, this was forcing me to enter my 2-factor authentication credentials every time I started the browser. So to avoid having to use Google Authenticator (or its Firefox OS cousin) every day, I ended up switching to custom privacy settings and enabling all cookies: It turns out however that there is a Firefox extension which can selectively delete unwanted cookies while keeping useful ones. Once that add-on is installed and the browser restarted, simply add accounts.google.com to the whitelist and set it to clear cookies when the browser is closed: Then log into GMail and tick the "Trust this computer" checkbox at the 2-factor prompt: With these settings, your browsing history will be cleared and you will be logged out of GMail every time you close your browser but will still be able to skip the 2-factor step on that device.

WebAuth is a site-wide web authentication system with single sign-on support. We use it extensively at Stanford for most of our user-facing web applications that need to authenticate people. This is the culmination of the last four months of work, so it's rather nice to finally get it out the door. Even if not everything I wanted to get done is done. That's always how it is with software releases. This release fixes a bug in the encoding of delegated Kerberos credentials when Heimdal is used as the Kerberos library. If you're building WebAuth against Heimdal, upgrade any mod_webauth modules that receive delegated credentials before upgrading mod_webkdc on the WebKDC. Otherwise, the flags of delegated tickets won't be sent correctly. The major new feature in this version is support for authorization identities separate from authentication identities. This can be used for testing, for administrators to proxy into an application as another user, or in other cases where there's a mismatch between the canonical concept of users and the capabilities of an application. The WebLogin server and WebKDC now support an ACL file that controls what alternative identities users can assert to specific web sites, and there is a new flow on the WebLogin side to set or change identities. All of this is disbled by default in each component. There is a new WebKDC directive to enable this support and specify the identity ACL file, and there is a new directive in mod_webauth to tell it to trust the authorization identity. There is a new environment variable that will be set to the authorization identity (trusted or not), and REMOTE_USER will be set to the authorization identity only if it is trusted. WEBAUTH_USER will always remain the authentication identity, so both identities can be logged and web applications can distinguish. To address one of the edge cases required by this support, there's a new mod_webkdc Apache directive, WebKdcLoginTimeLimit, which controls how long a multi-step login process can take, and also how recent a login has to be to contribute its authentication factors to the session factors for an authentication. This is also now used for WebAuthForceLogin, which means that forced logins won't require re-authentication if they're within the WebKdcLoginTimeLimit interval of the last login. This allows that feature to work properly in conjunction with authorization identities and with some multifactor authentication methods. Also in this release, optional replay detection and rate limiting of failed logins has been added to the WebLogin server. Either or both can be enabled in the configuration file. This support requires a memcached server (shared across any pool of WebLogin servers) be available to store the necessary state. There are new parameters in the error template to handle the error messages generated by these features. The WebLogin server also has a few more minor improvements: single sign-on cookies are now set even when displaying error pages if any are available, fixing some looping issues with some scenarios around restricting users to authenticate to specific sites, and the @REMUSER_REALMS setting has been broken into two settings so that its two properties can be changed independently. The old setting is still supported for backward compatibility. Multiple bugs in Kerberos ticket encoding have been fixed, some that were introduced in 4.3.0 and some that have been present since the first days of Heimdal support. Other fixes include the mapping of WebKDC error codes to names in WebLogin (which previously resulted in Perl warnings in the error log) and the missing documentation for the WebAuthRequireSSL directive. Finally, the thing that I'm the happiest about in this release is that I finished my complete refactoring of the libwebauth library. Most of the low-level interfaces are gone in favor of higher-level manipulation of WebAuth protocol objects. The last pieces of code that used the old token encoding system have been replaced with the new data-driven encoder. And the last parts of the library have been converted to APR, so there is no longer a mix of traditional memory management and APR pools. This comes with lots of simplification of the API and removal of old cruft from the public headers. You can get the latest release from the official WebAuth distribution site or from my WebAuth distribution pages.

Lately I've been working on making the redirector nicer to the mirrors and to some potential users. More specifically, those behind a caching proxy.

The redirector is now nicer to traditional web proxies by redirecting to objects that are known not to change with a "moved permanently" code (HTTP status code 301.) This applies to files in the pool/ directory and ".pdiff" files, among others.
Previously, a traditional caching web proxy would sometimes end up with multiple copies of the same object, fetched from different mirrors; and the redirection would not be cached at all. With this change, this is no longer the case.

Using a caching proxy that is aware of the Debian repository design is still more likely to yield better results, however: If my memory serves correctly, apt-cacher has the ability of updating the Packages, Sources, and similar files with the ".pdiff"s on the server side. Apt-Cacher-NG apparently can use debdelta, and so on.
Check my blog post about one APT caching proxy not being efficient for some comments related to those tools.

Another recent change is that mirrors that can't be used by the redirector will no longer be monitored as often as the other mirrors. For instance, if a mirror doesn't generate a trace file (used for monitoring) then the redirector will gradually limit the rate at which the mirror is checked.
This rate-limiting mechanism applies to different kinds of errors, and should reduce the amount of wasted time and bandwidth while still allowing automatic-detection of mirrors that recover.


The rate limiter applies an initial exception to allow temporary errors to not affect the use of the mirror by the redirector. After that exception, it is pretty much linear. However, that chart doesn't really reflect the effect of the rate limiter, so put in comparison with the normal checking behaviour:


The code to detect mirrors that don't perform a two-stages sync that I talked about in a previous post has not yet been integrated as the current implementation would be too expensive on the mirrors to just add it as-is.

While tracking down problems exposed to users, I decided to take a stricter approach as to what mirrors are used by the redirector. Suffice to say that the remaining mirrors using the obsolete anonftpsync are going to be ignored entirely. ftpsync has been around for a few years now and it is the standard tool.
Whether you are mirroring Debian, Raspbian, Ubuntu, or any other Debian-like packages repository, ftpsync is the right tool to use.

Most of the issues I've been discovering, and sometimes working around, affect direct users of the mirrors and are not related to the http.debian.net redirector. When not detected beforehand they happen to be exposed by the redirector, but like I said, I plan to be stricter in order to increase the redirector's reliability. Once a strict and reliable foundation is built, more workarounds might see their way in to better use the available resources.

That's it for now. The road is long, the challenge is great, and being an observer in an uncontrolled environment makes it even more interesting.

Review: Liars and Outliers, by Bruce Schneier

Publisher:	John Wiley
Copyright:	2012
ISBN:	1-118-14330-2
Format:	Hardcover
Pages:	285

One of the perils of buying a book written by a blogger one reads regularly is that the book may be little more than a rehashing of their blog, with insufficient original material to warrant the time investment. Sometimes it's still nice to support them financially, but it may not make sense to read the book. I've been following Schneier's blog for years (as should anyone with an interest in security), including through the entire process of writing Liars and Outliers, and was a bit worried that might be the case here. Thankfully, I can reassure any other worried potential readers that is not the case. This is substantial new material establishing a firm framework for thinking about incentives and controls in any society or organization. Liars and Outliers talks about security mechanisms, but it's not, at its core, a book about security. Rather, it's a book about incentives, order, and how order is established. It's a comprehensive reductionist analysis of how societies create predictability and compliance to allow us to trust other people who we have never met before and will never meet again. It's a unique (at least in my experience) combination of anthropology, sociology, security analysis, and political science. Schneier cuts across fields in an idiosyncratic but illuminating way that reminded me of (on an entirely separate topic) Jane Jacobs. This is not a prescriptive book, nor is a collection of answers, solutions, or even deep analysis of particular problems. Rather, it's an attempt to construct a general framework for analyzing societal dilemmas: conflicts between individual desires and social good, how those conflicts are resolved, and how societies can weigh the scales and influence the statistical outcomes. The closest Schneier comes to telling the reader how to solve problems is a checklist, at the end of the book, for designing effective societal pressures. Its primary contribution is vocabulary and structure. It also passes one of my litmus tests for any book about human behavior: Schneier complicates, broadens, deepens, and expands understanding, and points out complex interactions and complex feedback in effects we're inclined to consider simple, rather than simplifying or eliminating human complexity. One point I found refreshing about this book is that Schneier is scrupulous in refusing to define either society or individuals as good or bad, to the point of carefully defining terminology used in all of the social dilemmas. Following societal rules is called compliance; not following those rules is called defection. In some cases, defection is morally correct (Schneier's most frequent example is in the civil rights struggles of the 1960s in the United States). In all cases, social pressures are tools, which can be used to encourage compliance with moral or immoral systems, and which are deployed by totalitarian dictatorships and utopian communes alike. Schneier explicitly puts out of scope for this book the questions of how societal goals should be determined, how they change, and whether any given societal rule or interest is moral or immoral. He focuses, rather, only on the mechanisms, with a primary goal of informing and deepening debates over how best to encourage behavior that societies want to encourage and discourage behavior societies want to discourage. He also emphasizes discourage, as opposed to eliminate. Early on, Schneier shows some of the results of game theory, as well as basic common sense, that indicate that no healthy society can totally eliminate defection. Not only would that stifle valuable and important reform, such as changes in civil rights, but the degree of pressure required is immense. Defectors are natural and will always exist, and are sometimes valuable and necessary. Rather, the goal of a society is to reduce defectors to a level where most people can ignore their existence most of the time, a state that leads to the level of risk and trust required to have a functioning and healthy society. The word society, similarly, is intentionally broad, and can refer to just about any collection of people, from a circle of friends to a corporation, institution, or country. However, as Schneier points out early in the book, small societies rarely need much in the way of formal pressure and appear almost magically self-governing. That's the first property that he disassembles, resulting in a general classification of societal pressures into four categories: moral, reputational, institutional, and security systems. The last is an odd category that's partly orthagonal to the other three. Moral pressure is internalized conscience: the normal tendency of nearly everyone to follow their own moral code, a code that's at least partly constructed and certainly heavily influenced by the surrounding society. Reputational pressure is, in a sense, externalized morality: it's the informal reactions of others around one to one's past actions. Included in reputational pressure is shunning of every kind, from cutting off a friendship to boycotts against corporations, but it also includes confrontation from another member of one's society, the more subtle effects of our individual desires to be liked and respected, and all the various aspects of "face", honor, and respect within a community. Small communities are frequently self-governing, in Schneier's model, because they're small enough that moral and reputational pressures are sufficient and no other pressures are required. We're so used to applying moral and reputational pressure to other humans almost unconsciously that we sometimes don't even notice its existence, leading to that "magical" self-governing property. But Schneier puts pressures in a sequence: pressures that work extremely well with small groups often don't scale. Moral pressure works best with small groups and reputational pressure with somewhat larger groups, but when societies scale beyond the limits of reputational pressure when, for example, one frequently interacts with people whose reputations are unknown to you and whose subsequent opinion of you will not be relevant institutional pressure is required to force compliance. Institutional pressure is the sort of pressure that we all tend to think of first when we look for ways to enforce rules: laws, policies, contracts, and other codes of behavior that carry with them formal punishments and some enforcement mechanism. But even in societies so large that institutional pressures are frequently required, such as whole countries, moral and reputational pressures still exist and are extremely important. One of Schneier's most interesting points is his analysis of how institutional pressures can paradoxically undermine reputational and moral pressures, resulting in more defection than if the institutional pressure hadn't existed. This is just the basic framework of Schneier's analysis, hopefully giving you a feel for the structure of the book. He goes much deeper into the complicated interactions between the various levels of pressure, and then dives into an extensive look at competing societal dilemmas: cases where there is more than one society in play simultaneously, possibly demanding contradictory actions. Liars and Outliers also includes a wonderful analysis of organizational entities within the same framework, including their much-different reactions to moral, reputational, and institutional pressures. One of the most cogent analyses of the difficulties of regulating both corporations and governmental institutions falls out of that analysis, once one looks at them in light of Schneier's basic framework. Pressures quickly become complex and multi-layered, and human reactions to pressures are frequently counter-intuitive. Schneier draws extensively on game theory to show that some counter-intuitive responses are actually emergent properties of logical analysis of the situation, but that others are more uniquely human and have little or nothing to do with a mathematical cost-benefit analysis. I haven't even mentioned his discussion of security systems, and how they can extend moral, reputational, and institutional pressures, as well as add a new type of pressure (making a defecting action impossible) that scales even better than institutional pressures. Liars and Outliers has all of the supporting infrastructure you would expect in a scholarly book: notes, extensive references, and a good index. I suspect it will end up being used as at least additional reading in college classes. The notes are, unfortunately, end notes, making the full context of the book much harder to read than was necessary, but at least Schneier does separate the notes from the references so that one doesn't chase notes for further explanation and find a simple citation. As with any book like this, one always wishes it could end in a simple prescription to fix everything, but of course it doesn't. But that's also a measure of a good scholarly work. Human and organizational motivations are complex and tricky, and any framework for analyzing them needs to be able to represent that complexity. Schneier here has constructed a very powerful one, one that I started using in discussions before I'd even finished the book. Perhaps the most valuable contribution of Liars and Outliers to public discussion is clear terminology and categories, which can be of great help in finding the core components of a problem. Liars and Outliers can be slow going, particularly early on when Schneier is still defining terms and setting up the background of his analysis. One can get a bit tired of the analysis matrices of societal dilemmas. But stick with it through the groundwork, since the analyses of competing societal dilemmas and of the impact of societal pressures on organizations are exceptional. Highly recommended, particularly for anyone who is designing or implementing societal pressures: managers, political activists, or anyone in a security-related field. Rating: 8 out of 10

The text below just went out to r-sig-finance along with updates to the R/Finance website and its Call for Papers page.

Call for Papers: R/Finance 2013: Applied Finance with R
May 17 and 18, 2013
University of Illinois, Chicago, IL, USA
The fifth annual R/Finance conference for applied finance using R will be held on May 17 and 18, 2013 in Chicago, IL, USA at the University of Illinois at Chicago. The conference is expected to cover topics including portfolio management, time series analysis, advanced risk tools, high-performance computing, market microstructure, and econometrics. All will be discussed within the context of using R as a primary tool for financial risk management, portfolio construction, and trading. Over the past four years, R/Finance has included attendees from around the world. It featured presentations from prominent academics and practitioners, and we anticipate another exciting line-up for 2013. We invite you to submit complete papers in pdf format for consideration. We will also consider one-page abstracts (in txt or pdf format) although more complete papers are preferred. We welcome submissions for full talks and abbreviated "lightning talks". Both academic and practitioner proposals related to R are encouraged. Presenters are strongly encouraged to provide working R code to accompany the presentation/paper. Data sets should also be made public for the purposes of reproducibility (though we realize this may be limited due to contracts with data vendors). Preference may be given to presenters who have released R packages. The conference will award two (or more) $1000 prizes for best papers. A submission must be a full paper to be eligible for a best paper award. Extended abstracts, even if a full paper is provided by conference time, are not eligible for a best paper award. Financial assistance for travel and accommodation may be available to presenters at the discretion of the conference committee. Requests for assistance should be made at the time of submission. Please send submissions to: committee at RinFinance.com. The submission deadline is February 15, 2013. Submitters will be notified of acceptance via email by February 28, 2013. Notification of whether a presentation will be a long presentation or a lightning talk will also be made at that time. Additional details will be announced at this website as they become available. Information on previous year's presenters and their presentations are also at the conference website. For the program committee:

Gib Bassett, Peter Carl, Dirk Eddelbuettel, Brian Peterson,
Dale Rosenthal, Jeffrey Ryan, Joshua Ulrich

So see you in Chicago in May!

I've just pushed out the release files for Obnam version 1.3, my backup application, as well as Larch, my B-tree library, and cliapp, my Python framework for command line applications. They are available via my home page (http://liw.fi/). Since Debian is frozen, I am not uploading packages to Debian, but .deb files are available from my personal apt repository for the intrepid. (I will be uploading to Debian again after the freeze. I am afraid I'm too lazy to upload to experimental, or do backports. Help is welcome!) From the Obnam NEWS file:

• When creating files in the backup repository, Obnam tries to avoid NFS synchronisation problems by first writing a temporary file and then creating a hardlink to the actual filename. This works badly on filesystems that do not allow hard links, such as VFAT. If creating the hardlink fails, Obnam now further tries to use the open(2) system call with the O_EXCL flag to create the target file. This should allow things to work with both NFS and VFAT.
• More detailed progress reporting during the backup.
• Manual page now covers the diff subcommand. Patch by Peter Valdemar M rch.
• Speed optimisation patch for backing up files in inode numbering order, from Christophe Vu-Brugier.
• A setuid or setgid bit is now not restored if Obnam is not used by root or the same user as the owner of the restored file.
• Many new settings to control "obnam fsck", mainly to reduce the amount of checking being done in order to make it faster. However, fsck is has lost some features (checks), which will be added back in a future release.
• More frequent fsck progress reporting. Some speed optimisations to fsck.

Bug fixes for Obnam:

• Empty values for extended attributes are now backed up correctly. Previously they would cause an infinite loop.
• Extended attributes without values are now ignored. This is different from attributes with empty values. Reported by Vladimir Elisseev.
• An empty port number in sftp URLs is now handled correctly. Found based on report by Anton Shevtsov.
• A bad performance bug when backing up full systems (starting from the filesystem root directory) has been fixed. At the beginning of each generation, Obnam removes any directories that are not part of the current backup roots. This is necessary so that if you change the backup roots, the old stuff doesn't hang around forever. However, when the backup root is the filesystem root, due to the now-fixed bug Obnam would first remove everything, and then back it up all over again. This "worked", but was quite slow. Thanks to Nix for reporting the problem.
• Obnam now runs GnuPG explicitly with the "no text mode" setting, to override a "text mode" setting in the user's configuration. The files Obnam encrypts need to be treated as binary, not text files. Reported by Robin Sheat.
• A shared B-tree concurrency bug has been fixed: If another instance of Obnam was modifying a shared B-tree, Obnam would crash and abort a backup, possibly leaving lock files lying around. Now a failure to look up a chunk via its checksum is ignored, and the backup continues.
• Bugs in how Python OSError exceptions were being raises have been fixed. Error messages should now be somewhat clearer.
• Unset or wrongly set variable "full" fixed in "obnam diff". Reported by ROGERIO DE CARVALHO BASTOS and patched by Peter Valdemar M rch.
• Setuid and setgid bits are now restored correctly, when restore happens as root. Reported by Pavel Kokolemin.
• Obnam now complains if no backup roots have been specfied.

NEWS for Larch:

• Make fsck progress reporting be a bit more fine grained.

NEWS for cliapp:

• Options in option groups are now included in manual page SYNOPSIS and OPTIONS sections.
• --log=syslog message format improvement by Daniel Silverstone. No longer includes a timestamp, since syslog adds it anyway. Also, the process name is now set on Linux.
• Make the default subcommand argument synopsis be an empty string, instead of None. Reported by Sam Thursfield.
• Meliae memory dumping support has been fixed. Reported by Joey Hess.
• Memory profiling reports can now be done at minimum intervals in seconds, rather than every time the code asks for them. This can reduce the overhead of memory profiling quite a lot.
• If there are any subcommands, cliapp now adds a subcommand called help, unless one already exists.
• For every boolean setting foo, there will no be a --no-foo option to be used on the command line.

Since I've been working on qcontrol I've had need of a way to build packages in an ARM environment. Since all my native ARM hardware is "production" (at least so far as something can be production in a home environment) I prefer to avoid installing development tools on them. When I started looking into this all my ARM systems ran from relatively small flash devices which also ruled out the possibility of a development chroot. After search the web a bit I discovered that it is possible to use qemu-user-static together with [[!binfmt-support ]] to create cross architecture chroots which can be entered mostly transparently. It is also possible to combine this with schroot and sbuild which makes the day to day use pretty much the same as a native or biarch chroot. Since I've now had to rediscover how to do this twice (once for Sid and then again when I realized I also needed a Wheezy environment) I thought I should write it down for the benefit of my future self. First things first lets install the various packages which we need. As well as debootstrap to create the chroot we need qemu-user-static and binfmt-support to allow us to run ARM binaries:

$ sudo apt-get install debootstrap qemu-user-static binfmt-support

I'm using schroot's lvm-snapshot mode so we need an LVM volume, formatted with a filesystem and mounted in a temporary location:

$ sudo lvcreate -L8G -n sbuild-wheezy-armel /dev/VG
$ sudo mkfs.ext3 /dev/VG/sbuild-wheezy-armel
$ sudo mount /dev/VG/sbuild-wheezy-armel /mnt/

Next we need to create the chroot. For this we use debootstrap in foreign mode which does a 2 stage setup, firstly by using the --foreign option:

$ sudo debootstrap --variant=buildd --include=fakeroot,build-essential \
        --arch=armel --foreign \
        wheezy /mnt http://<DEBIAN-MIRROR>/debian/

Now we (temporarily) copy the qemu-arm-static binary into the chroot (at the place where binfmt-support expects it), change into the chroot and run debootstrap's second stage to finish the basic chroot setup:

$ sudo cp /usr/bin/qemu-arm-static /mnt/usr/bin/
$ sudo chroot /mnt
(chroot)# ./debootstrap/debootstrap --second-stage
(chroot)# exit

Now that we have a basic chroot in place it needs a little bit of tailoring. I've found the easiest way to do this is to use sbuild-createchroot and then tweak the result:

$ sudo sbuild-createchroot --arch=armel --foreign --setup-only \
    wheezy /mnt http://<DEBIAN-MIRROR>/debian/

If we were building a native chroot then we could have skipped the debootstrap stuff and let sbuild-createchroot handle things. Unfortunately this doesn't seem to play nicely with --foreign. Now we can remove the qemu-arm-static binary from the chroot. schroot will automatically bind mount the version from the host into the chroot which ensures it remains up to date. We can also unmount the chroot since we will let schroot manage that from here on:

$ sudo rm /mnt/usr/bin/qemu-arm-static
$ sudo umount /mnt

Next we need to tweak the resulting schroot/sbuild configuration since sbuild-createchroot assumes the use of schroot's directory mode. First remove the symlink to /mnt which it created:

$ sudo rm /etc/sbuild/chroot/wheezy-armel-sbuild

Lastly we need to edit /etc/schroot/chroot.d/wheezy-armel-sbuild-<SUFFIX> (where is randomly generated) and reconfigure it to use lvm-snapshot mode, by apply the following changes:

 [wheezy-armel-sbuild]
-type=directory
-directory=/mnt
+type=lvm-snapshot
+device=/dev/VG/sbuild-wheezy-armel
+lvm-snapshot-options=--size 2G
 description=Debian wheezy/armel autobuilder
 groups=root,sbuild
 root-groups=root,sbuild
 profile=sbuild

That's it, we now have a chroot which we can trivially invoke via sbuild to build armel packages:

$ sbuild --arch=armel --dist=wheezy <... other options...>

We can also use sbuild-update to keep the chroot up to date:

$ sudo sbuild-update -udcar wheezy-armel

You can also use schroot to login to the chroot directly, although for that case you'll likely want to create a different chroot using schroot's directory or block-device mode instead lvm-snapshot and drop some of the sbuild specific setup. I expect I'll blog about this configuration in some more detail at some point in the future since I'll want to document it to give a more convenient build environment for Xen on ARM than the build farm which we have at work.

I ve just installed BTRFS on some systems that matter to me. It is still regarded as experimental but Oracle supports it with their kernel so it can t be too bad and it s almost guaranteed that anything other than BTRFS or ZFS will lose data if you run as many systems as I do and I run lots of systems that don t have enough RAM for ZFS (4G isn t enough in my tests). So I have to use BTRFS. BTRFS and Virtual Machines I m running BTRFS for the DomUs on a virtual server which has 4G of RAM (and thus can t run ZFS). The way I have done this is to use ext4 on Linux software RAID-1 for the root filesystem on the Dom0 and use BTRFS for the rest. For BTRFS and virtual machines there seem to be two good options given that I want BTRFS to use it s own RAID-1 so that it can correct errors from a corrupted disk. One is to use a single BTRFS filesystem with RAID-1 for all the storage and then have each VM use a file on that big BTRFS filesystem for all it s storage. The other option is to have each virtual machine run BTRFS RAID-1. I ve created two LVM Volume Groups (VGs) named diska and diskb, each DomU has a Logical Volume (LV) from each VG and runs BTRFS. So if a disk becomes corrupt the DomU will have to figure out what the problem is and fix it. #!/bin/bash
for n in $(xm list cut -f1 -d\ egrep -v ^Name\ ^Domain-0) ; do
echo $n
ssh $n "btrfs scrub start -B -d /"
done I use the above script in a cron job from the Dom0 to scrub the BTRFS filesystems in the DomUs. I use the -B option so that I will receive email about any errors and so that there won t be multiple DomUs scrubbing at the same time (which would be really bad for performance). BTRFS and Workstations The first workstation installs of BTRFS that I did were similar to installations of Ext3/4 in that I had multiple filesystems on LVM block devices. This caused all the usual problems of filesystem sizes and also significantly hurt performance (sync seems to perform very badly on a BTRFS filesystem and it gets really bad with lots of BTRFS filesystems). BTRFS allows using subvolumes for snapshots and it s designed to handle large filesystems so there s no reason to have more than one filesystem IMHO. It seems to me that the only benefit in using multiple BTRFS filesystems on a system is if you want to use different RAID options. I presume that eventually the BTRFS developers will support different RAID options on a per-subvolume basis (they seem to want to copy all ZFS features). I would like to be able to configure /home to use 3 copies of all data and metadata on a workstation that only has a single disk. Currently I have some workstations using BTRFS with a BTRFS RAID-1 configuration for /home and a regular non-RAID configuration for everything else. But now it seems that this is a bad idea, I would be better off just using a single copy of all data on workstations (as I did for everything on workstations for the previous 15 years of running Linux desktop systems) and make backups that are frequent enough to not have a great risk. BTRFS and Servers One server that I run is primarily used as an NFS server and as a workstation. I have a pair of 3TB SATA disks in a BTRFS RAID-1 configuration mounted as /big and with subvolumes under /big for the various NFS exports. The system also has a 120G Intel SSD for /boot (Ext4) and the root filesystem which is BTRFS and also includes /home. The SSD gives really good read performance which is largely independent of what is done with the disks so booting and workstation use are very fast even when cron jobs are hitting the file server hard. The system has used a RAID-1 array of 1TB SATA disks for all it s storage ever since 1TB disks were big. So moving to a single storage device for /home is a decrease in theoretical reliability (in addition to the fact that a SSD might be less reliable than a traditional disk). The next thing that I am going to do is to install cron jobs that backup the root filesystem to something under /big. The server in question isn t used for anything that requires high uptime, so if the SSD dies entirely and I need to replace it with another boot device then it will be really annoying but it won t be a great problem. Snapshot Backups One of the most important uses of backups is to recover from basic user mistakes such as deleting the wrong file. To deal with this I wrote some scripts to create backups from a cron job. I put the snapshots of a subvolume under a subvolume named backup . A common use is to have everything on the root filesystem, /home as a subvolume, /home/backup as another subvolume, and then subvolumes for backups such as /home/backup/2012-12-17, /home/backup/2012-12-17:00:15, and /home/backup/2012-12-17:00:30. I make /home/backup world readable so every user can access their own backups without involving me, of course this means that if they make a mistake related to security then I would have to help them correct it but I don t expect my users to deal with security issues, if they accidentally grant inappropriate access to their files then I will be the one to notice and correct it. Here is a script I name btrfs-make-snapshot which has an optional first parameter -d to cause it do just display the btrfs commands it would run and not actually do anything. The second parameter is either minutes or days depending on whether you want to create a snapshot on a short interval (I use 15 minutes) or a daily snapshot. All other parameters are paths for subvolumes that are to be backed up: #!/bin/bash
set -e

# usage:
# btrfs-make-snapshot [-d] minutes days paths
# example:
# btrfs-make-snapshot minutes /home /mail

if [ "$1" == "-d" ]; then
BTRFS="echo btrfs"
shift
else
BTRFS=/sbin/btrfs
fi

if [ "$1" == "minutes" ]; then
DATE=$(date +%Y-%m-%d:%H:%M)
else
DATE=$(date +%Y-%m-%d)
fi
shift

for n in $* ; do
$BTRFS subvol snapshot -r $n $n/backup/$DATE
done Here is a script I name btrfs-remove-snapshots which removes old snapshots to free space. It has an optional first parameter -d to cause it do just display the btrfs commands it would run and not actually do anything. The next parameters are the number of minute based and day based snapshots to keep (I am currently experimenting with 100 100 for /home to keep 15 minute snapshots for 25 hours and daily snapshots for 100 days). After that is a list of filesystems to remove snapshots from. The removal will be from under the backup subvolume of the path in question. #!/bin/bash
set -e

# usage:
# btrfs-remove-snapshots [-d] MINSNAPS DAYSNAPS paths
# example:
# btrfs-remove-snapshots 100 100 /home /mail

if [ "$1" == "-d" ]; then
BTRFS="echo btrfs"
shift
else
BTRFS=/sbin/btrfs
fi

MINSNAPS=$1
shift
DAYSNAPS=$1
shift

for DIR in $* ; do
BASE=$(echo $DIR cut -c 2-200)
for n in $(btrfs subvol list $DIR grep $BASE/backup/.*: head -n -$MINSNAPS sed -e "s/^.* //"); do
$BTRFS subvol delete /$n
done
for n in $(btrfs subvol list $DIR grep $BASE/backup/ grep -v : head -n -$MINSNAPS sed -e "s/^.* //"); do
$BTRFS subvol delete /$n
done
done A Warning The Debian/Wheezy kernel (based on the upstream kernel 3.2.32) doesn t seem to cope well when you run out of space by making snapshots. I have a filesystem that I am still trying to recover after doing that. I ve just been buying larger storage devices for systems while migrating them to BTRFS, so I should be able to avoid running out of disk space again until I can upgrade to a kernel that fixes such bugs.

Why? I switched to Jekyll but it wasn't a fast decision. I really tried to give WordPress a chance, again, but I'm not getting along pretty well with PHP since like 2 years ago. In fact, is a headache to maintain on my server. In the course of the years I have had several downtimes on my PHP-based sites because of changes like PHP-FPM. Some sites just dropped working for me while others kept doing OK. This is in the same server, with the same database and the same codebase (WordPress). After a lot of time programming on Node.js and been learning more Ruby (CoffeeScript made me love the syntax) I just got tired of using more resources of what I really needed, and with a regular CMS that means a database. Why would I need a database if I just update like once a day? For a comment-system? There's Disqus already, and people can just comment you on Twitter or make a Branch for it, or just write you an email! Plain simple. Jekyll So there was Jekyll, a pretty nice tool creating static-sites very fast and easy to learn. It didn't needed a database and because all is static is pretty fast. Even the hosting requirements got lowered to the point I could (and I did) just upload this to GitHub and served it from there as a GitHub Page! One more thing One thing made me decided to upload to GitHub as well was Prose.io, a simple text-editor for GitHub with support for MarkDown made by the guys @ developmentseed.com. So while it was made for being updated on my terminal I could just do it as well from the web, from anywhere with just my GitHub account and having previews on-the-fly. The downside Since I don't control this server I couldn't setup redirections for my old feeds. Of course, I could have just copied the same pattern in a folder and put there an XML feeding the category X, but I think is easier to update the planets being feeded to the new URIs. On the bright-side I even could put pretty-permalinks to work! So the structure for my old links kept the same.

Leanne Rolston
Dear Leanne, Save your praat scripts before you run them. That way, if they take too long, you can run them from the command line and get on with work. Love, Leanne
Like 8 minutes ago near Seattle, WA C.J. Adams-Collier Yeah, by the way, the command line is usually the way to go if you want to get anything done. I get lost with all those buttons. I ve got 104 of them I keep pretty good track of. Those ones that keep moving around? Yeah, that s confusing.
6 minutes ago Like
Leanne Rolston I ve noticed. I need to add praat to my path so I can stop having to hunt it down every time I want to do something!
5 minutes ago Like
C.J. Adams-Collier also, teach me praat?
5 minutes ago Like
C.J. Adams-Collier cjac@phx1:/usr/share/wordpress/wp-content/plugins$ apt-cache search praat
praat program for speech analysis and synthesis
4 minutes ago Like
C.J. Adams-Collier oh, nevermind. my homies already taught me.
4 minutes ago Like
Leanne Rolston Sure. It s simple In the praat scripting window, there s a paste history command.. anything you want to do, do it using the buttons, paste history, and kapow!
3 minutes ago Like
C.J. Adams-Collier $ less /usr/share/doc/praat/README.Debian
3 minutes ago Like
C.J. Adams-Collier can I save the script that I put together with buttons to something I can open with emacs?
2 minutes ago Like
Leanne Rolston Yup. Just save it with a .praat extension.
about a minute ago Like
Leanne Rolston http://depts.washington.edu/phonlab/resources.htm <=- some good sample scripts, and praat has a scripting tutorial.
Phonetics Lab Department of Linguistics University of Washington
depts.washington.edu
The Phonetics Lab at UW is located in Padelford Hall, Room A-216 on the campus of UW Seattle.
a few seconds ago Like
C.J. Adams-Collier http://www.ling.lu.se/persons/Sidney/praate/frames.html

I just watched an interesting TED talk by Pankaj Ghemawat (of which the most important points are summarised in a TED blog post) about the world not being as globalised as people expect [1]. One point is that only 2% of traditional voice phone calling minutes (and ~6% when you include VOIP) are for international calls which is less than most people expect. After reading that it occurred to me that most of the included value in my mobile phone contract goes unused, I pay for extra data transfer and it includes voice calling credit that I don t use. So out of $450 of included calls I typically use much less than $50. $400 of calls to even the most expensive countries is about 100 minutes of talking. So the logical thing to do is to find people in other countries to call. If you are involved in the FOSS community and would like to speak to me then send me an email with your phone number, time zone, and a range of times that are convenient. I won t make any promises about calling you soon (I could use up my monthly credit on a single call), but I will call you eventually. I will also send email to some people I know by email and suggest a chat. Some years ago I did this with people who were involved in SE Linux development and it seemed to help the development of the SE Linux community. Also if anyone in Australia wants to speak to me then that s OK too. While Pankaj s talk inspired me to call people I m not dedicated to calling other countries.

• [1] http://tinyurl.com/czmmwxj

Yeah, wait, what? No, really! I took a look through clojure-py, and found out something pretty rad. In order to get clojurepy scripts to load other clojurepy, it had to shim out and add a new importer for clojure scripts. What this really means, is that you can import Clojure from Python, which is pretty sweet. I abused the internals here, and added it to dput-ng as a quick hack. Basically, I just had to import clojure.main, and let the main function set up the sys shim. I would set it up in dput, but I d have to import that module anyway. Remember, when doing this, the namespace ((ns ...)) needs to match the filename. So, I present the first Clojure hook for dput-ng:

; Copyright (c) Paul R. Tagliamonte <paultag@debian.org>, 2012, under the
; terms of dput-ng itsself.
(ns clojtest
  (:require dput.core
            dput.exceptions))
(defn log [x]  ; for debug output
  (.debug dput.core/logger x))
(defn dput-checker [changes profile interface]
  (cond (>= (-> changes (.get "maintainer") (.find "arno@debian.org")) 0)
    (throw (dput.exceptions/HookException. "Maintainer's Arno. Aborting upload"))
  :else
    (log "Nah, it's not arno, we're good")))

This, of course, checks if the maintainer is Arno, and throws a fit if it is. Gist is over on gist.github. After, Algernon added his own checker, which throws a fit if you have a package in the B-Ds that you aught to not use (such as dpatch)

; Copyright (c) Gergely Nagy <algernon@debian.org>, 2012, under the
; terms of dput-ng itself.
(ns bd-blacklist
  (:require dput.core
            dput.exceptions
            dput.dsc))
(defn prune-build-deps
  "Prune a string representation of the build-depends so that only a
  list of packages remain."
  [bd-string]
  (map #(first (-> % (.strip) (.split " "))) (.. bd-string (split ","))))
(defn has-blacklisted?
  "Given a dsc file and a blacklist, check if any of the
  build-depencencies are in that list. Throws an error if there are
  matches."
  [dsc-file blacklist]
  (let [dsc (dput.dsc/parse_dsc_file dsc-file)
        build-deps (prune-build-deps (.. dsc (get "build-depends")))]
    (if-let [bad-bd (some blacklist build-deps)]
      (throw (dput.exceptions/HookException. (str "Blacklisted build-dependency found: " bad-bd)))
      (-> dput.core/logger (.trace "Build-Dependencies do not have anything on the blacklist")))))
(defn blacklist-checker
  "Checks whether the dsc has blacklisted build-dependencies, ignores
  the check when no dsc is to be found."
  [changes profile interface]
  (if-let [dsc-file (.. changes (get_dsc))]
    (has-blacklisted? dsc-file (set (get profile "bd-blacklist")))
    (-> dput.core/logger (.trace "No .dsc found, build-dependencies cannot be checked"))))

You can check that out in the git tree Hacks welcome!

Have you ever heard someone talking extensively about Cloud Computing or generally Clouds? and have you ever noticed the fact many people (even the ones who present themselves as experts) don t really understand what a Cloud is at all? That happened to me multiple times and one of the most common misunderstandings is many see the Cloud as something being on the internet. Many companies add a little logo representing a cloud on their frontpage and without a single change on their infrastructure (but surely with a price increment) they start calling their products as being on the Cloud. Given the lack of knowledge about this specific topic people tend to buy the product presented as being on the Cloud without understanding what they really bought. But what Cloud Computing really means? it took several years and more than fifteen drafts to the National Institute of Standards and Technology s (NIST) to find a definition. The final accepted proposal:

Cloud computing is a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction.

The above definition requires a few more clarifications specifically when it comes to understand where should we focus on while checking for a Cloud Computing solution. A few key points:

1. On-demand self-service: every consumer will be able to unilaterally provision multiple computing capabilities like server time, storage, bandwidth, dedicated RAM or CPU without requiring any sort of human interaction from their respective Cloud providers.
2. Rapid elasticity and scalability: all the computing capabilities outlined above can be elastically provisioned and released depending on how much demand my company will have in a specific period of time. Suppose the X company is launching a new product today and it expects a very large number of customers. The X company will add more resources to their Cloud for the very first days (where they suppose the load to be very high) and then it ll scale the resources back as they were before. Elasticity and scalability permit the X company to improve and enhance their infrastructure when they need it with an huge saving in monetary terms.
3. Broad network access: capabilities are available over the network and accessed through standard mechanisms that promote use by heterogeneous thin or thick client platforms (e.g., mobile phones, tablets, laptops, and workstations).
4. Measured service: Cloud systems allow maximum transparency between the provider and the consumer, the usage of all the resources is monitored, controlled, and reported. The consumer knows how much will spend, when and in how long.
5. Resource pooling: each provider s computing resources are pooled to serve multiple consumers at the same time. The consumer has no control or knownledge over the exact location of the provided resources but may be able to specify location at a higher level of abstraction (e.g., country, state, or datacenter).
6. Resources price: when buying a Cloud service make sure the cost for two units of RAM, storage, CPU, bandwidth, server time is exactly the double of the price of one unit of the same capability. An example, if a provider offers you one hour of bandwitdh for 1 Euro, the price of two hours will have to be 2 Euros.

The Cloud Computing technology is reasonably the future but can we trust Cloud providers? Are we sure that no one will ever have access to our files except us? and what about governments interested in acquiring a specific customer data hosted on the Cloud? I always suggest to read deeply both the Privacy Policy and Terms of Use of a certain service before signing in especially when it comes to choose a Cloud storage provider. Many providers have the same aspect, they seem to provide the same resources, the same amount of storage for the same price but legally they may present different problems, and that is the case of Spideroak vs Dropbox. Quoting from the Dropbox s Privacy Policy:

Compliance with Laws and Law Enforcement Requests; Protection of DropBox s Rights. We may disclose to parties outside Dropbox files stored in your Dropbox and information about you that we collect when we have a good faith belief that disclosure is reasonably necessary to (a) comply with a law, regulation or compulsory legal request; (b) protect the safety of any person from death or serious bodily injury; (c) prevent fraud or abuse of DropBox or its users; or (d) to protect Dropbox s property rights. If we provide your Dropbox files to a law enforcement agency as set forth above, we will remove Dropbox s encryption from the files before providing them to law enforcement. However, Dropbox will not be able to decrypt any files that you encrypted prior to storing them on Dropbox.

It s evident that Dropbox employees can access your data or be forced by legal process to turn over your data unencrypted. On the other side, Spideroak on its latest update to its Privacy Policy states that data stored on their Cloud is encrypted and inaccessible without user s key, which is stored locally on user s computers. And what about the latest research paper, titled Cloud Computing in Higher Education and Research Institutions and the USA Patriot Act written by the legal experts of the University of Amsterdam s Institute for Information Law stating the anti-terror Patriot Act could be theoretically used by U.S. law enforcement to bypass strict European privacy laws to acquire citizen data within the European Union without their consensus? The only requirement for the data acquisition is the provider being an U.S company or an European company conducting systematic business in the U.S. For example an Italian company storing their documents (protected by the European privacy laws and under the general Italian jurisdiction) on a provider based in Europe but conducting systematic business in the United States, could be forced by U.S. law enforcement to transfer data to the U.S. territory for inspection by law enforcement agencies. Does someone really care about the privacy of companies, consumers and users at all? or better does privacy exists at all for the millions of the people that connect to the internet every day?