I'm currently running Debian's rolling release (aka "testing") on my main machine and have added some stuff to make that nicer. First thing I have is configuration and package management. Since I have relatively few machines, I am using a metapackage per machine that installs some configuration files with changes that I want. The metapackages depend on packages that I need installed so that I can mark all other packages as being automatically installed. The metapackages are also useful for documenting why I have things installed. It depends on things like task-laptop from tasksel, hardware support packages, the GUI I use, games I play often and so on. My laptop does not have a CD/DVD drive so I have some metapackages to fool apt into ignoring dependencies on CD/DVD related packages I don't need. I'm building the metapackages using equivs-build and a small Makefile. I use the File: header supported by equivs-build for installing config files. I have popcon installed and enabled but I don't want it to leak the names of the metapackages so I have added a prefix to my metapackages and modified the popcon cron job to remove anything containing that prefix. I also don't want apt to ever remove the metapackages so I mark them as Important: yes and configure apt to never autoremove them.

--- /etc/cron.daily/popularity-contest~
+++ /etc/cron.daily/popularity-contest
@@ -71,8 +71,8 @@
 # try to post the report through http POST
 if [ "$SUBMITURLS" ] && [ "yes" = "$USEHTTP" ]; then
     for URL in $SUBMITURLS ; do
-   if setsid /usr/share/popularity-contest/popcon-upload \
-       -u $URL -f $POPCON 2>/dev/null ; then
+   if grep -v myprefix- $POPCON   setsid /usr/share/popularity-contest/popcon-upload \
+       -u $URL 2>/dev/null ; then
        SUBMITTED=yes
    else
        logger -t popularity-contest "unable to submit report to $URL."
@@ -94,7 +94,7 @@
        echo "MIME-Version: 1.0"
        echo "Content-Type: text/plain"
        echo
-       cat $POPCON
+       grep -v myprefix- $POPCON
    )   do_sendmail
    SUBMITTED=yes
     else

/etc/apt/apt.conf.d/99metapackages:

APT::NeverAutoRemove   "^myprefix-.*";  ;

I am using Raphael Geissert's mirror redirector in order to automatically use up-to-date and hopefully non-broken mirrors. Unfortunately this often causes apt to complain about hash sum mismatches and then proceed to forget about all packages. I work around this by always running apt-get update in a loop until it succeeds.

while ! apt-get update ; do sleep 1m; done

A lot of the time I need to install packages from outside of testing. So my sources.list contains lines for testing, unstable and experimental. I have some apt pinning so that by default I only have packages from testing but if I manually upgrade some packages to unstable or experimental, then I will get upgrades within that suite until those packages migrate down to unstable or testing. The apt pinning needs priorities between 1000 and 500 for this to work nicely. I also pin some things like lintian, debian-policy and devref to unstable/experimental since having old versions of those is not useful. /etc/apt/sources.list:

# testing
deb http://security.debian.org/ testing/updates main contrib non-free
deb-src http://security.debian.org/ testing/updates main contrib non-free
deb http://http.debian.net/debian/ testing main contrib non-free
deb-src http://http.debian.net/debian/ testing main contrib non-free
# unstable
deb http://http.debian.net/debian/ unstable main contrib non-free
deb-src http://http.debian.net/debian/ unstable main contrib non-free
# experimental
deb http://http.debian.net/debian/ experimental main contrib non-free
deb-src http://http.debian.net/debian/ experimental main contrib non-free

/etc/apt/preferences.d/system:

Package: *
Pin: release a=testing
Pin-Priority: 800
Package: *
Pin: release a=unstable
Pin-Priority: 700
Package: *
Pin: release a=experimental
Pin-Priority: 600

/etc/apt/preferences.d/packages:

Package: lintian
Pin: release a=unstable
Pin-Priority: 900
Package: lintian
Pin: release a=experimental
Pin-Priority: 910
Package: debian-policy
Pin: release a=unstable
Pin-Priority: 999
Package: developers-reference
Pin: release a=unstable
Pin-Priority: 999

I have a few configuration files and a cron job to make all programs dump core files when they crash so that I can file bugs, even for random crashes that are not easy to reproduce. I enabled some kernel settings with sysctl, lifted some security limits to enable core dumps, and added a cron job to delete old core dumps and notify me of new core dumps. In my shell configuration I also turn on two glibc options to cause programs to crash when they have improper memory management. I also have a second machine I use for bug discovery where I have lots of stuff installed and everything apt pinned in the opposite way; experimental > unstable > testing. When I have time I use this machine to do testing of packages I use, classes of packages that I care about (such as games) and sometimes packages I do not use. /etc/sysctl.d/corefiles.conf:

fs.suid_dumpable = 1
kernel.core_uses_pid = 1
kernel.core_pattern = /var/cache/corefiles/core-%p-%u-%g-%s-%t-%h-%e

/etc/security/limits.d/corefiles.conf:

*              soft    core            unlimited
*              hard    core            unlimited

/etc/cron.daily/corefiles:

#!/bin/sh
mkdir -p /var/cache/corefiles
chmod 2777 /var/cache/corefiles
if [ $(find /var/cache/corefiles -mtime +100 -a ! -type d   wc -l) -gt 0 ]; then
    echo deleting:
    find /var/cache/corefiles -mtime +100 -a ! -type d
    find /var/cache/corefiles -mtime +100 -a ! -type d -print0   xargs -0 rm -f
fi
if [ $(find /var/cache/corefiles ! -type d   wc -l) -gt 0 ] ; then
    echo still present:
    find /var/cache/corefiles ! -type d
fi

~/.bash.d/malloc:

export MALLOC_CHECK_=2
export MALLOC_PERTURB_=$(($RANDOM % 255 + 1))

I unfortunately need some packages from contrib/non-free, so I have a cron job to let me know when I accidentally install new packages from contrib/non-free.

@daily diffcmdoutput ~/.cache/non-free-contrib aptitude search ~i~snon-free\ ~i~scontrib

I backup my dpkg package selections and debconf databases.

@daily diffcmdoutput ~/backup/packages dpkg --get-selections
@daily diffcmdoutput ~/backup/config debconf-get-selections 2> /dev/null

I notify myself of changes to the list of new packages so that I can review them, install any useful/interesting ones and tell aptitude to forget them all.

@daily diffcmdoutput ~/.cache/new aptitude search ~N

I notify myself of changes to the list of packages I have installed that are not up-to-date packages from testing. This helps me catch packages removed from testing/unstable/etc that I use.

@daily diffcmdoutput ~/.cache/apt-show-versions sh -c "apt-show-versions   grep -v '/testing uptodate'"

I notify myself of packages that I maintain that are having issues migrating to testing. I considered doing the same for teams I am involved in but they aren't particularly functional teams so there would be a lot of noise.

@daily grep-excuses 'Paul Wise'

I notify myself of RC bugs that apply to testing and are installed. The list is so long that it just makes me depressed instead of motivated to help fix RC bugs so I only notify myself of changes. Even then I rarely do anything other than delete the notifications. If you are looking for ways to help Debian, fixing RC bugs is a great choice.

@daily diffcmdoutput ~/.cache/rcbugs rc-alert -d T --exclude-tags IP+MR

I notify myself of packages that are orphaned or need a new maintainer. There are usually so many packages in this list that it is not useful, so I only notify myself of changes to the list. I rarely adopt packages because I feel overloaded already. If you are looking for ways to help Debian, adopting packages is a good choice.

@daily wnpp-alert --diff

One of my packages is for interacting with servers on the Internet, so I need to run tests periodically to ensure the package works. I do that with a simple Makefile but maybe I need to move to autopkgtest, need to find out if it saves data between runs first.

@monthly cd ~/devel/debian/tests ; make

I install debsecan so that I get notified of security updates in unstable and new security issues that are not fixed yet. The way debsecan works is that it notifies about changes in security issues and updates and also includes a full list of all known unfixed issues. I generally install security updates from unstable when I see them. The list of unchanged issues is so long that it makes me wonder how many times I've been cracked already. The oldest issue goes back to 2002 but most of them are 2010 or later. The various parts of WebKit are by far the worst security offenders. I don't bother with the white-listing functionality due to the quantity of security issues and because it isn't possible to add a comment about each white-list item. If you want to get involved with the security team, reporting issues with the data in the security tracker is a good idea. I subscribe to the ftpmaster RSS feeds for new and removed packages to keep up to date with changes in the archive. A lot of the above applies to running systems based on Debian stable too. If you have any other thoughts about running Debian systems, please blog about them. The diffcmdoutput command used above is a simple shell script:

#!/bin/sh
cache="$1"
shift
temp="$(mktemp "$cache"XXXXXXXXXXXXXX)"
"$@" > "$temp"
diff --unified "$cache" "$temp"
mv --force "$temp" "$cache"

The summer came finally to an end and (yes in Berlin we still had 20 C end of October), unfortunately, so did GSoC with it. This has been the second time for SHOGUN to be in GSoC. For those unfamiliar with SHOGUN - it is a very versatile machine learning toolbox that enables unified large-scale learning for a broad range of feature types and learning settings, like classification, regression, or explorative data analysis. I again played the role of an org admin and co-mentor this year and would like to take the opportunity to summarize enhancements to the toolbox and my GSoC experience: In contrast to last year, we required code-contributions in the application phase of GSoC already, i.e., a (small) patch was mandatory for your application to be considered. This reduced the number of applications we received: 48 proposals from 38 students instead of 70 proposals from about 60 students last year but also increased the overall quality of the applications. In the end we were very happy to get 8 very talented students and have the opportunity of boosting the project thanks to their hard and awesome work. Thanks to google for sponsoring three more students compared to last GSoC. Still we gave one slot back to the pool for good to the octave project (They used it very wisely and octave will have a just-in-time compiler now, which will benefit us all!). SHOGUN 2.0.0 is the new release of the toolbox including of course all the new features that the students have implemented in their projects. On the one hand, modules that were already in SHOGUN have been extended or improved. For example, Jacob Walker has implemented Gaussian Processes (GPs) improving the usability of SHOGUN for regression problems. A framework for multiclass learning by Chiyuan Zhang including state-of-the-art methods in this area such as Error-Correcting Output Coding (ECOC) and ShareBoost, among others. In addition, Evgeniy Andreev has made very important improvements w.r.t. the accessibility of SHOGUN. Thanks to his work with SWIG director classes, now it is possible to use python for prototyping and make use of that code with the same flexibility as if it had been written in the C++ core of the project. On the other hand, completely new frameworks and other functionalities have been added to the project as well. This is the case of multitask learning and domain adaptation algorithms written by Sergey Lisitsyn and the kernel two-sample or dependence test by Heiko Strathmann. Viktor Gal has introduced latent SVMs to SHOGUN and, finally, two students have worked in the new structured output learning framework. Fernando Iglesias made the design of this framework introducing the structured output machines into SHOGUN while Michal Uricar has implemented several bundle methods to solve the optimization problem of the structured output SVM. It has been very fun and interesting how the work done in different projects has been put together very early, even during the GSoC period. Only to show an example of this dealing with the generic structured output framework and the improvements in the accessibility. It is possible to make use of the SWIG directors to implement the application specific mechanisms of a structured learning problem instance in python and then use the rest of the framework (written in C++) to solve this new problem. Students! You all did a great job and I am more than amazed what you all have achieved. Thank you very much and I hope some of you will stick around. Besides all these improvements it has been particularly challenging for me as org admin to scale the project. While I could still be deeply involved in each and every part of the project last GSoC, this was no longer possible this year. Learning to trust that your mentors are doing the job is something that didn't come easy to me. Having had about monthly all-hands meetings did help and so did monitoring the happiness of the students. I am glad that it all worked out nicely this year too. Again, I would like to mention that SHOGUN improved a lot code-base/code-quality wise. Students gave very constructive feedback about our (lack) of proper Vector/Matrix/String/Sparse Matrix types. We now have all these implemented doing automagic memory garbage collection behind scenes. We have started to transition to use Eigen3 as our matrix library of choice, which made quite a number of algorithms much easier to implement. We generalized the Label framework (CLabels) to be tractable for not just classification and regression but multitask and structured output learning. Finally, we have had quite a number of infrastructure improvements. Thanks to GSoC money we have a dedicated server for running the buildbot/buildslaves and website. The ML Group at TU Berlin does sponsor virtual machines for building SHOGUN on Debian and Cygwin. Viktor Gal stepped up providing buildslaves for Ubuntu and FreeBSD. Gunnar Raetschs group is supporting redhat based build tests. We have Travis CI running testing pull requests for breakage even before merges. Code quality is now monitored utilizing LLVMs scan-build. Bernard Hernandez appeared and wrote a fancy new website for SHOGUN. A more detailed description of the achievements of each of the students follows:

• Kernel Two-sample/Dependence test
  • Student: Heiko Strathmann
  • Mentor: Arthur Gretton, Soeren Sonnenburg
  Heiko Strathmann, mentored by Arthur Gretton, worked on a framework for kernel-based statistical hypothesis testing. Statistical tests to determine whether two random variables are are equal/different or are statistically independent are an important tool in data-analysis. However, when data are high-dimensional or in non-numerical form (documents, graphs), classical methods fail. Heiko implemented recently developed kernel-based generalisations of classical tests which overcome these issues by representing distributions in high dimensional so-called reproducing kernel Hilbert spaces. By doing so, theoretically any pair samples can be distinguished. Implemented methods include two-sample testing with the Maximum Mean Discrepancy (MMD) and independence testing using the Hilbert Schmidt Independence Criterion (HSIC). Both methods come in different flavours regarding computational costs and test constructions. For two-sample testing with the MMD, a linear time streaming version is included that can handle arbitrary amounts of data. All methods are integrated into a newly written flexible framework for statistical testing which will be extended in the future. A book-style tutorial with descriptions of algorithms and instructions how to use them is also included.
• Implement multitask and domain adaptation algorithms
  • Student: Sergey Lisityn
  • Mentor: Christian Widmer
  Like Heiko, Sergey Lisitsyn did participate in the GSoC programme for the second time. This year he focused on implementing multitask learning algorithms. Multitask learning is a modern approach to machine learning that learns a problem together with other related problems at the same time using a shared representation. This approach often leads to a better model for the main task, because it allows the learner to use the commonality among the tasks. During the summer Sergey has ported a few algorithms from the SLEP and MALSAR libraries with further extensions and improvements. Namely, L12 group tree and L1q group multitask logistic regression and least squares regression, trace-norm multitask logistic regression, clustered multitask logistic regression, basic group and group tree lasso logistic regression. All the implemented algorithms use COFFIN framework for flexible and efficient learning and some of the algorithms were implemented efficiently utilizing the Eigen3 library.
• Implementation of / Integration via existing GPL code of latent SVMs.
  • Student: Viktor Gal
  • Mentor: Alexander Binder
  A generic latent SVM and additionally a latent structured output SVM has been implemented. This machine learning algorithm is widely used in computer vision, namely in object detection. Other useful application fields are: motif finding in DNA sequences, noun phrase coreference, i.e. provide a clustering of the nouns such that each cluster refers to a single object. It is based on defining a general latent feature Psi(x,y,h) depending on input variable x, output variable y and latent variable h. Deriving a class from the base class allows the user to implement additional structural knowledge for efficient maximization of the latent variable or alternative ways of computation or on-the-fly loading of latent features Psi as a function of the input, output and latent variables.
• Bundle method solver for structured output learning
  • Student: Michal Uricar
  • Mentor: Vojtech Franc
  We have implemented two generic solvers for supervised learning of structured output (SO) classifiers. First, we implemented the current state-of-the-art Bundle Method for Regularized Risk Minimization (BMRM) [Teo et al. 2010]. Second, we implemented a novel variant of the classical Bundle Method (BM) [Lamarechal 1978] which achieves the same precise solution as the BMRM but in time up to an order of magnitude shorter [Uricar et al. 2012]. Among the main practical benefits of the implemented solvers belong their modularity and proven convergence guarantees. For training a particular SO classifier it suffices to provide the solver with a routine evaluating the application specific risk function. This feature is invaluable for designers who can concentrate on tuning the classification model instead of spending time on developing new optimization solvers. The convergence guarantees remove the uncertainty inherent in use of on-line approximate solvers. The implemented solvers have been integrated to the structured output framework of the SHOGUN toolbox and they have been tested on real-life data.
• Built generic structured output learning framework
  • Student: Fernando Jose Iglesias Garcia
  • Mentor: Nico Goernitz
  During GSoC 2012 Fernando implemented a generic framework for structured output (SO) problems. Structured output learning deals with problems where the prediction made is represented by an object with complex structure, e.g. a graph, a tree or a sequence. SHOGUN's SO framework is flexible and easy to extend [1]. Fernando implemented a na ve cutting plane algorithm for the SVM approach to SO [2]. In addition, he coded a case of use of the framework for labelled sequence learning, the so-called Hidden Markov SVM. The HM-SVM can be applied to solve problems in different fields such as gene prediction in bioinformatics or speech to text in pattern recognition. [1] Class diagram of the SO framework.
  [2] Support Vector Machine Learning for Interdependent and Structured Output Spaces.
• Improving accessibility to shogun
  • Student: Evgeniy Andreev
  • Mentor: Soeren Sonnenburg
  During the latest google summer of code Evgeniy has improved the Python modular interface. He has added new SWIG-based feature - director classes, enabling users to extend SHOGUN classes with python code and made SHOGUN python 3 ready. Evgeniy has also added python's protocols for most usable arrays (like vectors, matrices, features) which makes possible to work with Shogun data structures just like with numpy arrays with no copy at all. For example one can now modify SHOGUN's RealFeatures in place or use.
• Implement Gaussian Processes and regression techniques
  • Student: Jacob Walker
  • Mentor: Oliver Stegle
  Jacob implemented Gaussian Process Regression in the Shogun Machine Learning Toolbox. He wrote a complete implementation of basic GPR as well as approximation methods such as the FITC (First Independent Training Conditional) method and the Laplacian approximation method. Users can utilize this feature to analyze large scale data in a variety of fields. Scientists can also build on this implementation to conduct research extending the capability and applicability of Gaussian Process Regression.
• Build generic multiclass learning framework
  • Student: Chiyuan Zhang
  • Mentor: Cheng Soon Ong
  Chiyuan defined a new multiclass learning framework within SHOGUN. He re-organized previous multiclass learning components and refactored the CMachine hierarchy in SHOGUN. Then he generalized the existing one-vs-one and one-vs-rest multiclass learning scheme to general ECOC encoding and decoding strategies. Beside this, several specific multiclass learning algorithms are added, including ShareBoost with feature selection ability, Conditional Probability Tree with online learning ability, and Relaxed Tree.

Review: Devil Take the Hindmost, by Edward Chancellor

Publisher:	Plume
Copyright:	1999
Printing:	June 2000
ISBN:	0-452-28180-6
Format:	Trade paperback
Pages:	349

Subtitled A History of Financial Speculation, Devil Take the Hindmost is exactly that: a history, with chapters on notable speculative bubbles or surges going back to the 1630s. The first significant bubble described is, of course, the notorious tulip speculation in Holland, but Chancellor is more thorough than any previous history I've read. There are chapters on the early stock market schemes, the South Sea bubble, 1820s emerging markets, the railway mania of 1845, the Gilded Age, the late 1970s and early 1980s, and the Japanese real estate bubble of the 1980s, as well as the mandatory look at the stock market crash of 1929. Each is thorough and detailed; indeed, perhaps slightly too detailed, as the clear narrative picture can occasionally groan under the weight of names and capsule biographies. The first thing one should realize about this book is that, despite the apparent currency of the topic, it was written in 1999. That means it predates both the subprime real estate collapse and the dot.com boom, as well as some more minor but possibly relevant events such as the Enron electricity market manipulation. Japan is the most recent bubble that Chancellor can cover; one can only speculate as to what he would make of the 2008 financial crash. The other focus to be aware of is the omission of sovereign debt. Chancellor is exclusively focused on private markets, usually stock markets, and (apart from the chapter on Japan) Western markets, mostly in the US and the UK. Governments play a role in some of the early speculative bubbles (most notably the South Sea bubble), but Chancellor's topic is private speculative frenzies, not the risks and effects of government borrowing. Interrogation of the role of government is mostly limited to cursory discussions of the impact of regulation, mostly in the negative. More on that in a moment. This is a history as opposed to either a polemic or a prescription. Chancellor does not set out to explain how to curb financial speculation. Insofar as he expresses an opinion on it, he presents speculation as a force of unbridled freedom in natural tension with regulation and financial conservatism and sees a cyclical pattern of speculative frenzy, collapse, retrenchment, and then renewed speculation. Speculation, as presented here, is more akin to a force of nature than, as seen in most other books on the topic, a phenomenon of greed that should be limited or prevented. Indeed, Chancellor seems skeptical of the entire project of preventing or limiting speculation, and presents considerable reason for the reader to become skeptical as well. The history he presents is one of speculative manias adapting to and circumventing every control that governments and markets have attempted to place on them. When certain practices are outlawed (and some modern complex financial practices have surprisingly long histories), people just find loopholes or alternative approaches through which to accomplish the same thing. Or the speculative mania takes an unexpected new form. Nor do the manias seem to be decreasing in severity as more regulation, custom, and restrictions attempt to blunt them. Even without adding the financial collapse of 2008, the impacts of the speculative frenzies documented here seem, if anything, to be growing. This is, therefore, a relatively straightforward history, but it's an entertaining one, filling in details that I'd not previously been aware of. The complexity of financial schemes as early as the 1600s was particularly eye-opening. There's a tendency to believe that complex derivative schemes are an artifact of modern financial markets, but they've been a part of speculation for as long as Chancellor goes back. Computers may have added layers to the mathematics, but they haven't added as much to the legal strategems and levels of indirection as one might think. Chancellor emphasizes that throughout: speculation involves a lot of complexity, not necessarily to deceive but to construct situations in which it's in the interest (at least short term) of all involved parties to continue and escalate the speculation. While Chancellor doesn't provide much encouragement to the reader hoping to find ways to curb speculation, he does show some commonalities that provide a bit of a handle on how speculation starts. The one that lept out at me was leverage. Every speculative bubble documented in Devil Take the Hindmost involves borrowing money to reinvest. The mechanism keeps changing, from straightforward loans during the tulip craze to much more complex constructions in modern finance, primarily in response to regulation that attempted to reduce leverage but only banned one specific technique. But whatever method is used, at the bottom is always some scheme for purchasing an asset using debt, magnifying both positive and negative price swings. Some variation of buying stocks on margin is particularly common; Chancellor documents how it was reinvented despite the limits on straight margin purchases instituted after the Great Depression. There is no recommendation here for any regulatory or legal action, but it's obvious from the repetition of history that limiting leverage would be the place to focus attention. I enjoyed this history, but it can be dry, and at times is a bit overstuffed with names, unimportant details, and microbiographies. It's also unfortunate that Chancellor missed two of the most significant (and two very different) speculative bubbles through the timing of his work. It would have been interesting to see them through the lens of his history. What we have is still worth reading, but also somewhat frustrating in Chancellor's refusal to draw or at least underline general conclusions. Devil Take the Hindmost is long on history but short on analysis, and while I appreciate the historian's focus in a source of data, it left me somewhat unsatisfied. Rating: 6 out of 10

I work at the University of Oslo looking after the computers, mostly on the unix side, but in general all over the place. I am also a member (and currently leader) of the NUUG association, which in turn make me a member of USENIX. NUUG is an member organisation for us in Norway interested in free software, open standards and unix like operating systems, and USENIX is a US based member organisation with similar targets. And thanks to these memberships, I get all issues of the great USENIX magazine ;login: in the mail several times a year. The magazine is great, and I read most of it every time. In the last issue of the USENIX magazine ;login:, there is an article by Stuart Kendrick from Fred Hutchinson Cancer Research Center titled "What Takes Us Down" (longer version also available from his own site), where he report what he found when he processed the outage reports (both planned and unplanned) from the last twelve years and classified them according to cause, time of day, etc etc. The article is a good read to get some empirical data on what kind of problems affect a data centre, but what really inspired me was the kind of reporting they had put in place since 2000. The centre set up a mailing list, and started to send fairly standardised messages to this list when a outage was planned or when it already occurred, to announce the plan and get feedback on the assumtions on scope and user impact. Here is the two example from the article: First the unplanned outage:

Subject:     Exchange 2003 Cluster Issues
Severity:    Critical (Unplanned)
Start: 	     Monday, May 7, 2012, 11:58
End: 	     Monday, May 7, 2012, 12:38
Duration:    40 minutes
Scope:	     Exchange 2003
Description: The HTTPS service on the Exchange cluster crashed, triggering
             a cluster failover.
User Impact: During this period, all Exchange users were unable to
             access e-mail. Zimbra users were unaffected.
Technician:  [xxx]

Next the planned outage:

Subject:     H Building Switch Upgrades
Severity:    Major (Planned)
Start:	     Saturday, June 16, 2012, 06:00
End:	     Saturday, June 16, 2012, 16:00
Duration:    10 hours
Scope:	     H2 Transport
Description: Currently, Catalyst 4006s provide 10/100 Ethernet to end-
	     stations. We will replace these with newer Catalyst
	     4510s.
User Impact: All users on H2 will be isolated from the network during
     	     this work. Afterward, they will have gigabit
     	     connectivity.
Technician:  [xxx]

He notes in his article that the date formats and other fields have been a bit too free form to make it easy to automatically process them into a database for further analysis, and I would have used ISO 8601 dates myself to make it easier to process (in other words I would ask people to write '2012-06-16 06:00 +0000' instead of the start time format listed above). There are also other issues with the format that could be improved, read the article for the details. I find the idea of standardising outage messages seem to be such a good idea that I would like to get it implemented here at the university too. We do register planned changes and outages in a calendar, and report the to a mailing list, but we do not do so in a structured format and there is not a report to the same location for unplanned outages. Perhaps something for other sites to consider too?

I ve been a wannabe GTD afficionado for some years. I ve wanted to do it, but managing lists has always been something that has too much friction, overhead, or whatever. Finally, though, I think I might have found a way to manage lists that works. My use-case isn t unique, although I will concede I m perhaps being more dogmatic than most. I want something that:

• Is electronic (yes, the hipster PDA is a cool idea, but my handwriting is beyond woeful and I already carry enough crap in my pockets as it is);
• Works offline (because I often do);
• Will work on both my phone and laptop (because I want to have my lists with me when I m not at my computer, but long-form data entry or manipulation on a phone is painful);
• Makes it easy to add, browse, modify, and above all remove items; and
• Is mine (no central databases I don t control, proprietary apps on my phone, etc).

My previous attempt was a tool I called tagnote it was a vim-outliner file full of hierarchically organised outliner entries, with tags inlined. It was a neat idea, but it wasn t smooth to add/browse/delete items, and didn t work with my phone at all (trying to use vim for any length of time on a bottom-of-the-range Android phone would kill me). The current iteration, as the title of this post suggests, is a list manager that entirely uses e-mail. It really is a perfect symbiosis:

• I want lists of text items with titles (Subject:), potentially other metadata (X-Whatever headers), and possibly some notes (the body of the e-mail);
• I need to be able to browse and remove completed items (that s what e-mail clients are for);
• Getting new items into the system is trivial (Anything I send to my PDA s e-mail address goes straight into the INBOX, which I can then process as time permits); and
• Syncing between my laptop and PDA is as simple as offlineimap and K-9 mail.

So what have I got, exactly? It s fairly straightforward:

• An IMAP account on my existing mail server;
• A slightly tweaked copy of mutt (different colours so I don t confuse myself, and a different layout of the index page to get rid of unnecessary columns);
• Another offlineimap account;
• Another K-9 mail account;
• Note2self (a neat little app to take a typed or voice-transcribed note and e-mail it to a pre-set address) on the phone, pre-programmed to e-mail any notes I write to the PDA s inbox; and
• A small shell script to make it trivial to create new lists (which has to be done on the IMAP server for reasons of offlineimap), add new items to a list, sync my lists (in other words, run offlineimap ), display my lists (in other words, run mutt ), and process my tickler file.

That last point is the one I m really happy I achieved. I ve always been a fan of hide it until you need it , but my previous system didn t let me do that. Now, though, I have a separate list called tickler, and all the items in there have an X-Tickle header, which specifies the date I want to see them. Each night a cronjob runs through the tickler and moves anything for today into the INBOX. An X-Tickle-Repeat header lets me have things that repeat over and over again. So in short, using entirely open-source tools and a couple of hours of my time doing things I enjoy anyway (shell scripts! woo!), I ve now got a list manager that doesn t get in my way more than it absolutely has to. We ll see how long I last this time before I feel the urge to improve my lists again.

This blog, together with Romain's, is one of the main homes of stories about how Rcpp can help with getting code to run faster in the context of the R system for statistical programming and analysis. By making it easier to get already existing C or C++ code to R, or equally to extend R with new C++ code, Rcpp can help in getting stuff done. And it is often fairly straightforward to do so. In this context, I have a nice new example. And for once, it is work-related. I generally cannot share too much of what we do there as this is, well, proprietary, but I have this nice new example. The other day, I was constructing (large) time series of implied volatilities. Implied volatilities can be thought of as the complement to an option's price: given a price (and all other observables which can be thought of as fixed), we compute an implied volatility price (typically via the standard Black-Scholes model). Given a changed implied volatility, we infer a new price -- see this Wikipedia page for more details. In essence, it opens the door to all sorts of arbitrage and relative value pricing adventures. Now, we observe prices fairly frequently to create somewhat sizeable time series of option prices. And each price corresponds to one matching implied volatility, and for each such price we have to solve a small and straightforward optimization problem: to compute the implied volatility given the price. This is usually done with an iterative root finder. The problem comes from the fact that we have to do this (i) over and over and over for large data sets, and (ii) that there are a number of callbacks from the (generic) solver to the (standard) option pricer. So our first approach was to just call the corresponding function GBSVolatility from the fOption package from the trusted Rmetrics project by Diethelm Wuertz et al. This worked fine, but even with the usual tricks of splitting over multiple cores/machines, it simply took too long for the resolution and data amount we desired. One of the problems is that this function (which uses the proper uniroot optimizer in R) is not inefficient per se, but simply makes to many function call back to the option pricer as can be seen from a quick glance at the code. The helper function .fGBSVolatility gets called time and time again:

R> GBSVolatility
function (price, TypeFlag = c("c", "p"), S, X, Time, r, b, tol = .Machine$double.eps, 
    maxiter = 10000) 
 
    TypeFlag = TypeFlag[1]
    volatility = uniroot(.fGBSVolatility, interval = c(-10, 10), 
        price = price, TypeFlag = TypeFlag, S = S, X = X, Time = Time, 
        r = r, b = b, tol = tol, maxiter = maxiter)$root
    volatility
 
<environment: namespace:fOptions>
R> 
R> .fGBSVolatility
function (x, price, TypeFlag, S, X, Time, r, b, ...) 
 
    GBS = GBSOption(TypeFlag = TypeFlag, S = S, X = X, Time = Time, 
        r = r, b = b, sigma = x)@price
    price - GBS
 
<environment: namespace:fOptions>

So the next idea was to try the corresponding function from my RQuantLib package which brings (parts of) QuantLib to R. That was seen as been lots faster already. Now, QuantLib is pretty big and so is RQuantLib, and we felt it may not make sense to install it on a number of machines just for this simple problem. So one evening this week I noodled around for an hour or two and combined (i) a basic Black/Scholes calculation and (ii) a standard univariate zero finder (both of which can be found or described in numerous places) to minimize the difference between the observed price and the price given an implied volatility. With about one hundred lines in C++, I had something which felt fast enough. So today I hooked this into R via a two-line wrapper in quickly-created package using Rcpp. I had one more advantage here. For our time series problem, the majority of the parameters (strike, time to maturity, rate, ...) are fixed, so we can structure the problem to be vectorised right from the start. I cannot share the code or more the details of my new implementation. However, both GBSVolatility and EuropeanOprionImpliedVolatility are on CRAN (and as I happen to maintain these for Debian, also just one sudo apt-get install r-cran-foptions r-cran-rquantlib away if you're on Debian or Ubuntu). And writing the other solver is really not that involved. Anyway, here is the result, courtesy of a quick run via the rbenchmark package. We create a vector of length 500; the implied volatility computation will be performed at each point (and yes, our time series are much longer indeed). This is replicated 100 times (as is the default for rbenchmark) for each of the three approaches:

xyz@xxxxxxxx:~$ r xxxxR/packages/xxxxOptions/demo/timing.R
    test replications elapsed  relative user.self sys.self user.child sys.child
3 zzz(X)          100   0.038     1.000     0.040    0.000          0         0
2 RQL(X)          100   3.657    96.237     3.596    0.060          0         0
1 fOp(X)          100 448.060 11791.053   446.644    1.436          0         0
xyz@xxxxxxxx:~$ 

The new local solution is denoted by zzz(X). It is already orders of magnitude faster than the RQL(x) function using RQuantLib (which is, I presume, due to my custom solution internalising the loop). And the new approach is a laughable amount faster than the basic approach (shown as fOp) via fOptions. For one hundred replications of solving implied volatilities for all elements of a vector of size 500, the slow solution takes about 7.5 minutes --- while the fast solution takes 38 milliseconds. Which comes to a relative gain of over 11,000. So sitting down with your C++ compiler to craft a quick one-hundred lines, combining two well-known and tested methods, can reap sizeable benefits. And Rcpp makes it trivial to call this from R.

Review: Passion Play, by Sean Stewart

Publisher:	Ace
Copyright:	1992
Printing:	December 1993
ISBN:	0-441-65241-7
Format:	Mass market
Pages:	194

Diane is a shaper: an empath, a sort of telepath, who sees emotions and the actions of others in the form of patterns and stories that she is nearly compelled to follow. She uses that skill as an investigator, a semi-official adjunct to the police who can read crime scenes and motives and hunt down criminals with uncanny ability. That makes her almost socially acceptable, but a shaper is still not something it's safe to be, something best kept hidden. Diane's world is one of theocracy, of dominance by aggressive religion, and even shapers working with the police are not exactly okay. This is Sean Stewart's first novel. He's since gone on to write many other books, one of which I've read and quite enjoyed (Nobody's Son) and another (Galveston) that won several awards, although he's probably most famous for his work on the ilovebees ARG. Passion Play won a minor award itself: the Aurora for the best Canadian SF novel of the year. I found it surprising as a first novel, since it's sharp, focused, and very short. Usually first novels are stuffed to the gills with ideas and plot, as if material had built up for years and exploded in the first book. Passion Play doesn't have that problem; if anything, it errs on the side of telling too little. This is, at its heart, a mystery. A famous actor died in the middle of a play. It was possibly suicide, possibly an accident, and possibly murder; the events of the death are quite ambiguous. But the actor is very important in Redemptionist politics, which warrants an investigation, and then Diane's shaper instincts start finding a pattern and a story in the death. The rest of the cast and crew form the obvious set of suspects, and have the normal mystery novel variety of flaws, foibles, strong personalities, and possible motives. I'm not much of a mystery novel reader, and to be honest I had some trouble tracking all of the characters and remembering Diane's suspicions about each one. But the details of the mystery are less important to this book than Diane herself. Shapers are not exactly stable. The patterns they see are all-consuming, the experience of others' strong emotions acts like a sort of drug, and shapers can burn themselves out. They can fall into a spiral of seeking stronger and stronger emotions until they can't feel anything at all. That's the core conflict of the book: Diane is slowly losing herself. She has a tenuous existence at the outskirts of a society that has inculcated her with an inflexible set of religious beliefs and an uncompromising attitude towards law and justice, but it's not a stable existence, she knows it, and she doesn't know what to do about it. She knows that the pattern of this particular death is deep and powerful, but she can't stay away from it. This is an unusual, and sometimes disturbing, focus for a story. It's risky, unconventional, and doesn't quite work. I think some of the failure is because of first novel problems, but most of it is due to character problems. In the first novel problems category, Passion Play is a bit too choppy and a bit too disjointed. I think Stewart was trying hard to keep the story focused, fast-moving, and tight, but the introduction of the world background is not particularly smooth, and Diane's first-person account of her emotional state is a bit too labored. The plot wants to build in a smooth arc towards its climax, but the writing and scene-setting jerks and jumps instead of flowing. But the larger problem for me is that there aren't enough interesting characters in this book; specifically, there aren't enough to show all sides of Diane's character. Stewart tries, by introducing Jim early in the book to serve essentially as Diane's friend (and she's desperately in need of one), but he's too much of a non-entity. He serves some plot purposes, but he doesn't have a strong enough voice and character to hold his ground against Diane and balance her obsessive internal monologue. And, other than Jim, all the other characters in the book (with the possible exception of the murder victim) are playing bit roles. They're not bad characters, but they don't feel fully real to either Diane or to the reader. It's very easy to put them into boxes that they never break out of. This does support the feeling of narrative inevitability about some parts of the book, but I think it would have been stronger with more conflict, more open challenging of Diane's perspective. I would characterize Passion Play as interesting but flawed, and ultimately a minor work. I'm not sorry I read it, but neither would I have missed much if I'd gone through life without reading it. It has a few interesting ideas, and a bravely unconventional narrative resolution, but it felt choppy and off-balance. Unless you love Stewart's writing and want to read everything he wrote, or unless you (like me) has a quirk about reading all award winners, this is probably best skipped over in favor of Stewart's later work. Rating: 6 out of 10

I've had a few weeks with very little time spent on Debian. I did set aside some time to try to find and fix an RC bug, to help the release. Steven C commented to let me know about the UDD Developer's Dashboard which is a great way to find RC bugs relevant to a team. I sat down to try and fix #648272 in 'Abuse' but sadly I didn't manage, instead finding another possible RC bug #689457 on the way. I was curious as to how long the version 1 web services API for musicbrainz will be kept around, in case there was a strong likelyhood it would be turned off during the lifetime of wheezy. If that were the case, then it might have been worth requesting freeze exceptions to patch the last few WS1 clients to use WS2 instead. However, the musicbrainz developers have no plans to turn off WS1 as long as there are people still using it. This is a bit of a chicken and egg situation for wheezy, but probably not strong enough grounds to persue it any further. So last night was my first "Debian Day" in a while, and by chance I've had a number of enquiries for software and packages which I haven't touched in a while. For the prboom package, we rebuild the upstream tarball to remove some material we don't have the right to distribute. The code that does the repackaging lives in a script called debian/fix_upstream.sh. Prboom upstream release very infrequently, so the script hasn't been run in a while. Fabian Greffrath noticed some bashisms which meant it would not work reliably with a non-bash /bin/sh. He also noticed that we couldn't build the binary lump data/prboom.wad using only free software since some changes that were made to freedoom. One possible fix was to expand the deutex package to recognise freedoom.wad, which I've now applied. Fabian has fixed the bashism issue, but I suspect a more robust solution long-term would be to always rebuild data/prboom.wad during package build, so that code gets exercised more often. The prboom changes are in the Git repository but I haven't release them yet. I need to decide whether it's worth requesting a freeze exception for the orig tarball build issue. It's not an FTBFS but it's similar: you could not rebuild a binary distributed in the source tarball using software only in Debian, at least not without some workarounds. We've had another request to package Prboom+. I've never been particularly interested in doing this myself, but it would probably be very little work, you could more-or-less take the prboom packaging as it stands. One thing I thought would be interesting to try would be to set up a git repository for the packaging and import the upstream Subversion repository in its entirety via git svn, rather than just import the released tarballs. Sadly their repository layout is not quite ideal for this. game-data-packager continues to grow. Following hot on the heels of Hexen 2 support, Markus Koschany has written preliminary support for LGeneral, a game which has been in Debian, at least in part, on and off. Hopefully we'll have that merged soon, and finish of Hexen 2 support whilst we're at it. On that note, here's a lazyweb question: what's the most convenient way to determine the length (at microsecond resolution) of a PCM WAV file from the command-line? Finally, we've been building Debian-based virtual machines at work, which meant dusting off some old preseed files and tweaking them to have unattended installs. This took a lot longer than I thought. It's certainly a lot more painful than kickstart and Redhat-based systems, and it's very unlikely that a preseed file that worked for one version of Debian would fully work for another. I'm not certain but I think even different releases of d-i targetting the same release "break" assumptions you might make, particularly about which questions you need to answer via the kernel command line, rather than in the preseed file. That's one thing we really need to sort. It's crazy having to specify keyboard layouts in the command line for VMs which won't ever have a keyboard. The disparity between what the user is asked, what value is written to the debconf database at that point in the install process, and what ends up in the debconf database at the end of the process is also a real pain: answering a question about keyboard layout sets half a dozen different debconf values in a cascade that is probably sensitive to other things like priority or whether d-i is in expert mode, too. So working backwards from the debconf database of an installed machine is an exercise in frustration, and figuring out which questions are asked prior to the preseed file being read for any given d-i or Debian release is a slow trial and error process. I'm going to get in touch with the d-i folks and see what kind of continuous integration systems they have in place. It would be great to have commit-level notification that you've broken preseed builds. If not for the d-i folks, then certainly at work.

A blog post from Martin Bekkelund today tell the story of how Amazon erased the books from a customer's kindle, locked the account and refuse to tell the customer why. If a real book store did this to a customer, it would be called breaking into private property and theft. The story has spread around the net today. A bit more background information is available in Norwegian from digi.no. It is no surprise that digital restriction mechanisms (DRM) are used this way, as it has been warned about such abuse since DRM was introduced many years back. And Amazon proved in 2009 that it was willing to break into customers equipment and remove the books people had bought, when it removed the book 1984 by George Orwell from all the customers who had bought it. From the official comments, it even sounded like Amazon would never do that again. And here we are, three years later. And thought this action is against Norwegian regulations and law, it is according to the terms of use as written by Amazon, and it is hard to hold Amazon accountable to Norwegian laws. It is just yet another example of unacceptable terms of use on the web, and how they are used to remove customer rights. Luckily for electronic books, there are alternatives without unacceptable terms. For example Project Gutenberg (about 40,000 books), Project Runenberg (1,652 books) and The Internet Archive (3,641,797 books) have heaps of books without DRM, which can read by anyone and shared with anyone. Update 2012-10-23: This story broke in the morning on Monday. In the evening after the story had spread all across the Internet, Amazon restored the account of the user, as reported by digi.no and NRK. Apparently public pressure work. The story from Martin have seen several twitter messages per minute the last 24 hours, which is quite a lot, and is still drawing a lot of attention. But even when the account is restored, the fundamental problem still exist. I recommend reading two opinions from Simon Phipps and Glen Moody if you want to learn more about the fundamentals and more details about the original story.

$ gsettings set \
org.gnome.desktop.interface gtk-color-scheme '
os_chrome_fg_color:black;
os_chrome_bg_color:#FED;
os_chrome_selected_fg_color:black;
os_chrome_selected_bg_color:#f5a089;
selected_fg_color:white;
selected_bg_color:#c50039;
theme_selected_fg_color:black;
theme_selected_bg_color:#c50039;
tooltip_fg_color:black;
tooltip_bg_color:#FFC;
'

A few days ago, Samsung released the source code of Synaps, an implementation of the Amazon Web Service CloudWatch API for OpenStack. Being a developer on the Ceilometer project, I've been curious to look on this project and how it could overlap with Ceilometer or other projects like Heat. What is CloudWatch? CloudWatch is a monitoring system provided by Amazon on its Web Services platform to monitor services. This allows you get notifications and trigger an action on certain threshold. For example, this can be used to scale your architecture by monitoring the number of requests you get on it and its general load by starting new servers. Synaps Synaps is written in around 7k lines of Python (with 28 % of which are comments), reuses at least one common module of OpenStack (openstack.common.cfg) and copy some modules from Nova. One thing that strikes me, is that there seems to be only a few unit tests compared to most OpenStack projects. Also, many parts of the code and documentation contains text written in korean, which won't be very helpful for most people! :-) It uses some external technologies, like Storm, Cassandra to store its persistent data and Pandas to do data analysis. The API server provides an EC2 compatible API only: no OpenStack specific API. This is probably not a bad thing for now, since I am not aware of any work in this direction. The API access directly the Cassandra back-end for read operation, but relies on RPC to do writes. This way, a set of daemon handles the write using the Storm part of Synaps and do data aggregation. The authentication only supports LDAP, but it should still be possible to add a driver for Keystone. A Java and a Python SDK are provided to record metrics into Synaps, but there's not enough documentation for it to be useful. Overlap with Heat For now, there's not a lot of overlap with Heat, because Heat does not implement completely the CloudWatch API. Heat actually still misses a lot of the CloudWatch functions. But as soon as it will implement the CloudWatch API completely, the overlap will be complete with Synaps in this regard. One divergence point however, is that Heat uses RPC to access data from the storage back-end via its engine (the central daemon), whereas Synaps directly connects to Cassandra. Also, Heat relies on SQLAlchemy, like most OpenStack projects needing a database. Overlap with Ceilometer One of the goal of Ceilometer is to provide data probes and pollsters for all OpenStack components (Nova, Swift, Quantum ) whereas Synaps let the OpenStack users to put any kind of metric inside it, and therefore doesn't provide anything for now. But the storage of metrics is the main common point between Synaps and Ceilometer. Synaps chose only one technology, Cassandra, to store its metrics, whereas Ceilometer took care of building an abstraction layer for the storage engine. Ceilometer currently allows an operator to use SQL or MongoDB, but Cassandra could likely be added. Data metric consolidation is done by Synaps. This makes sense, since Synaps don't need to have the full data history to trigger alarms. On the opposite, Ceilometer needs to have a full history to allow things like billing, and don't do any aggregation on data. Also, in Synaps, the data analysis is done using Pandas. This means the data used are retrieved from the Cassandra back-end, and then transformed by Pandas inside Synaps in something else. It's likely that in such a case, Synaps should use CQL to achieve that. Ceilometer manipulates the data near their storage: it means that the computation are done by back-end to be efficient (SQL, mapreduce ). Conclusion Considering Samsung open-sourced Synaps late in the development process, I don't feel like they aimed to have it becoming a core component. This is always sad, because the effort put into this implementation are big and it would have probably little to add some abstraction layers to follow what other OpenStack projects do. But this takes time and energy, and it's understandable that Samsung didn't want to achieve this in a short time frame. There's a part of the code and architecture that overlaps with Ceilometer and Heat. Ceilometer is becoming a specialized point to store data metrics from any source: so it's sad, but understandable, that Synaps did not tried to reuse it. Fortunately, Heat is working with Ceilometer to achieve exactly that. This means OpenStack would have only one metrics storage point, used for billing, for monitoring and alarming. Therefore, I think Synaps is an implementation of CloudWatch that should be looked at as an inspiration for Heat and Ceilometer to build a better and more integrated solution!

Christopher complained that in android, you need a user account to install software. That is not true, there is F-droid which is a catalog of Free/Open Source Software, no user account needed. Lack of multitasking is yes annoying sometimes, yes, but that is really a poweruser problem. Most users don't load web pages in background while playing. Most users will simply not bother with web pages that take too long to load. Given the choice, most users will prefer a snappy UI over a UI that allows proper multitasking. N900 was never snappy, stutters were common, and it would often hang for long times if you had too many apps or browser windows open at the same time. Even for Android there are way more complaints about non-fluid UI than lack of proper multitasking. It is thus only logical for the Android developers to concentrate in developing a snappier UI over improving multitasking. The fact that Android has given 400+ million Linux computers out in the hands of consumers is one of the MOST AMAZING THINGS ever. But it seems others on FOSS community seem to consider Android a rather unfortunate event, because it's not "Real Linux" or "100% free software" or "because it's not bug free". Err, like Maemo or MeeGo ever is any of those... It is fair to complain about usability quirks Android as enduser. But if you believe Free Software, you should also see the opportunity the open source parts of Android provides to you fix it to your needs. After all, the point of Free Software is that you don't need to depend on the upstream to fix you everything? Christoph complains that the stock email app is lacking. Android Email app is Open Source (Maemo and N9 email apps are not). Android email has been forked as K-9 Mail. The app lifecycle is fixable allowing you to return from browser to the mail you were at - at least a plenty of other apps manage to do that. The question is, are we consumers or creators. If we are just sitting waiting to Google provide us a perfect shrinkwrapped Android, we could just as well use iPhone or Windows phone. We should instead see Android as an opportunity - when it's 90% open source, fix the remaining closed source bits like open source 3D drivers and open source replacements for proprietary interfaces.

For long-winded reasons I'd rather not explain nor defend, my partner uses Dropbox for a large set of important files (~70G), which are stored on a removable drive. She is also a Windows user. Recently, by introducing a few new drives, I inadvertently changed the drive letter that is assigned to her portable device, which stopped Dropbox from working. Solving this properly is rather finicky so I thought I'd write down what I did. The end goal is to try and ensure that the portable drive always gets the same drive letter and that Dropbox is configured to use that drive letter, but before I can get that far I need to get Dropbox syncing again. It used to be F:, and I opted for U: going forward. Luckily the extra letters I'd introduced were all partitions on a separate hard drive from the OS, so I powered the machine down, unplugged the extra hard drive and booted back up. This freed up the stolen drive letters, but the portable drive did not re-inherit them. Running regedit as an administrative user and renaming keys in HKEY_LOCAL_MACHINE\SYSTEM\MountedDevices moved the USB drive back to the correct letter, but I suspected the extra hard drive would take precedence when it was back, so this was only an interim solution. With the portable drive back on F:, the Dropbox client was happy once again. However I needed to reconfigure the Dropbox client to use a different letter. I wasn't happy with the idea of altering the Dropbox client's configuration database under its feet, so I had to do everything "by the book". Luckily, the client supports the notion of moving your Dropbox folder. Combined with the command-prompt subst command (run as the local user rather than as an admin this time), I was able to clone the F: drive to a virtual U: drive, and then ask Dropbox to move the folder.

subst U: F:\

This was pretty awkward. F:\Dropbox and U:\Dropbox are in fact the same folder, so I needed to ask Dropbox to move it from F: to something other than U:. I opted for U:\tmp. The move took a long time (~2 hours). The proper solution is to try and get a stable drive letter for her device. This can be achieved using a tool called 'USB Drive letter Manager', or USBDLM. USBDLM is free for educational use, and my partner is a teacher, which is lucky. Once that's all sorted, reboot, insert portable device, ensure it's on the right drive letter, and move the Dropbox folder back down to U:. This time the move was near instantaneous.

as zack has mentioned earlier today, the count of RC bugs is falling. & the release team is again proposing RC buggy packages for removal from testing.

these were my RC bug related activities during the last week:

• #623578 gadfly: "gadfly: build wrong version of the packaging during a binNMU"
  apply patch from Guillem Jover, upload to DELAYED/2
• #675971 mumble: "Cannot communicate with the vast majority of Mumble servers due to lack of required baseline codec"
  file pre-approval unblock request #691098
• #677943 munin: "/etc/apache2/conf.d/munin removed on upgrade"
  send a patch to the BTS
• #685632 liquidsoap: "liquidsoap: missing versioned depend on libcamomile-ocaml-data"
  make dependency versioned, upload to DELAYED/2
• #688891 psad: "psad: modifies conffiles (policy 10.7.3): /etc/psad/psad.conf"
  send a patch to the BTS
• #689537 pdnsd: "pdnsd: deletes conffiles on package removal (policy 10.7.3): /etc/NetworkManager/dispatcher.d/002_NetMan_pdnsd"
  include patch from Stuart Prescott, do a QA upload
• #689683 prelude-manager: "prelude-manager: modifies conffiles (policy 10.7.3): /etc/prelude-manager/prelude-manager.conf"
  fix conffile handling, upload to DELAYED/2
• #690368 pdnsd: "pdnsd: postinst overwrites admin changes to /etc/default/pdnsd"
  fix conffile handling, do a QA upload
• #691015 conserver-client: "conserver-client: fails to upgrade from testing: Can't locate Tie/Hash/NamedCapture.pm in @INC"
  propose a patch to fix the mistake in my NMU

That was not my target in today's marathon (my second one in the year). The target was breaking my 3h38'45" record, set on June 10th in Caen. And, well, I did it. By more than 5 minutes \o/. Of course, the Seine-Eure marathon is the perfect place for this:

• really flat race with no up climbs above 5 meters and an overall up climbing of about 30 meters
• high level marathon. Even though it only has 600 runners, it is the fastest in France, when considering the "median" time (the time of the runner arrivng exactly half of all runners) of about 3h35.
• it is located in Normandy so the likeliness of heat in October is just science-fiction..:-)

My target was running 5'06"/km, so 3h35'11" (but the target was running this as long as possible and see what happens after km 35). I did this nearly perfectly in the first half, reaching the half-marathon in 1h47'30". Double this and you get 3h35'..:-) "However", I slightly accelerated in the second half, first without noticing, then because....I just could do it..:-). So I managed to run the second half at an average 5'/km, with even the last 2 kilometers around 4'50" and an amazing sprint at the end. Usain bubulle. The outcome is 3h33'35". Doh. Never thought I could do this. It seems that my unusual preparation (remember the "let's run a marathon when coming back from work, at night", in September? or the 5 half-marathons in a row....culminating in a 3/4 marathon in the 6th week-end?)...was not so bad. I'm now very close to qualifying times for Boston, my dream marathon (much more than NYC) as they are 3h30' for my age category. Well, another option is to wait turning 55, where the qualifying time is then 3h40..:-)...but it woudl be quite good to complete one under 3h30. Now, it doesn't look like a dream. We'll see at the end of next year as I will only run one marathon, in autumn, more focusing Spring on ultra running (and, technically, a marathon, but that will be the Mont-Blanc marathon, just before Debconf13....and these 42.195km are quite different from those in Normandy!

According to the release manager of Ubuntu the 10.12 release features the most minimal improvements: ''Doing is a quantum leap from imagining. Anyone having a minimal quantum physics exposure should be aware that a quantum leap is a very tiny leap. So according to the paraphrased intro this release should not even contain 'stable updates?!

Puppet I've really had it with Puppet. I used to be able to put up with all its downsides

• Non-Unix approach to everything (own transport, self-made PKI, non-intuitive configuration language, a faint attempt at versioning (bitbucket), and much much more )
• Ruby
• Abysmal slowness
• Lack of basic functionality (e.g. replace a line of text)
• Host management and configuration programming intertwined, lack of a high-level approach to defining functionality
• Horrific error messages
• Catastrophic upgrade paths
• Did I mention Ruby and its speed?
• Lack of IPv6 support
• [I could keep going ]

but now that my fourth attempt to upgrade my complex configuration from version 0.25.5 to version 2.7 failed due to a myriad of completely incomprehensible errors ("err: Could not run Puppet configuration client: interning empty string") and many hours were lost in trying to hunt these down using binary searches, I am giving up. Bye bye Puppet.

An alternative But I need an alternative. I want a system that is capable of handling a large number of hosts, but not so complex that one wouldn't put it to use for half a dozen machines. The configuration management system I want looks about as follows: It

• makes use of existing infrastructure (e.g. SSH transport and public keys, Unix toolchain, Debian package management and debconf)
• interacts with the package management system (Debian only in my case)
• can provision files whose contents might depend on context, particular machine data and conditionals. There should be a unified templating approach for static and dynamic files, with the ability to override the source of data (e.g. a default template used unless a template exists for a class of machine, or a specific hostname)
• can edit files on the target machine in a flexible and robust manner
• can remove files
• can run commands when files change
• can reference data from other machines (e.g. obtain the certificate fingerprint of each hosts that define me as their SMTP smarthost)
• can control running services (i.e. enable init.d scripts, check that a process is running
• is written in a sensible language
• is modular and easily extensible, ideally using a well-known language (e.g. Python!)
• allows to specify infrastructure with tags ("all webservers", "all machines in Zurich", "machines that are in Munich and receive mail"), but with the ability to override every parameter for a specific host
• should just do configuration management, and not try to take away jobs from monitoring software
• logs changes per-machine and collects data about applied configurations in a central location
• is configured using flat files that are human-readable so that the configuration may be stored in Git (e.g. YAML, not XML)
• can be configured using scripts in a flexible way

Since for me, Ruby is a downside of Puppet, I won't look at Chef, but from this page, I gleaned a couple of links: Ansible, Quattor, Salt, and bcfg2 (which uses XML though). And of course, there remains the ephemeral cfengine.

cfengine I haven't used cfengine since 2002, but I am not convinced it's worth a new look because it seems to be an academic project with gigantic complexity and a whole vernacular to its own. There is no doubt that it is a powerful solution, and the most mature of all of them, but it's far away from the Unix-like simplicity that I've come to love in almost 20 years of Debian. Do correct me if I am wrong.

Ansible Ansible looks interesting. It seems rather bottom-up, first introducing a way to remotely execute commands on hosts, which you can then later extend/automate to manage the host configurations. It uses SSH for transport, and its reason-to-be made me want to look at it. My ventures into the Ansible domain are not over yet, but I've put them on hold. First of all, it's not yet packaged for Debian (Ubuntu-PPA packages work on Debian squeeze and wheezy). Second, I was put off a bit by its gratuitous use of the shell to run commands, as well as other design decisions. Check this out: there are modules for the remote execution of commands, namely "shell", "command", and "raw". The shell modules should be self-explanatory; the command module provides some idempotency, such as not running the command if a file exists (or not). To do this, it creates a Python script in /tmp on the target and then executes that like so:

$SHELL -c /tmp/ansible/ansible-1350291485.22-74945524909437/command

Correct me if I am wrong, but there is zero need for this shell indirection. My attempts at finding an answer on IRC were met by user "daniel_hozac" with a reason along the lines of "it's needed, believe me", and on the mailing list, I am told that only the shell can execute a script by parsing the interpreter line at the top of the module. Finally, the raw execution module also executes using the shell And there a few other design decisions that I can't quite explain, around the command-line switch --sudo see the aforementioned message In short: running a command like

ansible -v arnold.madduck.net -a "/usr/bin/apt-get update" --sudo

does not invoke apt-get with sudo, as one might like; it invokes the shell that runs the Python script that runs the command. Effectively therefore, you need to allow sudo shell execution, and for proper automation, this has to be possible without a password. And then you might just as well allow root logins again. The author seems to think that "core behaviour" is that sudo allows all execution and that limiting the commands to run is not a use-case that Ansible will support. Apparently, I was the first to ever suggest this. There are always ways around (e.g. skip --sudo and just use sudo as the command, simply ignore the useless shell invocation and trust that your machine can handle it, but when such design decisions remain incomprehensible and get defended by the project people, then I am hesitant to invest more time on principle.

Salt Finally, I've looked at Salt, which is what I've spent most time on so far. From the discussions I started on host targeting and data collection, it soon became apparent that Salt is very thin and flexible, and that the user community is accomodating. Unfortunately, Salt does not use SSH, but at least it reuses existing functionality (ZeroMQ). As opposed to the push/pull model, Salt "minions" interestingly maintain a persistent connection to the server (which is not yet very stable), and while non-root usage is still not unproblematic, at least there has already been work done in this direction. I think I will investigate Salt more as it does look like it can do what I want. The YAML-based syntax does seem a bit brittle, but it's the best I've found so far. NP: The Pineapple Thief: Someone Here is Missing

As you may know, when Phil Zimmermann published PGP, his goal was to counter the wide spying possibilities the development of electronic communications offered to the authorities, by promoting the general use of encryption so that 1. governments would not be able to restrict it afterwards and 2. it becomes possible to encrypt messages for privacy without drawing suspicion. Now, cryptography has made its way and is widely use, but mostly for commercial website and only sparsely for mail. Indeed, contrary to signing, encrypting a message requires that the recipient has a crypto key and is able and willing to use it to decrypt your message. To alleviate this problem, I thought that it would be useful to have a way, when sending a message, to tell your recipients that you would like that they encrypt their answers, because you are able and willing to get encrypted mail. I think the most relevant way to do that would be by adding a dedicated header field:

Encryption-Desired: PGP

In that header, one would be able to indicate what protocols he accepts. I suggest a comma-separated list, with the following possible values: PGP (or PGP/MIME and PGP/traditional for people that only accept one of these two PGP formats), S/MIME. For instance:

Encryption-Desired: PGP/MIME, S/MIME

It may be useful to be able to indicate that you either always wish your messages to be encrypted, or that it is only for this message's replies. If this is really relevant, it would take two distinct headers: Encryption-Desired and Reply-Encryption-Desired. Any thought about that proposal? Next thing to do, determine how to have that standardized at IANA and how to promote it.

I want all my computing environments "furnished" similarly and as close to my personal taste as possible - be it big and small machines, self-administered and user accounts of others. and not only for myself - similar is needed for the environments I help maintain for my friends Siri, Erik and Peter. The environments are seldom fully equal, so simply copying things around is rarely useful. Here's a checklist of things that I need to customize and syncronize for me and my friends to feel at $HOME :

• System-wide
  • Tools available on the system
  • Graphical login and Desktop environments
  • Adjust fonts, colors etc. of terminal emulator, file manager etc.
  • Register SSH in monkeysphere, MAC in DHCP and hostname in DNS
• Account
  • Syncronize browser bookmarks
  • Install/configure browser search plugins, config tweaks etc.
  • Setup email and messaging accounts, and syncronize archives
  • Syncronize development scripts, git repositories etc.
  • Register account in monkeysphere

Above is work-in-progress. I hope to later extend with more details - probably by linking to separate pages about each subtopic.

Yesterday I went to FES Iztacala, the faculty where I worked between 1999 and 2003. It's nice to go visit some good friends (even if to talk for work issues). It is somewhat far from my usual roaming area (~25Km straight to the North), so I cannot do it as often as I'd like. But anyway - I had to be at work early in the morning, but leaving from here a bit early for lunchtime, and leaving home at ~14:30, I managed to arrive to Iztacala in ~75 minutes. Sustained cycling for 20 Km/h, even counting stops at traffic lights on the way, yay! Anyway, had a productive and fun evening there, but around 18:00 I decided to head back before night got me Specially for the first part of the way, as I'm not familiar with Atzcapozalco. Alejandro suggested me to go by the recently (some months ago) opened cycle path that covers 4Km and almost exactly crosses the delegation (each of the 16 constitutive parts of Distrito Federal, where an important part of Mexico City is located). The cycle path is a good initiative... But I must say, I'm very very glad I took it still with good daylight. As well as the Recreative cyclepath that goes to the South, until the border of Morelos state, this one was built over abandoned rail tracks. Good use for a vacant and useless public space Rail tracks which lay unclaimed in the city are uncomfortable to walk, and useless for anything else, so they basically mean a useless 2m-wide strip of common grounds. So, I welcome any initiatives that make it into a useful space again! And two meters are just enough for a comfortable cycling path - Yes, which will surely be shared with pedestrians, and sometimes becomes uncomfortable. But lets try it! However... When rail tracks are decomissioned and cycle paths built over them... the metal should be dismounted! Not only because of economic concerns (good metal used for rail tracks is much more expensive and useful than asphalt), but because if it stays there, it just becomes a danger. Specially, as is the case, if the asphalt is just deep enough to sometimes cover the tracks And sometimes not. Had I known, I would have taken several photographs of important mistakes in the rail layout. I know I was very close to having an accident at least once (this means, I lost balance and miraclously managed to slow down from ~15Km/h by running with the bike between my legs!), and got in uncomfortable situations several more times. For a good portion of the track, there is a train track running at about of its width, so I had to constantly ring the bell or shout whenever I saw pedestrians As changing from side to side to route around them would put both them and me in danger. Towards the Southern part of the cycle path, as it is a much more active industrial area, there are many places where multiple tracks cross each other under the thin asphalt, sometimes completely unpaved. In one of those points I even decided to step down of the bike and make ~20m walking. This cycle path seems like it was done in a great hurry to present a successful project to the Politicians in Charge, without much thought on what it requires to be really a good project. It provides, yes, a very useful and good mobility solution for cyclists in the North-West. But it is too dangerous... And I am not sure whether I'd take it again. Probably not. So, all in all...

• Thanks to the Atzcapozalco authorities who thought of the cyclists and claimed back unused public space for us all to enjoy!
• The job done was too hasty. Valuable money (selling the tracks) was lost by burying it under asphalt. But it was not enough asphalt It needs attention before somebody gets hurt (or, before more people I'd be surprised if nobody had yet felt adrenaline over there)
• I hope the entrant city government takes the self-powered vehicle promotion more seriously than the outgoing one Which has done great advances (such as the Ecobici program), but is still far from reaching its promises... As always, I know.
• I managed to get back home alive and complete, yay!
• Of course, the cycle path has been mapped into OpenStreetMap. My SportsTracker tracks are also there, but they require Flash (ugh): the way to Iztacala and the way back.

Oh, and lastly: Some might be surprised I'm using bits of Twitterspeak here. But well, I now have presence a bot repeating my posts over there, so I'd better get Alejandro to read this using the proper channels ;-)