Search Results: "tar"

30 October 2013

Daniel Kahn Gillmor: getting to TLS (STARTTLS HOWTO)

Many protocols today allow you to upgrade to TLS from within a cleartext version of the protocol. This often falls under the rubric of "STARTTLS", though different protocols have different ways of doing it. I often forget the exact steps, and when i'm debugging a TLS connection (e.g. with tools like gnutls-cli) i need to poke a remote peer into being ready for a TLS handshake. So i'm noting the different mechanisms here. lines starting with C: are from the client, lines starting with S: are from the server. many of these are (roughly) built into openssl s_client, using the -starttls option. Sometimes this doesn't work because the handshake needs tuning for a given server; other times you want to do this with a different TLS library. To use the techniques below with gnutls-cli from the gnutls-bin package, just provide the --starttls argument (and the appropriate --port XXX argument), and then hit Ctrl+D when you think it's ok to start the TLS negotiation. SMTP The polite SMTP handshake (on port 25 or port 587) that negotiates a TLS upgrade looks like:
C: EHLO myhostname.example
S: [...]
S: 250-STARTTLS
S: [...]
S: 250 [somefeature]
C: STARTTLS
S: 220 2.0.0 Ready to start TLS
<Client can begin TLS handshake>
IMAP The polite IMAP handshake (on port 143) that negotiates a TLS upgrade looks like:
S: OK [CAPABILITY IMAP4rev1 [...] STARTTLS [...]] [...]
C: A STARTTLS
S: A OK Begin TLS negotiation now
<Client can begin TLS handshake>
POP The polite POP handshake (on port 110) that negotiates a TLS upgrade looks like:
S: +OK POP3 ready
C: STLS
S: +OK Begin TLS 
<Client can begin TLS handshake>
XMPP The polite XMPP handshake (on port 5222 for client-to-server, or port 5269 for server-to-server) that negiotiates a TLS upgrade looks something like (note that the domain requested needs to be the right one):
C: <?xml version="1.0"?><stream:stream to="example.net"
C:  xmlns="jabber:client" xmlns:stream="http://etherx.jabber.org/streams" version="1.0">
S: <?xml version='1.0'?>
S: <stream:stream
S:  xmlns:db='jabber:server:dialback'
S:  xmlns:stream='http://etherx.jabber.org/streams'
S:  version='1.0'
S:  from='example.net'
S:  id='d34edc7c-22bd-44b3-9dba-8162da5b5e72'
S:  xml:lang='en'
S:  xmlns='jabber:server'>
S: <stream:features>
S: <dialback xmlns='urn:xmpp:features:dialback'/>
S: <starttls xmlns='urn:ietf:params:xml:ns:xmpp-tls'/>
S: </stream:features>
C: <starttls xmlns="urn:ietf:params:xml:ns:xmpp-tls" id="1"/>
S: <proceed xmlns='urn:ietf:params:xml:ns:xmpp-tls'/>
<Client can begin TLS handshake>
NNTP RogerBW (in the comments below) points out that NNTP has TLS support:
C: CAPABILITIES
S: [...]
S: STARTTLS
S: [...]
S: .
C: STARTTLS
S: 382 Continue with TLS negotiation
PostgreSQL I got mail from James Cloos suggesting how to negotiate an upgrade to TLS over the PostgreSQL RDBMS. He points to the protocol docs, and in particular, to multiple protocol flow documents, and SSLRequest and StartupMessage chunks of the protocol spec (and clarification that data is sent in network byte order). It won't work in a text-mode communication, but it's worth noting here anyway: The client starts by sending these eight octets:
0x00 0x00 0x00 0x08 0x04 0xD2 0x16 0x2F
and the server replies with 'S' for secure or 'N' for not. If the reply is S, TLS negotiation follows. The message represents int32(8) specifying that there are 8 octets and int16(1234),int16(5678). All sent in network order. (The non-TLS case starts with a similar message with int16(3),int16(0) for protocol version 3.0. Starttls is essentially pg protocol version 1234.5678.) what else? I don't know (but would like to) how to do: If you know other mechanisms, or see bugs with the simple handshakes i've posted above, please let me know either by e-mail or on the comments here. Other interesting notes: RFC 2817, a not-widely-supported mechanism for upgrading to TLS in the middle of a normal HTTP session.Tags: gnutls, imap, pop, postgresql, smtp, starttls, tls, xmpp

Thomas Goirand: OpenRC running on Debian / kFreeBSD

I have always pretended that OpenRC would be very easy to port to Debian GNU/kFreeBSD. Well, now I stop pretending: http://youtu.be/zoNoi8BgQjs This was done within a few hours working with upstream. Now, next-up: Debian GNU/Hurd. Hoping that porters will volunteer to do the work.

Matthew Palmer: How to deal with the "package gcc: " problem in Puppet

In a comment to yesterday s post on why your Puppet module sucks, Warren asks what can be done about the problem of multiple modules needing to include the same package (gcc being the most common example I m yet to come across). As I stated previously, there is no sane, standardised way for multiple independent modules to cooperate to ensure that certain packages are installed for their cooperative use. If it wasn t already obvious, I consider this to be a significant failing in Puppet, and one which essentially renders any attempt at public, wide-scale reuse of Puppet modules impossible. Software packages are such a fundamental part of system management today that it is rare for a module not to want to interact with them in some way. Without strong leadership from Puppet Labs, or someone core to the project who s willing to be very noisy and obnoxious on the subject, the issue is fundamentally unsolveable, because it requires either deep (and non-backwards-compatible) changes to how Puppet works at a core level, or it needs cooperation from everyone who writes Puppet modules for public consumption (to use a single, coordinated approach to common packages). The approaches that I ve seen people advocate using, or that I ve considered myself, roughly fall into the following camps.

Put your package into a class in your module Absofrickinglutely useless, because the elementary package resources you end up creating will still conflict with other modules package resources. Anyone who suggests this can be permanently ignored, as they don t understand the problem being solved.

Use a globally common module of classes representing common packages This could work, if you could get everyone who writes modules to use it (see someone willing to be very noisy and obnoxious ). Practically speaking, anyone who suggests this doesn t understand human nature. I don t see this actually happening any time soon.

Use the defined() function everywhere By wrapping all your package resources in if !defined(Package["foo"]) blocks, you can at least stop your module from causing explosions. What it doesn t do is make sure that the various definitions of the package resource are congruent (imagine if one was package "gcc": ensure => absent ). In order to safely avoid explosion, everyone would have to follow this approach, which (in the worst case) reduces the problem to the globally common module of classes. However, it s the least-worst approach I can practically consider. At least your module won t be the cause of explosions, and realistically no module s going to ask for gcc to be removed (or, if they are, you ll quickly find them and kill them). I ve seen calls from Puppet core dev team members to deprecate and remove the defined() function entirely. Given the complete lack of alternatives, this is a disturbing illustration of just how out of touch they are from the unfortunate realities of practical Puppetry.

Use virtual resources This is yet another variant of the common class technique that would require everyone, everywhere, to dance to the same tune. It has all the same problems, and hence gets a big thumbs-down from me.

Use singleton_packages This module is a valiant attempt to work around the problem, but practically speaking is no better than using defined() because again, if a module that isn t using singleton_packages specifies package "gcc": ensure => absent , you re going to end up with assplosion. The mandatory dependency on hiera doesn t win me over, either (I m a hiera-skeptic, for reasons which I might go into some other time).

Use only modules from a single source This is the solution that I use myself, if that s any recommendation As a result of pretty much every publically-available Puppet module sucking for one reason or another, I do not, currently, have any externally-written Puppet modules in the trees that I use. Every module (158, at current count) has been written by someone in the sysadmin team at $DAYJOB, to our common standards. This means that I can coordinate use of package resources, using our module of common classes where appropriate, or refactor modules to separate concerns appropriately. If you re wondering why we don t have all of these 158 modules available publically, well, we have a few of them up, but yes, the vast majority of them aren t publically available. Some of them suck mightily, while many others are just too intertwined with other modules to be able to use on their own, and we don t want to release crap code there s far too much of that out there already.

Matthew Palmer: How to deal with the "package gcc: " problem in Puppet

In a comment to yesterday s post on why your Puppet module sucks, Warren asks what can be done about the problem of multiple modules needing to include the same package (gcc being the most common example I m yet to come across). As I stated previously, there is no sane, standardised way for multiple independent modules to cooperate to ensure that certain packages are installed for their cooperative use. If it wasn t already obvious, I consider this to be a significant failing in Puppet, and one which essentially renders any attempt at public, wide-scale reuse of Puppet modules impossible. Software packages are such a fundamental part of system management today that it is rare for a module not to want to interact with them in some way. Without strong leadership from Puppet Labs, or someone core to the project who s willing to be very noisy and obnoxious on the subject, the issue is fundamentally unsolveable, because it requires either deep (and non-backwards-compatible) changes to how Puppet works at a core level, or it needs cooperation from everyone who writes Puppet modules for public consumption (to use a single, coordinated approach to common packages). The approaches that I ve seen people advocate using, or that I ve considered myself, roughly fall into the following camps.

Put your package into a class in your module Absofrickinglutely useless, because the elementary package resources you end up creating will still conflict with other modules package resources. Anyone who suggests this can be permanently ignored, as they don t understand the problem being solved.

Use a globally common module of classes representing common packages This could work, if you could get everyone who writes modules to use it (see someone willing to be very noisy and obnoxious ). Practically speaking, anyone who suggests this doesn t understand human nature. I don t see this actually happening any time soon.

Use the defined() function everywhere By wrapping all your package resources in if !defined(Package["foo"]) blocks, you can at least stop your module from causing explosions. What it doesn t do is make sure that the various definitions of the package resource are congruent (imagine if one was package "gcc": ensure => absent ). In order to safely avoid explosion, everyone would have to follow this approach, which (in the worst case) reduces the problem to the globally common module of classes. However, it s the least-worst approach I can practically consider. At least your module won t be the cause of explosions, and realistically no module s going to ask for gcc to be removed (or, if they are, you ll quickly find them and kill them). I ve seen calls from Puppet core dev team members to deprecate and remove the defined() function entirely. Given the complete lack of alternatives, this is a disturbing illustration of just how out of touch they are from the unfortunate realities of practical Puppetry.

Use virtual resources This is yet another variant of the common class technique that would require everyone, everywhere, to dance to the same tune. It has all the same problems, and hence gets a big thumbs-down from me.

Use singleton_packages This module is a valiant attempt to work around the problem, but practically speaking is no better than using defined() because again, if a module that isn t using singleton_packages specifies package "gcc": ensure => absent , you re going to end up with assplosion. The mandatory dependency on hiera doesn t win me over, either (I m a hiera-skeptic, for reasons which I might go into some other time).

Use only modules from a single source This is the solution that I use myself, if that s any recommendation As a result of pretty much every publically-available Puppet module sucking for one reason or another, I do not, currently, have any externally-written Puppet modules in the trees that I use. Every module (158, at current count) has been written by someone in the sysadmin team at $DAYJOB, to our common standards. This means that I can coordinate use of package resources, using our module of common classes where appropriate, or refactor modules to separate concerns appropriately. If you re wondering why we don t have all of these 158 modules available publically, well, we have a few of them up, but yes, the vast majority of them aren t publically available. Some of them suck mightily, while many others are just too intertwined with other modules to be able to use on their own, and we don t want to release crap code there s far too much of that out there already.

29 October 2013

Soeren Sonnenburg: Shogun Toolbox Version 3.0 released!

Dear all, we are proud to announce the 3.0 release of the Shogun Machine-Learning Toolbox. This release features the incredible projects of our 8 hard-working Google Summer of Code students. In addition, you get other cool new features as well as lots of internal improvements, bugfixes, and documentation improvements. To speak in numbers, we got more than 2000 commits changing almost 400000 lines in more than 7000 files and increased the number of unit tests from 50 to 600. This is the largest release that Shogun ever had! Please visit http://shogun-toolbox.org/ to obtain Shogun. News Here is a brief description of what is new, starting with the GSoC projects, which deserve most fame: Screenshots Everyone likes screenshots. Well, we have got something better! All of the above projects (and more) are now documented in the form of IPython notebooks, combining machine learning fundamentals, code, and plots. Those are a great looking way that we chose to document our framework from now on. Have a look at them and feel free to submit your use case as a notebook! FGM.html GMM.html HashedDocDotFeatures.html LMNN.html SupportVectorMachines.html Tapkee.html bss_audio.html bss_image.html ecg_sep.html gaussian_processes.html logdet.html mmd_two_sample_testing.html The web-demo framework has been integrated into our website, go check them out. Other changes We finally moved to the Shogun build process to CMake. Through GSoC, added a general clone and equals methods to all Shogun objects, and added automagic unit-testing for serialisation and clone/equals for all classes. Other new features include multiclass LDA, and probability outputs for multiclass SVMs. For the full list, see the NEWS. Workshop Videos and slides In case you missed the first Shogun workshop that we organised in Berlin last July, all of the talks have been put online. Shogun in the Cloud As setting up the right environment for shogun and installing it was always one of the biggest problems for the users (hence the switching to CMake), we have created a sandbox where you can try out shogun on your own without installing shogun on your system! Basically it's a web-service which give you access to your own ipython notebook server with all the shogun notebooks. Of course you are more than welcome to create and share your own notebooks using this service! *NOTE*: This is a courtesy service created by Shogun Toolbox developers, hence if you like it please consider some form of donation to the project so that we can keep up this service running for you. Try shogun in the cloud. Thanks The release has been made possible by the hard work of all of our GSoC students, see list above. Thanks also to Thoralf Klein and Bj rn Esser for the load of great contributions. Last but not least, thanks to all the people who use Shogun and provide feedback. S ren Sonnenburg on behalf of the Shogun team (+ Viktor Gal, Sergey Lisitsyn, Heiko Strathmann and Fernando Iglesias)

28 October 2013

Keith Packard: Quaternions

Tracking Orientation with Quaternions I spent the flight back from china and the weekend adding orientation tracking to AltOS. I d done a bit of research over the last year or so working out the technique, but there s always a big step between reading about something and actually doing it. I know there are a pile of quaternion articles on the net, but I wanted to write down precisely what I did, mostly as a reminder to myself in the future when I need to go fix the code Quaternion Basics Quaternions were invented by Sir William Rowan Hamilton around 1843. It seems to have started off as a purely theoretical piece of math, extending complex numbers from two dimensions to four by introducing two more roots of -1 and defining them to follow:
i  = j  = k  = ijk = -1
Use these new roots to create numbers with four real components, three of which are multiplied by our three roots:
r + ix + jy + kz
With a bit of algebra, you can figure out how to add and multiply these composite values, using the above definition to reduce and combine terms so that you end up with a set which is closed under the usual operations. Then we add a few more definitions, like the conjugate:
q = (r + ix + jy + kz)
q* = (r - ix - jy - kz)
The norm:
  q   =  (qq*) =  (r  + x  + y  + z )
u is a unit quaternion if its norm is one:
  u   = 1
Quaternions and Rotation Ok, so we ve got a cute little 4-dimensional algebra. How does this help with our rotation problem? Let s figure out how to rotate a point in space by an arbitrary rotation, defined by an axis of rotation and an amount in radians. First, take a vector, v , and construct a quaternion, q as follows:
q = 0 + ivx + jvy + kvz
Now, take a unit quaternion u , which represents a vector in the above form along the axis of rotation, and a rotation amount, , and construct a quaternion r as follows:
r = cos  /2 + u sin  /2
With a pile of algebra, you can show that the rotation of q by r is:
q  = r q r*
In addition, if you have two rotations, s and r , then the composite rotation, t , a rotation by r followed by s can be computed with:
q  = s (r q r*) s*
    = (sr) q (r*s*)
    = (sr) q (sr)*
t   = s r
q  = t q t*
That s a whole lot simpler than carrying around a 3x3 matrix to do the rotation, which makes sense as a matrix representation of a rotation has a bunch of redundant information, and it avoids a pile of problems if you try to represent the motion as three separate axial rotations performed in sequence. Computing an initial rotation Ok, so the rocket is sitting on the pad, and it s tilted slightly. I need to compute the initial rotation quaternion based on the accelerometer readings which provide a vector, g pointing up. Essentially, I want to compute the rotation that would take g and make it point straight down. Construct a vector v , which does point straight up:
g = (0, ax, ay, az) / norm(0, ax, ay, az)
v = (0, 0, 0, 1)
G is normalized so that it is also a unit vector. The cross product between g and v will be a vector normal to both, which is the axis of rotation. As both g and v are unit vectors, the length of their cross product will be sin
a = g   v
  = u sin  
The angle between g and v is the dot product of the two vectors, divided by the length of both. As both g and v are unit vectors, the product of their lengths is one, so we have
cos   = g   v
For our quaternion, we need cos /2 and sin /2 which we can get from the half-angle formulae:
cos  /2 =  ((1 + cos  )/2)
sin  /2 =  ((1 - cos  )/2)
Now we construct our quaternion by factoring out sin from the a and:
q = cos  /2 + u sin   sin  /2 / sin  
Updating the rotation based on gyro readings The gyro sensor reports the rate of rotation along all three axes, to compute the change in rotation, we take the instantaneous sensor value and multiply it by the time since the last reading and divide by two (because we want half angles for our quaternions). With the three half angles, (x,y,z), we can compute a composite rotation quaternion:
   cos x cos y cos z + sin x sin y sin z +
i (sin x cos y cos z - cos x sin y sin z) +
j (cos x sin y cos z + sin x cos y sin z) +
k (cos x cos y sin z - sin x sin y cos z)
Now we combine this with the previous rotation to construct our current rotation. Doing this faster If we read our sensor fast enough that the angles were a small fraction of a radian, then we could take advantage of this approximation:
sin x   x
cos x   1
that simplifies the above computation considerably:
1 + xyz + i (x - yz) + j (y + xz) + k (z - xy)
And, as x, y, z 1, we can further simplify by dropping the quadratic and cubic elements as insignificant:
1 + ix + jy + kz
This works at our 100Hz sampling rate when the rotation rates are modest, but quick motions will introduce a bunch of error. Given that we ve got plenty of CPU for this task, there s no reason to use this simpler model. If we did crank up the sensor rate a bunch, we might reconsider. Computing the Current Orientation We have a rotation quaternion which maps the flight frame back to the ground frame. To compute the angle from vertical, we simply take a vector in flight frame along the path of flight (0, 0, 0, 1) and rotate that back to the ground frame:
g = r (0 0 0 1) r*
That will be a unit vector in ground frame pointing along the axis of the rocket. The arc-cosine of the Z element will be the angle from vertical. Results All of the above code is checked into the AltOS git repository I added a test mode to the firmware that just dumps out the current orientation over the USB link which lets you play with rotating the board to see how well the system tracks the current orientation. There s a bit of gyro drift, as you d expect, but overall, the system tracks the current orientation within less than a tenth of a degree per second. Even with all of this computation added, the whole flight software is consuming less than 7% of the STM32L CPU time.

Matthias Klumpp: Tanglu update & release plans

Long time with no article about Tanglu! This was mainly because we were busy with the project itself, improving various aspects of the distribution. So, here is a new summary on what has been done and what you can expect from the first release ;-) Tanglu QA We further improved the automatic archive QA. There is now qa.tanglu.org, which constantly monitors the number of uninstallable or unbuildable packages in the Tanglu suites. It also provides tanglu-platform-smallstatus information on the metapackage generator, which helps us in finding out which packages are available on the live-cds. Furthermore, information about the staging->devel migration process is provided, to answer the question why a package does not migrate (this still needs some improvements, but it is being worked on). We also use some code from Ubuntu to monitor package versions in Debian and upstream, which helps to see if others have released newer versions of software shipped with Tanglu. This already resulted in many improvements: The Tanglu Aequorea suite does not contain unbuildable packages (at least not due to build-dependency changed), and all live-cds are working well. We will soon migrate the archive to a new server, which frees some server capacities we can use for automated QA and things like automatic live-CD building. Live-CDs, Installer, Alpha-Releases We currently don t do Alpha-Releases of Tanglu, but we create Live-CD snapshots of Tanglu, which are available at releases.tanglu.org (or mirror1, mirror2). These snapshots still have issues and are just early previews. They also ship without and installer, we are still working on that part. Please note that CD more or less means DVD or USB-Stick right now (and this won t change the expected image size will be around 800MB). Release Planning I am happy to announce that we will do a release this year, most likely in December. But what can you expect from the release? KDE 4.11kde-konqui We will ship with KDE 4.11, which will be the only desktop we officially support so far. The reason is simply lack of manpower we could promise to support more, but that would just be not realistic for the small team. So we focus on KDE (Plasma Shells) right now, and try to make it awesome. Also, the team consists mostly of KDE people right now, which contributed to that decision ;-) . If you want to try Tanglu, right now the KDE live-images are the best to try it out. GNOME 3.8 gnome-logoWe will also provide images with GNOME. The problem with GNOME is, that the GNOME team does not have enough manpower to maintain the whole desktop or to upgrade it to the latest version (it is essentially just me looking at it from time to time). So GNOME will be available in a preview state. We invite everyone with GNOME knowledge to join the project and help improving Tanglu-GNOME GNOMErs, we want you! systemd >= 204 We ship with systemd by default, which works nicely so far, although more testing needs to be done. The logind service will be used to replace ConsoleKit, if we manage to get everything in place and working in time (if there are issues, we might switch back to CK for one release). There are some plans to use a higher systemd version, due to some improvements made there, but if this will be done is still unclear (Debian will most likely stick to 204 for some time, because with systemd > 205, running it as pid 1 will be mandatory to use logind (and Debian is just in the process to decide which init-system we will use there)). Systemd will run in SysVInit compatibility mode for most of the available services. This will improve in later Tanglu releases. Of course, systemd is usable, even if not every init-script has been converted to a service file. It just has an impact on startup times, so Tanglu will not be the distributions with the fastest startup times (yet ;-) ). Debian Package Mixdebian_logo1 Tanglu consists mostly of packages from Debian Testing (Jessie), but we take full advantage of the Debian archive, so you will also find package versions from Unstable or even Experimental (where it made sense). A very small portion of packages has also been merged with Ubuntu. Although stuff has been changed, the incompatibilities with Debian are almost zero, so if you are installing Tanglu, it will currently feel like an more up-to-date Debian Testing with some extras. Still, the differences are large enough that upgrading a Debian system to Tanglu might result in some issues. Installer Right now, the installer is a major field of work. Tanglu will most likely ship with the Debian-Installer, because it is the easiest thing to do right now. For later releases, it is planned to also offer the Ubiquity installer (the thing Ubuntu uses), or a new installer with a similar UI and concept. Other Cool Things266px-Wayland_Logo Tanglu will ship with a fully working Qt5 (which is currently being tested and updated) and the latest version of Wayland/Weston as a preview for developers to play around with. We also ship with Perl 5.18 and Haskell GHC 7.6, as well as with GCC 4.8 as default compiler (although the whole distribution does not yet compile with GCC 4.8). We might ship with Linux 3.12, but this also depends on other things. The Linux kernel build will be the same as in Debian. There might be more things to come :-) Please do always keep in mind that Tanglu is a new project, so the first release might still have some teething problems but we re working on it. The Aequorea release will be supported one to two more months after the next release is out. Tanglu Policy I started to draft a Tanglu policy, which defines stuff like the procedures to become a Tanglu project member and/or developer, some basic values of the distribution, decision making processes etc. This work is at a very early stage right now, and will need lots of feedback later. But as soon as it is done, joining the project will be easier and what Tanglu is will be well-defined. The policy will also include a Code-of-Conduct as an additional document from the start. We need help! ;-) First of all, thanks to everyone working on Tanglu! You are amazing! Also, many thanks to every Debian developer or contributor who helped a lot in setting the project up and contributed knowledge. And of course, thanks to everyone else who contributed by creating awesome artwork, helped with code, TangluTanglu Logo (big) archive mirrors, buildd-servers or by testing the distribution and providing feedback! (given the state the aequorea suite was in at the beginning, testing was a really frustrating activity but people still did it!) So, if you would like to help, just find us on #tanglu-devel on Freenode or join the tg-devel mailinglist. We also really need some people taking care of the Tanglu website and updating it with some recent information from time to time, so people can see what is going on. Before we have the Tanglu policy finished and the Tanglu members and developers directory in place (a software which allows us to track all registered developers and gives them an @tanglu.org mailalias), the start might be a bit confusing, but we do our best to make it easy for new people to join. The best way is asking people. Tanglu is still created by an incredibly small team, which has a large task to accomplish. Help is welcome :-)

Dirk Eddelbuettel: Kurt Elling at Dominican University

Some catchup blogging: Kurt Elling was back in town on October 19, and not just in Chicago but quite literally in our little 'burb as the headliner for this season's concerts at Domiminican University. They had done us the favour of inviting Dianne Reeves back in March 2009, but this was clearly going to be a real treat. And I managed to snatch four front row (!!) tickets as soon as I heard about it, and the show once again didn't disappoint. Elling was as usual accompanied by his long-time collaborator Laurence Hobgood on piano, as well as local heros Clark Sommers on bass and John McLean on guitar. This time, Quincy Davies was on drums. And this band is realiably excellent, as is Elling in any live setting. The set was dominated by pieces from his most recent record and augmented by a few other standout pieces. There is a pretty rich set of live music by Elling on YouTube, see for example On Broadway (which was part of the set in a wonderfully fast and funked-up variant), Golden Lady (from the previous album, and also played live this time) or for example this wonderful version of You Send Me (also part of the concert). I snatched a photo or two (while at least turning my flash off, unlike the weird gal to my left, but I digress) and posted one on Google+. Kurt Elling will be back in Chicago at the Green Mill in early January. Expect to me there, and I hope to see you too!

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

27 October 2013

Gregor Herrmann: RC bugs 2013/41-43

after a bit of a break I try to get back into regular RC bug fixing. here's what I've done recently:

Petter Reinholdtsen: Teaching vmdebootstrap to create Raspberry Pi SD card images

The vmdebootstrap program is a a very nice system to create virtual machine images. It create a image file, add a partition table, mount it and run debootstrap in the mounted directory to create a Debian system on a stick. Yesterday, I decided to try to teach it how to make images for Raspberry Pi, as part of a plan to simplify the build system for the FreedomBox project. The FreedomBox project already uses vmdebootstrap for the virtualbox images, but its current build system made multistrap based system for Dreamplug images, and it is lacking support for Raspberry Pi. Armed with the knowledge on how to build "foreign" (aka non-native architecture) chroots for Raspberry Pi, I dived into the vmdebootstrap code and adjusted it to be able to build armel images on my amd64 Debian laptop. I ended up giving vmdebootstrap five new options, allowing me to replicate the image creation process I use to make Debian Jessie based mesh node images for the Raspberry Pi. First, the --foreign /path/to/binfm_handler option tell vmdebootstrap to call debootstrap with --foreign and to copy the handler into the generated chroot before running the second stage. This allow vmdebootstrap to create armel images on an amd64 host. Next I added two new options --bootsize size and --boottype fstype to teach it to create a separate /boot/ partition with the given file system type, allowing me to create an image with a vfat partition for the /boot/ stuff. I also added a --variant variant option to allow me to create smaller images without the Debian base system packages installed. Finally, I added an option --no-extlinux to tell vmdebootstrap to not install extlinux as a boot loader. It is not needed on the Raspberry Pi and probably most other non-x86 architectures. The changes were accepted by the upstream author of vmdebootstrap yesterday and today, and is now available from the upstream project page. To use it to build a Raspberry Pi image using Debian Jessie, first create a small script (the customize script) to add the non-free binary blob needed to boot the Raspberry Pi and the APT source list:
#!/bin/sh
set -e # Exit on first error
rootdir="$1"
cd "$rootdir"
cat <<EOF > etc/apt/sources.list
deb http://http.debian.net/debian/ jessie main contrib non-free
EOF
# Install non-free binary blob needed to boot Raspberry Pi.  This
# install a kernel somewhere too.
wget https://raw.github.com/Hexxeh/rpi-update/master/rpi-update \
    -O $rootdir/usr/bin/rpi-update
chmod a+x $rootdir/usr/bin/rpi-update
mkdir -p $rootdir/lib/modules
touch $rootdir/boot/start.elf
chroot $rootdir rpi-update
Next, fetch the latest vmdebootstrap script and call it like this to build the image:
sudo ./vmdebootstrap \
    --variant minbase \
    --arch armel \
    --distribution jessie \
    --mirror http://http.debian.net/debian \
    --image test.img \
    --size 600M \
    --bootsize 64M \
    --boottype vfat \
    --log-level debug \
    --verbose \
    --no-kernel \
    --no-extlinux \
    --root-password raspberry \
    --hostname raspberrypi \
    --foreign /usr/bin/qemu-arm-static \
    --customize  pwd /customize \
    --package netbase \
    --package git-core \
    --package binutils \
    --package ca-certificates \
    --package wget \
    --package kmod
The list of packages being installed are the ones needed by rpi-update to make the image bootable on the Raspberry Pi, with the exception of netbase, which is needed by debootstrap to find /etc/hosts with the minbase variant. I really wish there was a way to set up an Raspberry Pi using only packages in the Debian archive, but that is not possible as far as I know, because it boots from the GPU using a non-free binary blob. The build host need debootstrap, kpartx and qemu-user-static and probably a few others installed. I have not checked the complete build dependency list. The resulting image will not use the hardware floating point unit on the Raspberry PI, because the armel architecture in Debian is not optimized for that use. So the images created will be a bit slower than Raspbian based images.

Charles Plessy: Debian-Installer in the Amazon cloud.

I prepared machine images of the Debian Installer in each region of the Amazon Elastic Computing Cloud, following the method that I described earlier, and that I updated and documented in Debian's wiki. Started with a preseed file, these images install Debian la carte on a storage volume that can then be registered as a new machine image. This is not yet automatic, but I am working on it. In the meantime, it is possible to install Debian interactively using the network console.

26 October 2013

Paul Tagliamonte: systemd vs upstart vs sysvinit

That thread


My feels:



So, I submitted the issue to tech-ctte.


Let s make this painless and let ctte deal with it :)

Russ Allbery: Review: Why We Fail

Review: Why We Fail, by Victor Lombardi
Publisher: Rosenfeld Media
Copyright: 2013
ISBN: 1-933820-17-9
Format: Kindle
Pages: 248
Subtitled Learning from Experience Design Failures, Why We Fail is a case study book. It defines experience design, discusses the usefulness of analyzing failure, and then walks through ten marketplace failures attributed to failures of user experience (as opposed to engineering or user interface) and tries to draw some conclusions about those failures and how they could have been prevented. Lombardi then presents a more general (if fairly simple) framework for iterative experience design, along with some discussion of how that approach could have helped in these cases. Hardware and software designs can work properly but still fail with users and customers because the overall experience isn't right. Sometimes this is because something else got there first with more market share and people got used to it, sometimes it's because a system is too complicated, sometimes it's too simple, sometimes it's just uninspiring. Most of the news media coverage and many of the blog posts on this topic look to inspirational examples of successes. Lombardi points out here that successes can be accidental and failures are often more informative, a concept already common in many more mature areas of engineering. His hope, with Why We Fail, is to push a broader industry practice of taking apart and analyzing the failures. User experience design is still in its infancy, since many of the complex interactions now possible in modern hardware and software have just recently crossed into levels of complexity that qualify as experiences rather than interactions, and could benefit from more rigor. The concept sounds great. The approach in Why We Fail is at least interesting; Lombardi tells good stories, and I think they do tell some useful lessons. Drawing general principles is harder, which is always the weakness of case-study books. Lombardi's own attempt doesn't go far beyond a familiar mantra of the scientific method, small and responsive teams, cross-functional teams with flexibility to address a whole range of technical and business issues, and honest and forthright analysis of and iteration after failure. All of this is doubtless good advice; none of it should surprise anyone with even a smattering of engineering background or familiarity with current software design philosophy. The heart of the book is in the stories, and I did like that Lombardi captured a wide range of different designs. Included are embedded systems (BMW's iDrive), traditional software packages, web services of various types, and even a protocol (OAuth). Some were vaguely familiar and some were not familiar at all, although I don't follow UI and UX discussions to any large extent. In each case, Lombardi gives a short but effective introduction to the problem space and the product, walks through the design choices, and then talks about the failure. Usefully, he doesn't stop with the failure but continues the story through the reaction of the company to the failure and any subsequent actions they took, which in some cases contain their own useful lessons. All of this, I should note, is a bit far afield of my own area of expertise (architectural building blocks and sub-surface components without as much direct interaction with the user), so I'm not the best person to judge the quality and originality of the analysis. But I liked Lombardi's approach and his analysis of multiple possible failure causes, and all of his conclusions seemed quite plausible and sensible given the case studies as described. I'm best-qualified to judge the chapter on OAuth, since I've worked extensively on web authentication systems, and his analysis of the challenges and user experiences in that case closely match my own. I didn't notice any significant errors, which is hopefully a good sign for the rest of the book. As one might expect from a short book about a complex field aimed mostly at starting a conversation, there isn't much here that's earth-shattering, nor are there simple extractable principles that will make your experience designs better. As always, the takeaway boils down to "invest time and resources into this and try to think about it systematically," which of course is true of every possible axis of improvement for any product. (The challenge is usually where to spend those limited resources.) I will also note in passing that Lombardi assumes an entirely market-driven metric for success, and all of the case studies are exclusively commercial not unexpected, since that's where most of the money and a lot of the resources are, but not that helpful for bespoke work where the nature of the problem is subtlely different. But, despite the expected shortcomings and the inherent weakness of the case study approach to analysis, which tries to pluralize anecdote to get data I enjoyed this book. I also got at least one useful trick out of it: the premortem, to look for potential failure points in the experience before rolling out, or even designing, the product. It's a fairly light-weight book, but I think it's moderately inspirational, and I wholeheartedly concur with Lombardi on the merits of failure analysis as an engineering approach. Rating: 7 out of 10

25 October 2013

David Welton: MicroConf Europe

I don't envy conference organizers these days - most of what's being said can be read the next day, for free, on line, at your own pace, from the comfort of your own home, and without spending a bundle of time and money to sleep in a far away hotel. Competing with that is not easy, but the guys at MicroConf managed to. I would sum up the weekend by saying that it was a "very high bandwidth experience". Every day, from breakfast until I turned in, I was chatting with people or listening to speakers during the conference itself. That's aproximately 16 hours of being "on", and by the time I got home to Padova, I was exhausted! But at the end of the day, I felt like it was worth it being there in person, because of all the interaction with other people. The speakers' talks all ended up on line, more or less, but all the chatting and discussion and getting to know everyone is the human element that is tough to replicate on line, and one of the most important reasons to attend a conference in person. Prague is also a beautiful city - I wish I had had more time there to check it out. Here are some highlights and notes, in no particular order: However, since it was a business conference, I also have to put on my cold, hard accountant hat. Will the conference pay for itself? Only time will tell. I learned a variety of interesting and useful things, many of which I think I can put into practice. The problem is finding the time between consulting work and family, but that was a bottleneck before, too - I had, and have, more things to do than time. Also, to be very direct about it, how much of what I learned could not have been learned by carefully reading accounts of the conference, slides, and other material published on the internet? A lot of it. I'm not sure I would have paid attention to all of it though, so the conference was definitely nice in that it exposed me to some talks and ideas that otherwise I might have brushed off before giving them a chance. In terms of dollars and cents, I won't be able to say for a while whether it was a sensible investment or not. Would I go again? I'd like to - it was a lot of fun and the people were great. Like I said, it's tough doing conferences because your competition is the internet!

Enrico Zini: A vision wanted

A vision wanted Today Richard Stallman mailed all Italian LUGs asking that tomorrow's LinuxDay be called "GNU/Linux Day" instead. I wonder how that is ever going to help a community so balkanised, that the only way Italian LUGs manage to do something together, is to say "let's not care what we all do, let's just do it on the same day and call it a national event". Of course a few LUGs still make a point of not doing anything on that day, because you know, Judean People's Front. Cawk. Today a friend asked me if I could help her support people in installing Citrix Whatsit to set up a video conference to network meetings that will take place in a month in different cities. Those meetings are something I look forwad to. It wasn't much of a problem to say "no, I can't do that"; it was a problem to be unable to come up with some viable, Free alternatives. I sometimes have to use Skype to talk with friends who also are Debian Developers, because I still haven't managed to make VoIP work unless I go through a commercial proxy. There was the happy news that our regional administration is switching from MS Word to OpenOffice. It soon became a flamewar, because some people loudly complained that they should have used LibreOffice instead. At DebConf, after spending an hour getting frustrated with the default formatting of bullet points in WhateverOffice Impress, I did my Debian Contributors talk using a text file in vim. And it was a success! Thanks Francois Marier for maintaining cowsay. I can't sync contact lists and appointments between my N900, which runs a Debian derivative, and my laptop, because I don't want to have a Google account, and nothing else would work out of the box. I don't even know how to keep a shared calendar with other DDs, without using a 3rd party cloud service that I don't want to trust with my life's personal schedule. I need to do a code review of every vim plugin I need to use, because you can only get them by cloning GitHub repositories over plain http, and they don't even provide signed tags. That code is run with my own privileges every time I start a text editor, which is, like, all the time. I'm frightened at the idea of how many people blissfully don't think about what that means. Vim users. Programmers. Cool chaps. Yet the most important thing in Debian today seems to be yet another crusade between upstart and systemd. But we haven't had a lengthy discussion on why, although the excellent OpenStreetMap exists and many of us contribute to it, it seems to still be more immediate to hit Google Maps to get a route computed. How can we change that? We haven't had a lengthy discussion on what can we offer to allow anyone to set up some social platform that won't get swamped with spam the first day and cracked open the second; that would allow people to share some photos with their friends only, and some with the rest of the world; that won't required a full-time paid person to maintain. That won't be obsolete and require a migration to a new platform in a year. That isn't Facebook or Google Plus. I stopped taking photos because it's too much work to show them to people. Other people use Instagram. Whatever the hipster trend is for photo sharing today, October 25, 2013, I'm pretty sure it's not a Free platform. But we can do something. We technology leaders. We are those who drive technological change! For example, today I invested two hours of hard effort trying to figure out why libbuffy's test suite fails on kfreebsd. All while wondering why I was doing that, since I know all buffy's users personally, and none of them uses kfreebsd. And I will take a day off work to study the library symbols file specification, so that next time I'll know right away if the new version of a C++ compiler decides that a template-generated symbol isn't worth adding to a library anymore. What is this effort really about? It sometimes feel like micromanaging to me. It's good to have excellent quality standards. But not without a vision. Not until "reliable network printing with all PDF viewers and print servers we ship" is among our release goals. Not until we commit to making sure that "sharing files between Debian users" will work out of the box, without the need of going through a 3rd party website, or email. I'm not interested in spending energy discussing init systems. I'm interested in spending energy sharing stories of what cool stuff we can do in Debian today, out of the box. And what cool stuff we'll be able to do tomorrow. Let's spend time on IRC, on mailing lists, and at the next Debian events, talking about why we are really into this. Talking about a vision! Note: Please don't spend time telling me how to fix the problems I mentioned above. I'm not interested in help fixing some problems for me today. I'm interested in asking help fixing problems for everybody, right in the next stable release. Remember, remember, the 5th of November, 2014.

Jorge Salamero Sanz: Zentyal is hiring (C/C++ developers for Samba4/OpenChange)

Zentyal is seeking to hire a results driven and autonomous software C/C++ developers, motivated to work in a start-up environment and to become part of an open source company. You will be actively involved into Zentyal products -OpenChange and Samba projects-, digging into the internals, answering escalated customers requests and participating in pushing the full solution to enterprise level. Also, you will take part in discussions and proof of concept developments related to scalability, high availability and best efficiency scenarios. Check Zentyal careers website for more information, or ping us in IRC!

Clint Adams: The Legend of Natalie Dormer

In the future there will be an episode of Elementary with Frankie Faison playing Judge O'Hare.

24 October 2013

Daniel Pocock: Final report on GSoC 2013 projects

Google Summer of Code finished recently. This is the first year that I have participated as a mentor for the Debian Project. Its a big responsibility to be part of the Debian team and to be one of the Debian team members representing Debian at the GSoC Mentor Summit. Birthdays all round This has been a particularly important year for the Debian Project, as the project celebrated our 20th birthday recently, on 16 August 2013, one of the final days of DebConf13 in Switzerland. GSoC celebrates its 10th year in 2014, with a generous 10% boost in the student stipend to mark the occasion. Google: don't be evil Google's "Don't be evil" approach to business has always been an interesting point for discussion. In my view, there are few cases of absolute good or absolute evil that we can universally classify and agree on. At a more pragmatic level, various people have commented that GSoC is a recruiting program for Google: some have even cited this as a reason not to participate. This is a more interesting point for discussion. The results of the program are not exclusive to the headhunters at Google. Anybody can browse through the Debian GSoC weekly student reports from each student to find out exactly what the students were up to and try to recruit them. Mentors and the projects they belong to have not been forced to sign any wide-ranging non-disclosure agreements or non-compete agreements with Google. In one recent example, the xWiki open source project even went as far as setting up a new office in Romania and employing some former GSoC students on a permanent basis.
Assorted images from Iasi, Romania, where xWiki set up an office employing former GSoC students How does Google benefit then? Well, there appear to be several possibilities:
  • Generating a huge amount of goodwill and recognition for the value that they place on the development of the next generation of software engineers
  • Stimulating the expansion of high quality free software projects, many of which they use directly or indirectly for their internal projects
  • Access to private evaluations prepared by the mentors. These evaluations don't take more than an hour for the mentors to complete but they do serve to give Google's headhunters a slight headstart over anybody else who is scouring the web for the names of graduates who completed GSoC
Overall, it appears to fit the definition of a "win-win" scenario: yes, Google gets stuff out of it, but the benefit is not exclusive to Google and projects like Debian are winners too. My initial approach I started by reviewing some of these materials: One particular concept that I took note of was the need to give students some opportunity to be innovative and original in their project proposals. In other words, it has been suggested that students should not be given a precise specification: rather, they should be given some very general concepts or goals and asked to suggest some funky, innovative new idea. The wisdom of this approach varies. If your free software project has some very specific gap that you need to fill, you may only be interested in taking a student who can fill that gap and you may not want to waste the time of other applicants. On the other hand, if you do precisely document this gap you want filled, you may get a bunch of very similar proposals from students and it may be more difficult to distinguish which of them is the most desirable candidate. Furthermore, if your expectations are too rigid, you may not benefit from a top-gun coming in and contributing some really exciting piece of work that you hadn't even thought of. In the end, I opted for providing more generic project briefs and looking at how students responded. The briefs that I published are available here. Coding tasks during the selection process There is no doubt about it, coding tasks are an important part of ensuring students are capable of development work before they are selected. They are also very important for us as mentors. It is not uncommon for some of us to be completely out of touch with the capabilities of other developers. The coding tasks completed by the students during the selection phase allow us to get a feel for their capabilities and set reasonable expectations for how much they can achieve during the course of the project. To simplify the management of coding tasks and engage the students in the Debian community at large, I decided to create tasks as wishlist items in the Debian bug tracking system, such as this dynalogin bug report. The prospective GSoC students were asked to put their name in the bug tracker to take ownership of the task they would complete. Student project proposals For the real-time communications project, I felt all the students slightly underestimated the complexity of this field and would have to be extremely lucky to complete everything they hoped to. After all, this is a field that the free software movement has been wrestling with for years without gaining the upper hand. For somebody with good knowledge of the projects in this space, it is possible to guide the students towards meaningful and achievable subprojects. Nonetheless, the proposals were useful for me in understanding which part of the topics were most attractive to the students. For the other project areas, the student proposals were slightly more specific. Fabian's appeared to be the most specific and well considered proposal for the one-time-password project, in fact, it appeared very much like a project that he could work through from start to finish. Debian teamwork Anybody looking at the GSoC web site will notice that there is a certain amount of overhead in joining the program, promotion, payment administration and optional participation in the GSoC Mentor Summit. As a large organisation, Debian established a dedicated SoC administration team who looked after the GSoC and Outreach Program for Women administrivia. As a developer and mentor, I found this arrangement was highly effective and allowed me to focus over 99% of my efforts on the front-line work, selecting students and supporting them through their projects. Project results One of the more controversial issues during the Debian 7 (wheezy) release cycle was the inclusion of the Mumble voice conferencing software. Although the software has some serious issues, it was eventually escalated to the technical committee and allowed to remain in Debian, partly due to lack of any alternative. One of the first results from Catalin's project was the creation of an alternative, the new reConServer project, based on the librecon conversation manager API from the reSIProcate SIP stack. reConServer is a significantly more generic solution than Mumble, as it allows any SIP peer to participate in an audio conference. It supports TURN for NAT traversal too. Catalin then went on to make the first working implementation of a scheme for passing context information from a web site to SIP WebRTC call as described in the IETF draft. All work has been integrated into release tarballs and packaging. Fabian worked through the latest standards for one-time passwords, extending the oath-toolkit and dynalogin projects to implement mutual two-factor authentication with just about any arbitrary crypto suite using one-time passwords. This work is yet to be merged into an official release of dynalogin as it requires co-ordination with the oath-toolkit release and package updates. However, there is a compelling demonstration in his video from DebConf13 and the source is available under Fabian's github account. In the post-Snowden world, the work from both of these student projects will be hugely valuable to people wanting to tighten up their computer security and communication security practices. The development process Deciding to mentor more than one project turned out to be a worthwhile decision. At first, this might seem like a big risk but it is manageable with support from co-mentors. The upside is that different students work at different speeds, they have different code styles, they even work at different times of the day or week and mentoring more than one student/project allows the mentor to gain a better appreciation of how the students differ than if mentoring only a single project. For Catalin's project, it became obvious at an early stage that he would benefit from having a dedicated server to run the VoIP applications on public IP addresses and low-latency networks while testing them. I spun up a virtual machine with IPv4 and IPv6 in my Xen Cloud Platform and this was extremely helpful for both of us. Throughout the project, I took a hands-off approach and often left the students to their own devices. I gave some suggestions to Catalin about the object-oriented design and this was enough for him to fill in the gaps and write all the code by himself. Fabian's project had fewer ambiguities and he was able to use existing code as a model more frequently than Catalin. It is possible that the students may have produced more code if I had thought through the designs myself and given them definite blue-prints. However, I feel that this would have made it harder for me to appreciate their own abilities and test the limits of their design skills. DebConf13 participation One of the highlights for the students was their participation in DebConf13. Both students presented a session at the conference showcasing some of their work. The DebConf video team produces high quality videos of all sessions at the conference and this in turn provides some great evidence of the students' capabilities and what GSoC adds to Debian and the wider free software universe. I would encourage other mentors to actively contemplate ways to involve their students in conferences during or shortly after the completion of their project work. Looking ahead Google has recently confirmed that Google Summer of Code will be repeated in 2014. It is the 10th anniversary of the program and to celebrate, students will get 10% extra pay. Thank you There are many people to thank for the success of these projects and the wider success of the Debian GSoC team:
  • The Debian GSoC and OPW organisation admin team
  • Google and the Google Open Source Programs Office
  • All the co-mentors who assisted on these projects with me: Simon Josefsson, Luke Faraone, Sylvain Berfini, Eloy Coto and Jes s P rez Rubio
  • The students themselves, C t lin Constantin U urelu and Fabian Gr nbichler
  • All the other very capable and motivated students who applied, submitted code samples but were not selected to participate: many of these students could have also completed a successful project and turning them away is one of the most difficult tasks for mentors in a volunteer organisation like Debian

Russ Allbery: Review: Applied Cryptography

Review: Applied Cryptography, by Bruce Schneier
Publisher: John Wiley & Sons
Copyright: 1996
ISBN: 0-471-11709-9
Format: Trade paperback
Pages: 621
This is the next entry in the series of Russ reading books that he bought years ago and never got around to reading. Thankfully, this time, the book has aged somewhat better. This review is for the second edition of Applied Cryptography, published in 1996. Given how important computer security has become, and how central cryptography is to computer security, one might think that the passage of 17 years would make a book effectively obsolete. This turns out not to be the case. Yes, Rijndael (the current AES standard and the most widely-used block cipher), Camellia (the up-and-comer in the block cipher world), and the SHA-2 hash postdate this book and aren't discussed. Yes, there have been some further developments in elliptic-curve public-key cryptography. And yes, much of the political information in this book, as well as the patent situation for public-key cryptosystems, is now mostly of historical interest. But a surprising amount of this book still applies directly. Partly that's because much of Applied Cryptography is focused on general principles, classes of algorithms, and cryptographic protocols, and those do not change quickly. A rather amazing number of techniques and protocols still in widespread use today originated in the 1970s. Block ciphers, stream ciphers, cipher modes, public-key systems, signature systems, hash functions, key exchange, secret splitting, key management, and other, similar topics have not changed substantially. And even in the area of specific algorithms, DES is still in use (unfortunately) and its analysis is valuable for understanding any block cipher. RC4, Triple DES, CAST, RSA, MD4, MD5, and SHA-1 are still part of the cryptography landscape, and Schneier was already warning against using MD5 in 1996. The source code included in this book has, thankfully, been made obsolete by widely-available, high-quality free implementations of all common ciphers, available for any modern operating system. But the math, the explanations, and much of the cryptanalysis is still quite relevant. While it contains a comprehensive introduction to cryptographic concepts, Applied Cryptography is structured more like a reference manual than a tutorial. The first two chapters provide some foundations. Then, a wide variety of protocols, from the common to the completely esoteric, are discussed across the subsequent four chapters. There are four chapters on the basics of ciphers, another chapter on the mathematics, and then the longest part of the book: twelve chapters that provide a tour of all major and several interesting minor protocols that existed in 1996. These are organized into block ciphers, stream ciphers (and pseudo-random number generators), hash functions, and public-key systems. An entire chapter is devoted to the history and analysis of DES, and I think that was my favorite chapter of the book. Not only is it worth understanding how DES works, but it also provides a comprehensive introduction to the complexities of block cipher design, the politics of interactions with the NSA, and a fascinating history of early computer cryptosystems. Finally, Applied Cryptography closes with chapters on real-world systems and on politics. The politics section is mostly a historical curiosity, as is much of the chapter on real-world systems. Schneier discusses some of the PKCS and X.509 protocols here but doesn't use SSL as one of his examples despite its specification predating this book (perhaps the most glaring omission of the book). But he does discuss Kerberos (which is of personal interest to me), and it was useful to see it analyzed in the broader context of this sort of book. Those who read Schneier's blog regularly, or who have read any of his other books, will know that he's concise, incisive, and very readable, even with difficult material. I think he does err on the side of compactness this is not a book that handholds the reader or explains something more than once but there's a lot of ground to cover, so I can't really blame him. Expect to have to read the more difficult parts several times, and (at least on a first read) expect to just skip past some of the most complex sections as probably not worth the effort. But I think Schneier does a great job structuring the book so that one comes away with a high-level impression and overview sufficient to make informed guesses about capabilities and relative difficulty even if one doesn't have the patience to work through the details. I suspect most readers will want to skim at least part of this book. Unless you're willing to do a lot of background reading to understand the math, or a lot of careful study to follow the detailed descriptions and flaws in often-obscure algorithms, the algorithm discussions start to blend together. I found myself skipping a lot of the math and focusing on the basic descriptions and the cryptanalysis, particularly for the obscure ciphers. Many of the algorithms are also highly opaque and of obscure benefit; I doubt I will ever care about blind signatures or oblivious transfer, and I struggled to wrap my mind around the details of bit commitment (although it's apparently important in zero knowledge proofs, which in turn are relevant to authentication). But this is not the sort of book where you have to read every word. It's very well-structured, provides clear resynchronization points for readers who have had enough of a particular topic, and marks the topics that are more obscure or of dubious general usefulness. The most useful parts of this book, in my opinion, are the basic conceptual introductions to each class of cryptographic algorithm or protocol, and those chapters are excellent. I've been working in applied computer security, at the application and deployment level, for years, and this is the first time that I've felt like I really understood the difference between a block cipher and a stream cipher, or the implications of block cipher mode selection. Applied Cryptography is also a fascinating exercise in adjusting one's intuition for where the complexity lies. It's a memorable experience to move from the chapters on block ciphers, full of S-boxes and complex mixing and magic numbers, into the far more straightforward mathematical analysis of stream ciphers. Or the mathematical simplicity of RSA (although quite a bit of complexity is lurking on the cryptanalysis side, much of which is only touched on here). Of more historical interest, but still quite impressive, is that Applied Cryptography doubles as a comprehensive literature review. Not only does it cover nearly every algorithm in use at the time, it discusses the cryptography literature on nearly every topic with brief summaries of results and lines of investigation. The references section is a stunning 66 pages of small print featuring 1,653 references, and those are mentioned and put in context throughout the book. I'm not personally interested in chasing those lines of research further, and of course 17 years means there are many new papers that are as important to read today, but it's obvious that this book would have been an amazing and invaluable map to the research territory when it was first published. There are now newer books on this topic that you should consider if you're purchasing a book today. Cryptography Engineering in particular is in my to-read pile, and I'm interested to see how much of this book it could replace, although I believe it's lighter on the mathematical analysis and details. But I think Applied Cryptography still has a place on the bookshelf of any computer security professional. Schneier is succinct, detailed, straightforward, and comprehensive, at least for 1996, and I know I'll be reaching for this book the next time I forget the details of CFB mode and then get hopelessly confused by the Wikipedia article on the topic. Rating: 8 out of 10

Dirk Eddelbuettel: Introducing the CRAN Repository Policy Watch

CRAN is the repository network for R. It is a resounding success with (as of right now) almost 5000 packages, and growth rate which has been estimated (cf John Fox's keynote at useR! a few years ago) to be near 40% per year. We as R community members owe a great deal of thanks to the CRAN maintainers, and the R Core team. The tight integration between the language and the contributed code repository is unique among programming languages, and one of the many reasons why CRAN has been such a success and driver of growth and adoption for R. And the amount of work the CRAN maintainers put into this is substantial, and we owe them. Yet there is some friction between the repo maintainers, and the community of developers. There have numerous discussions on the main developer list about various aspect of how CRAN maintains the impeccable quality of the code in the archive. One particular aspect which has bugging (at least me) is the lack of communication when policy changes are made. It would be trivial to posts a set of changes to the developer list, and I suggested as much. Only to be soundly ignored. But changes to text (or html) files can monitored rather easily, and when such changes occur an alert can be sent. So I cooked up rather simple system for this which I called the CRAN Policy Watch. In essence, a simple cronjob monitors changes, and records new versions in a Github repo (which you can follow or star). Alternatively, the cronjob now also tweets from the @CRANPolicyWatch account which I invite everyone to subscribe to as well. If someone knows of a simple tool to summarize diffs of html or text files in static html pages, I'd be interested to expand the service to some github.io pages. Alernatively I could also just commit to a single file too and let Github summarize the changes. This was an itch I needed to scratch, and I hope some other people will fine this useful too.

Next.

Previous.