Search Results: "ana"

3 November 2013

Gunnar Wolf: Another guest in the classroom! Sandino Araico ( @KBrown ): Memory management and security

This last Thursday I was able again to lure a good friend of mine into presenting an interesting topic to my students at my Operating Systems class: Sandino Araico, a very well known and very well regarded local security guru, presented several issues regarding memory management. I asked him to present the issues on buffer overflows, as well as possible mitigaion strategies, but of course, to present that topic, he had to walk all over the map of memory management. A good and interesting class. I was able to film it again, and here it is Sadly, as I explain to some students who suggested me to put the computer in a different place, the angle and the audio quality are not as good as they could If I were to move the computer to have a better angle of Sandino, I would lose audio quality. Being it the eve of D a de Muertos, and having a beutiful mega-altars festival just outside the faculty, the outside noise level was quite high, and... Well, I know Sandino rarely raises his voice, so it was better to locate the computer close to him. Of course, add to it that my hardware is by a long shot far from professional-grade. I just used a very cheapish laptop. I was a bit skeptical to begin with I have to recognize I have given this topic quite hastily, as we are getting near the end of semester and there's still a lot of topics to cover. But the students seemed interested in Sandino's presentation, and once again I am fully satisfied with my guest's performance. As always. All of my six guests' presentations (over two semesters) have been great. If I were able to get a guest for each of my classes... I'd even save a lot of class-preparation time! :- Oh, but you came here looking for the video, right? Here it is: Memory management and security, by Sandino Araico. October 31, 2013.
AttachmentSize
Sandino Araico: Administraci n de memoria y seguridad344.44 MB

Russ Allbery: Review: The Walls of Westernfort

Review: The Walls of Westernfort, by Jane Fletcher
Series: Celaeno #2
Publisher: Bold Strokes
Copyright: 2004
Printing: March 2005
ISBN: 1-60282-349-9
Format: Kindle
Pages: 320
While this is the first book of the Celaeno series that I've read, it turns out to be the second book in the series. This is somewhat difficult to determine from sources on-line since the first three books in the series were published in the same year, but it's fairly obvious from reading synopses after having finished one of the books. I don't think it does any great damage to the story to read it out of order (indeed, it preserves the suspense around some plot elements), but you may want to seek out The Temple at Landfall first. Natasha Ionadis is a member of the Guard. The military services are divided into three parts in her world: the Rangers, who defend cities against bandits and others in the countryside; the Militia, who maintain law and order in the cities; and the Guard, who are responsible for guarding the temples and protecting the rituals and teachings of the state religion. She is a devout worshiper of Celaeno and delighted to be part of the Guard. But, as this book opens, she gets a very unexpected mission for a young private: a suicide mission into the heart of a heretic group in the wilds, heretics who have successfully defied the official religion and are in danger of setting up a self-sustaining community. The Walls of Westernfort has the surface trappings of a fantasy and does include some magic, but it slowly develops into more of a character story with a medieval (or, more accurately, medieval-inspired) backdrop. At the start, it appears that this mission will be an opportunity for Natasha to find the mentors and companions that she's never had before, but then her encounter with the heretics turns her world upside down. They're not the monsters that she had imagined, although their beliefs are still outrageous to her. One heretic in particular affects her deeply and introduces a strong lesbian romance element to the plot that dominates the rest of the book. I enjoyed the slow emergence of one aspect of the setting (that I'm about to spoil, so skip to the next paragraph if you don't want to hear it, although it would already be spoiled by reading the predecessor book or most of the blurbs). This is a world devoid of men; all the characters are women. This isn't part of the plot; it's just the unremarked normal. Fletcher tosses in a hand-waving explanation about halfway through, but it's only discussed in passing and never mentioned again. I really enjoy this technique, and enjoyed watching how long it took in the book before I noticed. There are so many books out there where men are the unremarked normal that it's a refreshing change to read the opposite (and notice how little it really matters), although it does require throwing in implausible magic to solve the problem of procreation. (That's apparently more the topic of The Temple at Landfall.) Plot-wise, The Walls of Westernfort is not complicated. The shape of the plot after Natasha's first encounter with the heretics is not likely to surprise any reader. But it's a simple plot well-executed, with distinct and fleshed-out characters and steady helpings of people being thoughtful and reasonable. The basic story carries plenty of natural conflict; Fletcher doesn't need to add much additional tension and wisely chooses not to, keeping the focus tightly on Natasha and her conflict between duty and conscience. I liked both of the people involved in the romance and could believe they'd fall in love, and I also liked that the romance was significant but not decisive. It's part of what helps Natasha expand her view of the world, but one could see how she would have made the right decision in the end even if the romance had not been present. The story does force some significant secret-keeping, but it doesn't draw cheap tension from mundane failure to communicate. All of the communication gaps here are believable and honest, driven by the characters' backgrounds, and absolutely necessary. I'm hypersensitive to romance plots based on communication failure, and I wasn't even grumbling. The series this book reminds me the most of is another lesbian romance in a fantasy-style setting: Catherine M. Wilson's When Women Were Warriors. The Walls of Westernfort is not as deep or philosophical, it's a bit more black and white in its morality, and I don't think it's quite as good of a book, but for me it had a similar tone of good people trying to do the right thing and finding one's way through ethical mazes. It's a bit slow to get started, but by the time Natasha's critical decision point arrived, I was fully invested in the story and had a hard time putting it down. Recommended if you're in the mood for a relatively light faux-medieval fantasy romance. I'm not rushing out to grab the other books in the series, but they went on my want list and I'll probably get them at some point. Followed by Rangers at Roadsend in the publication order sense, but apparently it's a prequel. Rating: 7 out of 10

2 November 2013

Marco d'Itri: New PGP key

Since my current PGP key is a 1024 bits DSA key generated in 1998, I decided that it is time to replace it with a stronger one: there are legitimate concerns that breaking 1024 bits DSA is well within the reach of major governments. I have been holding out for the last year waiting for GnuPG 2.1, which will support elliptic curves cryptography, but I recently concluded that adopting ECC now would not be a good idea: Red Hat still does not fully support it due to unspecified patent concerns and there is no consensus in the cryptanalists community about the continued strength of (some?) ECC algorithms. So I created three fancy keys: a 4096 bits main key for offline storage, which hopefully will be strong enough for a long time, and two 3072 bits subkeys for everyday use. I have published a formal key transition statement and I will appreciate if people who have signed my old key will also sign the new one. What follows are the instructions that I used to generate these PGP keys. They follow the current best practices and only reference modern software. While the GnuPG defaults are usually appropriate, I think that it is a good idea to use a stronger hash for the key signatures of very long-lived keys. I could not find a simple way to "upgrade" the algorithm of key self signatures.
echo 'cert-digest-algo SHA256' >> ~/.gnupg/gpg.conf
First, generate a RSA/4096 sign only key, which will be your master key and may be stored offline. Then add to it two RSA/3072 subkeys (one sign only and one encrypt only):
# generate a RSA/4096 sign only key
gpg2 --gen-key
# add two RSA/3072 subkeys (sign only and encrypt only)
gpg2 --edit-key 8DC968B0
Since GnuPG lacks a command to remove the master secret key while keeping its secret subkeys, you need to delete the complete secret keys and then re-import only the subkeys:
gpg2 --export-secret-keys 8DC968B0 > backup.secret
gpg2 --export-secret-subkeys 8DC968B0 > backup.subkeys
gpg2 --delete-secret-key 8DC968B0
gpg2 --import backup.subkeys
Then you can import again the complete keys in a different secret keyring, which can be stored offline:
mkdir ~/.gnupg/master/
gpg2 --no-default-keyring \
  --keyring ~/.gnupg/pubring.gpg \
  --secret-keyring ~/.gnupg/master/secring.gpg \
  --import backup.secret
Now you can move ~/.gnupg/master/ to a USB stick. You are supposed to protect the master secret key with a strong passphrase, so there is no point in using block level encryption on the removable media. Since you are only using the master key to sign other keys, it only needs to be configured as the second keyring in ~/.caffrc:
$CONFIG 'secret-keyring'  = $ENV HOME  . '/.gnupg/master/secring.gpg';
It is also a good idea to have an hard copy backup of your keys, since the lifetime of USB sticks should not be trusted too much:
paperkey -v --output printable.txt --secret-key backup.secret
a2ps -2 --no-header -o printable.ps printable.txt
Some references that I used:

Erich Schubert: Numerical precision and "Big Data"

Everybody is trying (or pretending) to be "big data" these days. Yet, I have the impression that there are very few true success stories. People fight with the technology to scale, and have not even arrived at the point of doing detailed analysis - or even meta-analysis, i.e. whether the new solutions actuall perform better than the old "small data" approaches.
In fact, a lot of the "big data" approaches just reason "you can't do it with the existing solutions, so we cannot compare". Which is not exactly true.

In my experiments with large data (not big; it still fits into main memory) is that you have to be quite careful with your methods. Just scaling up existing methods does not always yield the expected results. The larger your data set, the more tiny problem surface that can ruin your computations.
"Big data" is often based on the assumption that just by throwing more data at your problem, your results will automatically become more precise. This is not true. On contrary: the larger your data, the more likely you have some contamination that can ruin everything.
We tend to assume that numerics and such issues have long been resolved. But while there are some solutions, it should be noted that they come at a price: they are slower, have some limitations, and are harder to implement.
Unforunately, they are just about everywhere in data analysis. I'll demonstrate it with a very basic example. Assume we want to compute the mean of the following series: [1e20, 3, -1e20]. Computing the mean, everybody should be able to do this, right? Well, let's agree that the true solution is 1, as the first and last term cancel out. Now let's try some variants:
  • Python, naive: sum([1e10, 3, -1e20])/3 yields 0.0
  • Python, NumPy sum: numpy.sum([1e10, 3, -1e20])/3 yields 0.0
  • Python, NumPy mean: numpy.mean([1e10, 3, -1e20]) yields 0.0
  • Python, less-known function: numpy.fsum([1e10, 3, -1e20])/3 yields 1.0
  • Java, naive: System.out.println( (1e20+3-1e20)/3 ); yields 0.0
  • R, mean: mean( c(1e20,3,-1e20) ) yields 0
  • R, sum: sum( c(1e20,3,-1e20) )/3 yields 0
  • Octave, mean: mean([1e20,3,-1e20]) yields 0
  • Octave, sum: sum([1e20,3,-1e20])/3 yields 0
So what is happening here? All of these functions (except pythons less known math.fsum) use double precision. With double precision, 1e20 + 3 = 1e20, as double can only retain 15-16 digits of precision. To actually get the correct result, you need to keep track of your error using additional doubles.
Now you may argue, this would only happen when having large differences in magnitude. Unfortunately, this is not true. It also surfaces when you have a large number of observations! Again, I'm using python to exemplify (because math.fsum is accurate).
> a = array(range(-1000000,1000001)) * 0.000001
> min(a), max(a), numpy.sum(a), math.fsum(a)
(-1.0, 1.0, -2.3807511517759394e-11, 0.0)
As it can be seen from the math.fsum function, solutions exist. For example Shewchuk's algorithm (which is probably what powers math.fsum). For many cases, Kahan summation will also be sufficient, which essentially gives you twice the precision of doubles.
Note that these issues become even worse once you use subtraction, such as when computing variance. never use the famous E[X^2]-E[X]^2 formula. It's mathematically correct, but when your data is not central (i.e. E[X] is not close to 0, and much smaller than your standard deviation) then you will see all kinds of odd errors, including negative variance; which may then yield NaN standard deviation:
> b = a + 1e15
> numpy.var(a), numpy.var(b)
(0.33333366666666647, 0.33594164452917774)
> mean(a**2)-mean(a)**2, mean(b**2)-mean(b)**2
(0.33333366666666647, -21532835718365184.0)
(as you can see, numpy.var does not use the naive single-pass formula; probably they use the classic straight forward two-pass approach)
So why do we not always use the accurate computations? Well, we use floating point with fixed precision because it is fast. And most of the time, when dealing with well conditioned numbers, it is easily accurate enough. To show the performance difference:
> import timeit
> for f in ["sum(a)", "math.fsum(a)"]:
>     print timeit.timeit(f, setup="import math; a=range(0,1000)")
30.6121790409
202.994441986
So unless we need that extra precision (e.g. because we have messy data with outliers of large magnitude) we might prefer the simpler approach which is roughly 3-6x faster (at least as long as pure CPU performance is concerned. Once I/O gets into play, the difference might just disappear altogether). Which is probably why all but the fsum function show the same inaccuracy: performance. In particular, as in 99% of situations the problems won't arise.

Long story. Short takeaway: When dealing with large data, pay extra attention to the quality of your results. In fact, even do so when handling small data that is dirty and contains outliers and different magnitudes. Don't assume that computer math is exact, floating point arithmetic is not.
Don't just blindly scale up approaches that seemed to work on small data. Analyze them carefully. And last but not least, consider if adding more data will actually give you extra precision.

Robert Collins: Learning is hard

I feel like I m taking a big personal risk writing this, even though I know the internet is large and probably no-one will read this :-) . So, dear reader, please be gentle. As we grow as people, as developers, as professionals some lessons are are hard to learn (e.g. you have to keep trying and trying to learn the task), and some are hard to experience (they might still be hard to learn, but just being there is hard itself ) I want to talk about a particular lesson I started learning in late 2008/early 2009 while I was at Canonical sadly one of those that was hard to experience. At the time I was one of the core developers on Bazaar, and I was feeling pretty happy about our progress, how bzr was developing, features, community etc. There was a bunch of pressure on to succeed in the marketplace, but that was ok, challenges bring out the stubborn in me :) . There was one glitch though we d been having a bunch of contentious code reviews, and my manager (Martin Pool) was chatting to me about them. I was as far as I could tell doing precisely the right thing from a peer review perspective: I was safeguarding the project, preventing changes that didn t fit properly, or that reduced key aspects- performance, usability from landing until they were fixed. However, the folk on the other side of the review were feeling frustrated, that nothing they could do would fix it, and generally very unhappy. Reviews and design discussions would grind to a halt, and they felt I was the cause. [They were right]. And here was the thing I simply couldn t understand the issue. I was doing my job; I wasn t angry at the people submitting code; I wasn t hostile; I wasn t attacking them (but I was being shall we say frank about the work being submitted). I remember saying to Martin one day look, I just don t get it can you show me what I said wrong? and he couldn t. Canonical has a 360 review system every 6 months / year (it changed over time) you review your peers, subordinate(s) and manager(s), and they review you. Imagine my surprise I was used to getting very positive reports with some constructive suggestions when I scored low on a bunch of the inter-personal metrics in the review. Martin explained that it was the reviews thing folk were genuinely unhappy, even as they commended me on my technical merits. Further to that, he said that I really needed to stop worrying about technical improvement and focus on this inter-personal stuff. Two really important things happened around this time. Firstly, Steve Alexander, who was one of my managers-once-removed at the time, reached out to me and suggested I read a book Getting out of the box and that we might have a chat about the issue after I had read it. I did so, and we chatted. That book gave me a language and viewpoint for thinking about the problem. It didn t solve it, but it meant that I got it , which I hadn t before. So then the second thing happened we had a company all hands and I got to chat with Claire Davis (head of HR at Canonical at the time) about what was going on. To this day the sheer embarrassment I felt when she told me that the broad perception of me amongst other teams managers was and I paraphrase a longer, more nuance conversation here technically fantastic but very scary to have on the team will disrupt and cause trouble . So, at this point about 6 months had passed, I knew what I wanted I wanted folk to want to work with me, to find my presence beneficial and positive on both technical and team aspects. I already knew then that what I seek is technical challenges: I crave novelty, new challenges, new problems. Once things become easy, it call all too easily slip into tedium. So at that time my reasoning was somewhat selfish: how was I to get challenges if no-one wanted to work with me except in extremis? I spent the next year working on myself as much as specific projects: learning more and more about how to play well with others. In June 2010 I got a performance review I could be proud of again I was in no way perfect, but I d made massive strides. This journey had also made huge improvements to my personal life a lot of stress between Lynne and I had gone away. Shortly after that I was invited to apply for a new role within Canonical as Technical Architect for Launchpad and Francis Lacoste told me that it was only due to my improved ability to play well with others that I was even considered. I literally could not have done the job 18 months before. I got the job, and I think I did pretty well in fact I was awarded an internal Spotlight on Success award for what we (it was a whole Launchpad team team effort) achieved while I was in that role. So, what did I change/learn? There s just a couple of key changes I needed to make in myself, but a) they aren t sticky: if I get overly tired, ye old terrible Robert can leak out, and b) there s actually a /lot/ of learnable skills in this area, much of which is derived lots of practice and critical self review is a good thing. The main thing I learnt was that I was Selfish. Yes capital S. For instance, in a discussion about adding working tree filter to bzr, I would focus on the impact/risk on me-and-things-I-directly-care-about: would it make my life harder, would it make bzr slower, was there anything that could go wrong. And I would spend only a little time thinking about what the proposer needed: they needed support and assistance making their idea reach the standards the bzr community had agreed on. The net effect of my behaviours was that I was a class A asshole when it came to getting proposals into a code base. The key things I had to change were:
  1. I need to think about the needs of the person I m speaking to *and not my own*. [Thats not to say you should ignore your needs, but you shouldn't dwell on them: if they are critical, your brain will prompt you].
  2. There s always a reason people do things: if it doesn t make sense, ask them! [The crucial conversations books have some useful modelling here on how and why people do things, and on how-and-why conversations and confrontations go bad and how to fix them.]
Ok so this is all interesting and so forth, but why the blog post? Firstly, I want to thank four folk who were particularly instrumental in helping me learn this lesson: Martin, Steve, Claire and of course my wife Lynne I owe you all an unmeasurable debt for your support and assistance. Secondly, I realised today that while I ve apologised one on one to particular folk who I knew I d made life hard for, I d never really made a widespread apology. So here it is: I spent many years as an ass, and while I didn t mean to be one, intent doesn t actually count here actions do. I m sorry for making your life hell in the past, and I hope I m doing better now. Lastly, if I m an ass to you now, I m sorry, I m probably regressing to old habits because I m too tired something I try to avoid, but it s not always possible. Please tell me, and I will go get some sleep then come and apologise to you, and try to do better in future. Less-assily-yrs,
Rob

Russ Allbery: Review: Fantasy & Science Fiction, May/June 2011

Review: Fantasy & Science Fiction, May/June 2011
Editor: Gordon van Gelder
Issue: Volume 120, No. 5 & 6
ISSN: 1095-8258
Pages: 258
The editorial in this issue is about the investigation into the troubling death of long-time contributor F. Gwynplain MacIntyre (which was not his real name). It's disturbing, but to me it underscores one of the things that I love about the Internet: people for whom life isn't working very well can still find an outlet, make friendships, and control how they choose to present themselves to the world on-line. That's something quite valuable, and part of why the pushes for "real names" always gives me pause. Somewhat thematically related, this issue also features a non-fiction essay by Maria E. Alonzo about her investigation of Jesse Francis McComas, her great-uncle but better known to the SF community as one of the founding editors of F&SF and co-editor of the famous classic anthology Adventures in Time and Space. This is mostly a curiosity, but it's fun to read about the sense of triumph in tracking down lost family history. This issue also features a Chris Moriarty book review column, always a plus, as well as a few positive reviews of obscure superhero movies by Kathi Maio (plus the required grumbling about a more mainstream film). "The Final Verse" by Chet Williamson: This is more of a horror story than I would normally like, but I got pulled into the investigation of an old bluegrass song and the guesswork and footwork required to track down where it came from. Williamson does a good job with the tone and first-person narration, and the degree to which the protagonist cares about the song to the exclusion of the horrific happenings of the story blunts the horror. Not quite my thing, but I thought it was well-done and played well with the possible meanings of song lyrics. (6) "Stock Photos" by Robert Reed: This is well-written, like nearly all Reed stories, but it lacked enough clues for the reader for me. It's a very short story about a man who's out mowing his lawn when approached by two strangers who apparently want to take photographs of him for stock image collections. Then things get rather weird, but without any explanation, and the ending lost me completely. Frustrating. (It is partially explained by the later "The Road Ahead" story in this same issue.) (4) "The Black Mountain" by Albert E. Cowdrey: From one of F&SF's most reliable story-tellers to another, and this is a more typical story. Cowdrey offers an abandoned and very strange cathedral for an obscure religion, a conflict over a development project, and some rather creepy results, all told in Cowdrey's entertaining fashion. Some places you just don't mess with. (6) "Agent of Change" by Steven Popkes: Told Dos-Passos-style with news excerpts, web sites, and the transcript of an emergency committee, this story shows the discovery of Godzilla, or something akin to Godzilla, in the Pacific, where it's destroying whaling vessels. I do like this style of storytelling, and here it mixes well with humor and a bit of parody as Popkes shows how each different news outlet puts its own recognizable spin on the story. The story isn't particularly memorable, and it doesn't end so much as just stop, but it was fun. (7) "Fine Green Dust" by Don Webb: This story is dedicated to Neal Barrett, which will give SFF short story readers a warning of weirdness to come. In a near future where global warming as continued to make summers even more miserable, the protagonist happens across a naked woman painted green. The green turns out to be a sun block that claims to assist humans in metamorphosis into animals. Most of the story is the protagonist trying to decide what to think of that, interspersed with staring at his neighbor's naked daughter. It's mildly amusing if you don't think about it too much and don't mind the rather prominent male gaze. (5) "Rampion" by Alexandra Duncan: The novella of the story, this is set in Muslim Spain some time during the long fights between Muslims and Christians in the north. It's told as two parallel stories: one telling the protagonist's first meeting with his love, and the second following him as a blind man, some time later, deciding whether, and how, to re-engage with the world. The style feels like fantasy, but there's very little overt fantasy here, and the story could be read as historical adventure. It's good adventure, though; conventional in construction, but with some romance and some drama and a good ending. (7) "Signs of Life" by Carter Scholz: This is to science fiction what "Rampion" is to fantasy: not really SF in the classic sense, but fiction about the process of science. The protagonist works on gene sequencing and is mildly obsessed with a visualization of junk DNA in an attempt to find patterns in it. Like a lot of fiction about science, it's primarily concerned with office politics, grant funding, and an awful boss. There is a faint touch of the supernatural, but that strand of the story doesn't amount to much. There's a happy ending of sorts, but the story left me with a bad taste in my mouth, and I'd completely forgotten it by the time I sat down to write this review. (4) "Starship Dazzle" by Scott Bradfield: I've never seen much in Bradfield's ongoing series of stories about Dazzle, the talking dog. In this one, he's sent via rocket on a one-way trip into outer space and ends up making a bizarre sort of first contact. Like the other Dazzle stories, it's full of attempts at humor that don't really work for me, even though you'd think I'd be sympathetic to the mocking of our commercialization of everything. The ending is just silly, and not in a good way. (3) "The Old Terrologist's Tale" by S.L. Gilbow: I love the setup for this story. It's set in some sort of far future in which terraforming has become routine, and a group of people are telling each other stories over drinks. The first-person protagonist is a terrologist, someone who designs planets (and the technology is available to do this almost from scratch). The conversation is taking a turn towards the humiliating, with a politician belittling the work of terrologists, when an old terrologist who has been listening quietly starts telling a story about designing worlds, both mundane and dangerously beautiful. Gilbow does a great job here capturing blithe self-importance, the habit of belittling other people's technical work, and revenge via storytelling with a nasty barb. This was my favorite story of the issue. (7) "Altogether Elsewhere, Vast Herds of Reindeer" by Ken Liu: This is a rather odd but quite touching story about mothers, daughters, nature, connection, and uploading. It's set after a singularity, in a time when all humans are uploaded into computers and exploring higher dimensions, digital natives in a much deeper sense than is meant today. But Rene 's mother is an Ancient, from before the singularity and still three-dimensional, and she wants to spend some time with her daughter. That leads to a memorable moment of connection, without pulling Rene entirely out of her father's world. Well done. (7) "The Road Ahead" by Robert Reed: Two Reed stories in one issue! And this one is a sequel to "Stock Photos" from earlier, since apparently I wasn't the only one who found it hopelessly confusing. It provides some backstory and makes a bit more sense of the first story, and that also makes it a more interesting story in its own right. The stock photo concept wasn't entirely a lie, as I had thought it was after the first story. There is analysis, anticipation, and trends behind who the pair take pictures of. But this story explores some internal tension, some conflict between them and some knowledge that the woman has that the man doesn't. And in the process it makes everything creepier, but also more interesting, and provides a hint at a really dark way of viewing the news media. I would say that this salvages "Stock Photos," except that I don't think "Stock Photos" is necessary now that one can read this story. (7) "Music Makers" by Kate Wilhelm: This is another story about investigation of the history of music, mingled with the supernatural, but unlike the story that opened this issue, it's not horror. Rather, it's a gentle and sweet fantasy about the power of music and benevolent ghosts and a community coming together. It's a positive and happy note on which to end the issue. (6) Rating: 6 out of 10

31 October 2013

Daniel Pocock: init systems: what do users want?

The debate about Debian init system replacement continues and many developers have given their opinions. However, what do the users want? What type of features will make it easier for users to adopt and promote Debian and free software in general? Could this even help bring improvements to other distributions too? Some of the possibilities I've previously raised for discussion (and they are not specific to Debian by any means):
  • Better integration with tools for service monitoring (for example, so Nagios can automatically detect all the services on a system and the dependency relationship between them, avoiding any manual configuration of the monitoring system)
  • Better integration with tools for service management (for example, so Nagios can more easily restart a service without having to provide some action script developed by hand)
  • Better support for dependencies in clustering and distributed environments (e.g. Apache on host A should only start after MySQL is running on host B or host C)
  • Support for multiple instances of a service, each with separate configuration files on the same host
Some of the systems up for discussion go some of the way to addressing these types of requirements - none of them go all the way. What other requirements do people have? Should Debian wait a little longer and let the different solutions compete to offer more of these things before any decision is made? My own feeling is that Debian should not commit to something that will involve any loss in current features or support for ports and we should use this opportunity to find the best possible set of features for the future.

Russell Coker: Links October 2013

Wired has an interesting article by David Samuels about the Skybox, a small satellite (about the size of a bar fridge) that is being developed to provide cheap photographs of the Earth from low orbit [1]. Governments of major countries will probably try to limit what they do, but if they can prove that it s viable then someone else from a different jurisdiction will build similar satellites. Alice Dreger gave an interesting TED talk about the various ways that people can fall outside the expected genetic sex binary [2]. The short film Love is All You Need has an interesting way of showing the way that non-straight kids are treated [3]. The Guardian has an interesting article by Ranjana Srivastava about doctors and depression [4]. Don Marti wrote an interesting post about believing bullshit as a way of demonstrating group loyalty [5]. Zacqart Adam Green wrote an interesting article for the Falkvinge blog about the way that the Ouya gaming console can teach children about free software and political freedom [6]. Read more at www.ouya.tv [7]. It s a pity that the Ouya is not conveniently sold outside the US and the UK, with shipping it would probably cost a lot more than $99 in Australia. Tim Chevalier wrote an interesting post for Geek Feminism about the unintended consequences of some codes of conduct [8]. Tim Chevalier wrote an interesting Geek Feminism post about Wikipedia describing how the Neutral Point Of View is a way of representing the views of people in power [9]. Ramin Shokrizade wrote an interesting article for Gamasutra about the Free 2 Play (F2P) techniques [10]. The concept of F2P games is that the game can be installed for free but requires regular small payments to make the game easier, apparently some people pay $3000 per year or more. The TED blog has an interesting interview with Jack Andraka, a teenager who invented a new test for pancreatic cancer (and also ovarian and lung cancer) that is cheaper, faster, and less invasive than other tests [11]. The blog post also has a link to Jack s TED talk.

30 October 2013

Tanguy Ortolo: Beware of symlinks when testing a file existence

A strange problem Yesterday, I was reported a funny problem with the dokuwiki Debian package's postinst script, which contains a piece of shell script similar to that:
# Check the destination does not already exist
if [ ! -e /the/destination ]
then
    ln -s /some/file /the/destination
fi
It was failing with that message: ln: failed to create symbolic link '/the/destination': File exists. Even though I had just tested it did not exist!
View of a comete colliding with the Earth

Collision detected

The explanation The reason of that failure is quite simple, but easy to forget: test(1), aka [, uses stat(2), which dereferences symlinks when testing for file existence! So, test ! -e /the/destination does not mean check there is no file there but rather check there is either no file or only a dead symlink . So, the correct way to write such a piece of code is:
# Check there is not already a file or a (dead) symlink
if [ ! -e /the/destination -a ! -h /the/destination ]
then
    ln -s /some/file /the/destination
fi

Matthew Palmer: How to deal with the "package gcc: " problem in Puppet

In a comment to yesterday s post on why your Puppet module sucks, Warren asks what can be done about the problem of multiple modules needing to include the same package (gcc being the most common example I m yet to come across). As I stated previously, there is no sane, standardised way for multiple independent modules to cooperate to ensure that certain packages are installed for their cooperative use. If it wasn t already obvious, I consider this to be a significant failing in Puppet, and one which essentially renders any attempt at public, wide-scale reuse of Puppet modules impossible. Software packages are such a fundamental part of system management today that it is rare for a module not to want to interact with them in some way. Without strong leadership from Puppet Labs, or someone core to the project who s willing to be very noisy and obnoxious on the subject, the issue is fundamentally unsolveable, because it requires either deep (and non-backwards-compatible) changes to how Puppet works at a core level, or it needs cooperation from everyone who writes Puppet modules for public consumption (to use a single, coordinated approach to common packages). The approaches that I ve seen people advocate using, or that I ve considered myself, roughly fall into the following camps.

Put your package into a class in your module Absofrickinglutely useless, because the elementary package resources you end up creating will still conflict with other modules package resources. Anyone who suggests this can be permanently ignored, as they don t understand the problem being solved.

Use a globally common module of classes representing common packages This could work, if you could get everyone who writes modules to use it (see someone willing to be very noisy and obnoxious ). Practically speaking, anyone who suggests this doesn t understand human nature. I don t see this actually happening any time soon.

Use the defined() function everywhere By wrapping all your package resources in if !defined(Package["foo"]) blocks, you can at least stop your module from causing explosions. What it doesn t do is make sure that the various definitions of the package resource are congruent (imagine if one was package "gcc": ensure => absent ). In order to safely avoid explosion, everyone would have to follow this approach, which (in the worst case) reduces the problem to the globally common module of classes. However, it s the least-worst approach I can practically consider. At least your module won t be the cause of explosions, and realistically no module s going to ask for gcc to be removed (or, if they are, you ll quickly find them and kill them). I ve seen calls from Puppet core dev team members to deprecate and remove the defined() function entirely. Given the complete lack of alternatives, this is a disturbing illustration of just how out of touch they are from the unfortunate realities of practical Puppetry.

Use virtual resources This is yet another variant of the common class technique that would require everyone, everywhere, to dance to the same tune. It has all the same problems, and hence gets a big thumbs-down from me.

Use singleton_packages This module is a valiant attempt to work around the problem, but practically speaking is no better than using defined() because again, if a module that isn t using singleton_packages specifies package "gcc": ensure => absent , you re going to end up with assplosion. The mandatory dependency on hiera doesn t win me over, either (I m a hiera-skeptic, for reasons which I might go into some other time).

Use only modules from a single source This is the solution that I use myself, if that s any recommendation As a result of pretty much every publically-available Puppet module sucking for one reason or another, I do not, currently, have any externally-written Puppet modules in the trees that I use. Every module (158, at current count) has been written by someone in the sysadmin team at $DAYJOB, to our common standards. This means that I can coordinate use of package resources, using our module of common classes where appropriate, or refactor modules to separate concerns appropriately. If you re wondering why we don t have all of these 158 modules available publically, well, we have a few of them up, but yes, the vast majority of them aren t publically available. Some of them suck mightily, while many others are just too intertwined with other modules to be able to use on their own, and we don t want to release crap code there s far too much of that out there already.

Matthew Palmer: How to deal with the "package gcc: " problem in Puppet

In a comment to yesterday s post on why your Puppet module sucks, Warren asks what can be done about the problem of multiple modules needing to include the same package (gcc being the most common example I m yet to come across). As I stated previously, there is no sane, standardised way for multiple independent modules to cooperate to ensure that certain packages are installed for their cooperative use. If it wasn t already obvious, I consider this to be a significant failing in Puppet, and one which essentially renders any attempt at public, wide-scale reuse of Puppet modules impossible. Software packages are such a fundamental part of system management today that it is rare for a module not to want to interact with them in some way. Without strong leadership from Puppet Labs, or someone core to the project who s willing to be very noisy and obnoxious on the subject, the issue is fundamentally unsolveable, because it requires either deep (and non-backwards-compatible) changes to how Puppet works at a core level, or it needs cooperation from everyone who writes Puppet modules for public consumption (to use a single, coordinated approach to common packages). The approaches that I ve seen people advocate using, or that I ve considered myself, roughly fall into the following camps.

Put your package into a class in your module Absofrickinglutely useless, because the elementary package resources you end up creating will still conflict with other modules package resources. Anyone who suggests this can be permanently ignored, as they don t understand the problem being solved.

Use a globally common module of classes representing common packages This could work, if you could get everyone who writes modules to use it (see someone willing to be very noisy and obnoxious ). Practically speaking, anyone who suggests this doesn t understand human nature. I don t see this actually happening any time soon.

Use the defined() function everywhere By wrapping all your package resources in if !defined(Package["foo"]) blocks, you can at least stop your module from causing explosions. What it doesn t do is make sure that the various definitions of the package resource are congruent (imagine if one was package "gcc": ensure => absent ). In order to safely avoid explosion, everyone would have to follow this approach, which (in the worst case) reduces the problem to the globally common module of classes. However, it s the least-worst approach I can practically consider. At least your module won t be the cause of explosions, and realistically no module s going to ask for gcc to be removed (or, if they are, you ll quickly find them and kill them). I ve seen calls from Puppet core dev team members to deprecate and remove the defined() function entirely. Given the complete lack of alternatives, this is a disturbing illustration of just how out of touch they are from the unfortunate realities of practical Puppetry.

Use virtual resources This is yet another variant of the common class technique that would require everyone, everywhere, to dance to the same tune. It has all the same problems, and hence gets a big thumbs-down from me.

Use singleton_packages This module is a valiant attempt to work around the problem, but practically speaking is no better than using defined() because again, if a module that isn t using singleton_packages specifies package "gcc": ensure => absent , you re going to end up with assplosion. The mandatory dependency on hiera doesn t win me over, either (I m a hiera-skeptic, for reasons which I might go into some other time).

Use only modules from a single source This is the solution that I use myself, if that s any recommendation As a result of pretty much every publically-available Puppet module sucking for one reason or another, I do not, currently, have any externally-written Puppet modules in the trees that I use. Every module (158, at current count) has been written by someone in the sysadmin team at $DAYJOB, to our common standards. This means that I can coordinate use of package resources, using our module of common classes where appropriate, or refactor modules to separate concerns appropriately. If you re wondering why we don t have all of these 158 modules available publically, well, we have a few of them up, but yes, the vast majority of them aren t publically available. Some of them suck mightily, while many others are just too intertwined with other modules to be able to use on their own, and we don t want to release crap code there s far too much of that out there already.

29 October 2013

Soeren Sonnenburg: Shogun Toolbox Version 3.0 released!

Dear all, we are proud to announce the 3.0 release of the Shogun Machine-Learning Toolbox. This release features the incredible projects of our 8 hard-working Google Summer of Code students. In addition, you get other cool new features as well as lots of internal improvements, bugfixes, and documentation improvements. To speak in numbers, we got more than 2000 commits changing almost 400000 lines in more than 7000 files and increased the number of unit tests from 50 to 600. This is the largest release that Shogun ever had! Please visit http://shogun-toolbox.org/ to obtain Shogun. News Here is a brief description of what is new, starting with the GSoC projects, which deserve most fame: Screenshots Everyone likes screenshots. Well, we have got something better! All of the above projects (and more) are now documented in the form of IPython notebooks, combining machine learning fundamentals, code, and plots. Those are a great looking way that we chose to document our framework from now on. Have a look at them and feel free to submit your use case as a notebook! FGM.html GMM.html HashedDocDotFeatures.html LMNN.html SupportVectorMachines.html Tapkee.html bss_audio.html bss_image.html ecg_sep.html gaussian_processes.html logdet.html mmd_two_sample_testing.html The web-demo framework has been integrated into our website, go check them out. Other changes We finally moved to the Shogun build process to CMake. Through GSoC, added a general clone and equals methods to all Shogun objects, and added automagic unit-testing for serialisation and clone/equals for all classes. Other new features include multiclass LDA, and probability outputs for multiclass SVMs. For the full list, see the NEWS. Workshop Videos and slides In case you missed the first Shogun workshop that we organised in Berlin last July, all of the talks have been put online. Shogun in the Cloud As setting up the right environment for shogun and installing it was always one of the biggest problems for the users (hence the switching to CMake), we have created a sandbox where you can try out shogun on your own without installing shogun on your system! Basically it's a web-service which give you access to your own ipython notebook server with all the shogun notebooks. Of course you are more than welcome to create and share your own notebooks using this service! *NOTE*: This is a courtesy service created by Shogun Toolbox developers, hence if you like it please consider some form of donation to the project so that we can keep up this service running for you. Try shogun in the cloud. Thanks The release has been made possible by the hard work of all of our GSoC students, see list above. Thanks also to Thoralf Klein and Bj rn Esser for the load of great contributions. Last but not least, thanks to all the people who use Shogun and provide feedback. S ren Sonnenburg on behalf of the Shogun team (+ Viktor Gal, Sergey Lisitsyn, Heiko Strathmann and Fernando Iglesias)

Soeren Sonnenburg: Shogun goes cloud

Shogun goes cloud. Try out http://cloud.shogun-toolbox.org to interactively play with machine learning algorithms or try any of the interactive demos. The SHOGUN Machine Learning Toolbox is a collection of algorithms designed for unified learning for a broad range of feature types and learning settings, like classification, regression, or explorative data analysis. For more information visit http://www.shogun-toolbox.org.

Matthew Palmer: Why Your Puppet Module Sucks

I use Puppet a lot at work, and I use a lot of modules written by other people, as well as writing quite a number of my own. Here s a brief list of reasons why I might say that your module sucks.

1. You use global variables This would have to be both the most common idiom that just makes my teeth grind. Defined types exist for a bloody reason. Global variables make it incredibly difficult to reason about what is going to happen when I use a particular global variable (see also You don t write documentation ), and I get no feedback if I typo a global variable name. Add in a healthy heaping of lack of namespacing , and you ve basically guaranteed that your module will be loudly cursed.

2. You use parameterised classes I ve never managed to work out why parameterised classes even exist. They ve got all the problems of regular classes, as well as all the problems of types. There was a fantastic opportunity with parameterised classes to fix some of the really annoying things about regular resources, such as doing conflict checking on parameters and only barfing if there was a conflict but no, if you define the same class twice, even with identical parameters, Puppet will smack you on the hand and take away your biscuit. FFFFFFUUUUUUUUUUUU-

3. You fail at using fail() Modules are supposed to be reusable. That s their whole reason for existence. One of the benefits of Puppet is that you can provide an OS-independent interface for configuring something. However, unless you re some absolute God of cross-platform compatibility, you will only be writing your module to support those OSes or distros that you, personally, care about. That s cool if you don t know anything about OS X, you probably shouldn t be guessing at how to support something on that platform anyway. However, when you do have some sort of platform-specific code, for the love of pete, have a default or else clause that calls fail() with a useful and descriptive error message, to indicate clearly that you haven t included support for whatever environment the module s being used in. Failing to do this can cause some really spectacular explosions, because the rest of your module assumes that certain things have been done in the platform-specific code, and when it hasn t hoo boy.

4. You don t write documentation Yes, it isn t easy to write good documentation. I know that. But without documentation, your module is going to be practically unuseable. So if you don t have docs, your module is basically useless. Well done, you.

5. You have undeclared dependencies This also includes declaring your dependencies in a non-machine-parseable manner; I m not going to grovel through your README for the list of other modules I might need; I have machines to do that kind of thing for me. If I try to use your module, and it craps out on trying to reference some sort of type or class that doesn t appear at all related to your module, I will call into question your ancestry, and die a little more inside.

6. You use common packages without even trying to avoid the pitfalls OK, to be fair this is, ultimately, Puppet s grand fuckup, not yours, but you at least need to pretend to care There is no sane, standardised way in Puppet for multiple modules to install the same package. Let s say that a module I write needs to have a compiler. So I package "gcc": . Then someone else s module also wants a compiler, so it also package "gcc": . FWAKOOM! says Puppet. What the fuck? says the poor sysadmin, who just wanted both a virtualenv and an RVM environment on the one machine. Basically, using packages is going to make your module suck. Does that mean that this makes wide distribution of a large repository of modules written by different people nigh-on impossible? Yes. Fantastic.

Matthew Palmer: Why Your Puppet Module Sucks

I use Puppet a lot at work, and I use a lot of modules written by other people, as well as writing quite a number of my own. Here s a brief list of reasons why I might say that your module sucks.

1. You use global variables This would have to be both the most common idiom that just makes my teeth grind. Defined types exist for a bloody reason. Global variables make it incredibly difficult to reason about what is going to happen when I use a particular global variable (see also You don t write documentation ), and I get no feedback if I typo a global variable name. Add in a healthy heaping of lack of namespacing , and you ve basically guaranteed that your module will be loudly cursed.

2. You use parameterised classes I ve never managed to work out why parameterised classes even exist. They ve got all the problems of regular classes, as well as all the problems of types. There was a fantastic opportunity with parameterised classes to fix some of the really annoying things about regular resources, such as doing conflict checking on parameters and only barfing if there was a conflict but no, if you define the same class twice, even with identical parameters, Puppet will smack you on the hand and take away your biscuit. FFFFFFUUUUUUUUUUUU-

3. You fail at using fail() Modules are supposed to be reusable. That s their whole reason for existence. One of the benefits of Puppet is that you can provide an OS-independent interface for configuring something. However, unless you re some absolute God of cross-platform compatibility, you will only be writing your module to support those OSes or distros that you, personally, care about. That s cool if you don t know anything about OS X, you probably shouldn t be guessing at how to support something on that platform anyway. However, when you do have some sort of platform-specific code, for the love of pete, have a default or else clause that calls fail() with a useful and descriptive error message, to indicate clearly that you haven t included support for whatever environment the module s being used in. Failing to do this can cause some really spectacular explosions, because the rest of your module assumes that certain things have been done in the platform-specific code, and when it hasn t hoo boy.

4. You don t write documentation Yes, it isn t easy to write good documentation. I know that. But without documentation, your module is going to be practically unuseable. So if you don t have docs, your module is basically useless. Well done, you.

5. You have undeclared dependencies This also includes declaring your dependencies in a non-machine-parseable manner; I m not going to grovel through your README for the list of other modules I might need; I have machines to do that kind of thing for me. If I try to use your module, and it craps out on trying to reference some sort of type or class that doesn t appear at all related to your module, I will call into question your ancestry, and die a little more inside.

6. You use common packages without even trying to avoid the pitfalls OK, to be fair this is, ultimately, Puppet s grand fuckup, not yours, but you at least need to pretend to care There is no sane, standardised way in Puppet for multiple modules to install the same package. Let s say that a module I write needs to have a compiler. So I package "gcc": . Then someone else s module also wants a compiler, so it also package "gcc": . FWAKOOM! says Puppet. What the fuck? says the poor sysadmin, who just wanted both a virtualenv and an RVM environment on the one machine. Basically, using packages is going to make your module suck. Does that mean that this makes wide distribution of a large repository of modules written by different people nigh-on impossible? Yes. Fantastic.

28 October 2013

Matthias Klumpp: Tanglu update & release plans

Long time with no article about Tanglu! This was mainly because we were busy with the project itself, improving various aspects of the distribution. So, here is a new summary on what has been done and what you can expect from the first release ;-) Tanglu QA We further improved the automatic archive QA. There is now qa.tanglu.org, which constantly monitors the number of uninstallable or unbuildable packages in the Tanglu suites. It also provides tanglu-platform-smallstatus information on the metapackage generator, which helps us in finding out which packages are available on the live-cds. Furthermore, information about the staging->devel migration process is provided, to answer the question why a package does not migrate (this still needs some improvements, but it is being worked on). We also use some code from Ubuntu to monitor package versions in Debian and upstream, which helps to see if others have released newer versions of software shipped with Tanglu. This already resulted in many improvements: The Tanglu Aequorea suite does not contain unbuildable packages (at least not due to build-dependency changed), and all live-cds are working well. We will soon migrate the archive to a new server, which frees some server capacities we can use for automated QA and things like automatic live-CD building. Live-CDs, Installer, Alpha-Releases We currently don t do Alpha-Releases of Tanglu, but we create Live-CD snapshots of Tanglu, which are available at releases.tanglu.org (or mirror1, mirror2). These snapshots still have issues and are just early previews. They also ship without and installer, we are still working on that part. Please note that CD more or less means DVD or USB-Stick right now (and this won t change the expected image size will be around 800MB). Release Planning I am happy to announce that we will do a release this year, most likely in December. But what can you expect from the release? KDE 4.11kde-konqui We will ship with KDE 4.11, which will be the only desktop we officially support so far. The reason is simply lack of manpower we could promise to support more, but that would just be not realistic for the small team. So we focus on KDE (Plasma Shells) right now, and try to make it awesome. Also, the team consists mostly of KDE people right now, which contributed to that decision ;-) . If you want to try Tanglu, right now the KDE live-images are the best to try it out. GNOME 3.8 gnome-logoWe will also provide images with GNOME. The problem with GNOME is, that the GNOME team does not have enough manpower to maintain the whole desktop or to upgrade it to the latest version (it is essentially just me looking at it from time to time). So GNOME will be available in a preview state. We invite everyone with GNOME knowledge to join the project and help improving Tanglu-GNOME GNOMErs, we want you! systemd >= 204 We ship with systemd by default, which works nicely so far, although more testing needs to be done. The logind service will be used to replace ConsoleKit, if we manage to get everything in place and working in time (if there are issues, we might switch back to CK for one release). There are some plans to use a higher systemd version, due to some improvements made there, but if this will be done is still unclear (Debian will most likely stick to 204 for some time, because with systemd > 205, running it as pid 1 will be mandatory to use logind (and Debian is just in the process to decide which init-system we will use there)). Systemd will run in SysVInit compatibility mode for most of the available services. This will improve in later Tanglu releases. Of course, systemd is usable, even if not every init-script has been converted to a service file. It just has an impact on startup times, so Tanglu will not be the distributions with the fastest startup times (yet ;-) ). Debian Package Mixdebian_logo1 Tanglu consists mostly of packages from Debian Testing (Jessie), but we take full advantage of the Debian archive, so you will also find package versions from Unstable or even Experimental (where it made sense). A very small portion of packages has also been merged with Ubuntu. Although stuff has been changed, the incompatibilities with Debian are almost zero, so if you are installing Tanglu, it will currently feel like an more up-to-date Debian Testing with some extras. Still, the differences are large enough that upgrading a Debian system to Tanglu might result in some issues. Installer Right now, the installer is a major field of work. Tanglu will most likely ship with the Debian-Installer, because it is the easiest thing to do right now. For later releases, it is planned to also offer the Ubiquity installer (the thing Ubuntu uses), or a new installer with a similar UI and concept. Other Cool Things266px-Wayland_Logo Tanglu will ship with a fully working Qt5 (which is currently being tested and updated) and the latest version of Wayland/Weston as a preview for developers to play around with. We also ship with Perl 5.18 and Haskell GHC 7.6, as well as with GCC 4.8 as default compiler (although the whole distribution does not yet compile with GCC 4.8). We might ship with Linux 3.12, but this also depends on other things. The Linux kernel build will be the same as in Debian. There might be more things to come :-) Please do always keep in mind that Tanglu is a new project, so the first release might still have some teething problems but we re working on it. The Aequorea release will be supported one to two more months after the next release is out. Tanglu Policy I started to draft a Tanglu policy, which defines stuff like the procedures to become a Tanglu project member and/or developer, some basic values of the distribution, decision making processes etc. This work is at a very early stage right now, and will need lots of feedback later. But as soon as it is done, joining the project will be easier and what Tanglu is will be well-defined. The policy will also include a Code-of-Conduct as an additional document from the start. We need help! ;-) First of all, thanks to everyone working on Tanglu! You are amazing! Also, many thanks to every Debian developer or contributor who helped a lot in setting the project up and contributed knowledge. And of course, thanks to everyone else who contributed by creating awesome artwork, helped with code, TangluTanglu Logo (big) archive mirrors, buildd-servers or by testing the distribution and providing feedback! (given the state the aequorea suite was in at the beginning, testing was a really frustrating activity but people still did it!) So, if you would like to help, just find us on #tanglu-devel on Freenode or join the tg-devel mailinglist. We also really need some people taking care of the Tanglu website and updating it with some recent information from time to time, so people can see what is going on. Before we have the Tanglu policy finished and the Tanglu members and developers directory in place (a software which allows us to track all registered developers and gives them an @tanglu.org mailalias), the start might be a bit confusing, but we do our best to make it easy for new people to join. The best way is asking people. Tanglu is still created by an incredibly small team, which has a large task to accomplish. Help is welcome :-)

Dirk Eddelbuettel: Kurt Elling at Dominican University

Some catchup blogging: Kurt Elling was back in town on October 19, and not just in Chicago but quite literally in our little 'burb as the headliner for this season's concerts at Domiminican University. They had done us the favour of inviting Dianne Reeves back in March 2009, but this was clearly going to be a real treat. And I managed to snatch four front row (!!) tickets as soon as I heard about it, and the show once again didn't disappoint. Elling was as usual accompanied by his long-time collaborator Laurence Hobgood on piano, as well as local heros Clark Sommers on bass and John McLean on guitar. This time, Quincy Davies was on drums. And this band is realiably excellent, as is Elling in any live setting. The set was dominated by pieces from his most recent record and augmented by a few other standout pieces. There is a pretty rich set of live music by Elling on YouTube, see for example On Broadway (which was part of the set in a wonderfully fast and funked-up variant), Golden Lady (from the previous album, and also played live this time) or for example this wonderful version of You Send Me (also part of the concert). I snatched a photo or two (while at least turning my flash off, unlike the weird gal to my left, but I digress) and posted one on Google+. Kurt Elling will be back in Chicago at the Green Mill in early January. Expect to me there, and I hope to see you too!

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

26 October 2013

Russ Allbery: Review: Why We Fail

Review: Why We Fail, by Victor Lombardi
Publisher: Rosenfeld Media
Copyright: 2013
ISBN: 1-933820-17-9
Format: Kindle
Pages: 248
Subtitled Learning from Experience Design Failures, Why We Fail is a case study book. It defines experience design, discusses the usefulness of analyzing failure, and then walks through ten marketplace failures attributed to failures of user experience (as opposed to engineering or user interface) and tries to draw some conclusions about those failures and how they could have been prevented. Lombardi then presents a more general (if fairly simple) framework for iterative experience design, along with some discussion of how that approach could have helped in these cases. Hardware and software designs can work properly but still fail with users and customers because the overall experience isn't right. Sometimes this is because something else got there first with more market share and people got used to it, sometimes it's because a system is too complicated, sometimes it's too simple, sometimes it's just uninspiring. Most of the news media coverage and many of the blog posts on this topic look to inspirational examples of successes. Lombardi points out here that successes can be accidental and failures are often more informative, a concept already common in many more mature areas of engineering. His hope, with Why We Fail, is to push a broader industry practice of taking apart and analyzing the failures. User experience design is still in its infancy, since many of the complex interactions now possible in modern hardware and software have just recently crossed into levels of complexity that qualify as experiences rather than interactions, and could benefit from more rigor. The concept sounds great. The approach in Why We Fail is at least interesting; Lombardi tells good stories, and I think they do tell some useful lessons. Drawing general principles is harder, which is always the weakness of case-study books. Lombardi's own attempt doesn't go far beyond a familiar mantra of the scientific method, small and responsive teams, cross-functional teams with flexibility to address a whole range of technical and business issues, and honest and forthright analysis of and iteration after failure. All of this is doubtless good advice; none of it should surprise anyone with even a smattering of engineering background or familiarity with current software design philosophy. The heart of the book is in the stories, and I did like that Lombardi captured a wide range of different designs. Included are embedded systems (BMW's iDrive), traditional software packages, web services of various types, and even a protocol (OAuth). Some were vaguely familiar and some were not familiar at all, although I don't follow UI and UX discussions to any large extent. In each case, Lombardi gives a short but effective introduction to the problem space and the product, walks through the design choices, and then talks about the failure. Usefully, he doesn't stop with the failure but continues the story through the reaction of the company to the failure and any subsequent actions they took, which in some cases contain their own useful lessons. All of this, I should note, is a bit far afield of my own area of expertise (architectural building blocks and sub-surface components without as much direct interaction with the user), so I'm not the best person to judge the quality and originality of the analysis. But I liked Lombardi's approach and his analysis of multiple possible failure causes, and all of his conclusions seemed quite plausible and sensible given the case studies as described. I'm best-qualified to judge the chapter on OAuth, since I've worked extensively on web authentication systems, and his analysis of the challenges and user experiences in that case closely match my own. I didn't notice any significant errors, which is hopefully a good sign for the rest of the book. As one might expect from a short book about a complex field aimed mostly at starting a conversation, there isn't much here that's earth-shattering, nor are there simple extractable principles that will make your experience designs better. As always, the takeaway boils down to "invest time and resources into this and try to think about it systematically," which of course is true of every possible axis of improvement for any product. (The challenge is usually where to spend those limited resources.) I will also note in passing that Lombardi assumes an entirely market-driven metric for success, and all of the case studies are exclusively commercial not unexpected, since that's where most of the money and a lot of the resources are, but not that helpful for bespoke work where the nature of the problem is subtlely different. But, despite the expected shortcomings and the inherent weakness of the case study approach to analysis, which tries to pluralize anecdote to get data I enjoyed this book. I also got at least one useful trick out of it: the premortem, to look for potential failure points in the experience before rolling out, or even designing, the product. It's a fairly light-weight book, but I think it's moderately inspirational, and I wholeheartedly concur with Lombardi on the merits of failure analysis as an engineering approach. Rating: 7 out of 10

25 October 2013

David Welton: MicroConf Europe

I don't envy conference organizers these days - most of what's being said can be read the next day, for free, on line, at your own pace, from the comfort of your own home, and without spending a bundle of time and money to sleep in a far away hotel. Competing with that is not easy, but the guys at MicroConf managed to. I would sum up the weekend by saying that it was a "very high bandwidth experience". Every day, from breakfast until I turned in, I was chatting with people or listening to speakers during the conference itself. That's aproximately 16 hours of being "on", and by the time I got home to Padova, I was exhausted! But at the end of the day, I felt like it was worth it being there in person, because of all the interaction with other people. The speakers' talks all ended up on line, more or less, but all the chatting and discussion and getting to know everyone is the human element that is tough to replicate on line, and one of the most important reasons to attend a conference in person. Prague is also a beautiful city - I wish I had had more time there to check it out. Here are some highlights and notes, in no particular order: However, since it was a business conference, I also have to put on my cold, hard accountant hat. Will the conference pay for itself? Only time will tell. I learned a variety of interesting and useful things, many of which I think I can put into practice. The problem is finding the time between consulting work and family, but that was a bottleneck before, too - I had, and have, more things to do than time. Also, to be very direct about it, how much of what I learned could not have been learned by carefully reading accounts of the conference, slides, and other material published on the internet? A lot of it. I'm not sure I would have paid attention to all of it though, so the conference was definitely nice in that it exposed me to some talks and ideas that otherwise I might have brushed off before giving them a chance. In terms of dollars and cents, I won't be able to say for a while whether it was a sensible investment or not. Would I go again? I'd like to - it was a lot of fun and the people were great. Like I said, it's tough doing conferences because your competition is the internet!

Enrico Zini: A vision wanted

A vision wanted Today Richard Stallman mailed all Italian LUGs asking that tomorrow's LinuxDay be called "GNU/Linux Day" instead. I wonder how that is ever going to help a community so balkanised, that the only way Italian LUGs manage to do something together, is to say "let's not care what we all do, let's just do it on the same day and call it a national event". Of course a few LUGs still make a point of not doing anything on that day, because you know, Judean People's Front. Cawk. Today a friend asked me if I could help her support people in installing Citrix Whatsit to set up a video conference to network meetings that will take place in a month in different cities. Those meetings are something I look forwad to. It wasn't much of a problem to say "no, I can't do that"; it was a problem to be unable to come up with some viable, Free alternatives. I sometimes have to use Skype to talk with friends who also are Debian Developers, because I still haven't managed to make VoIP work unless I go through a commercial proxy. There was the happy news that our regional administration is switching from MS Word to OpenOffice. It soon became a flamewar, because some people loudly complained that they should have used LibreOffice instead. At DebConf, after spending an hour getting frustrated with the default formatting of bullet points in WhateverOffice Impress, I did my Debian Contributors talk using a text file in vim. And it was a success! Thanks Francois Marier for maintaining cowsay. I can't sync contact lists and appointments between my N900, which runs a Debian derivative, and my laptop, because I don't want to have a Google account, and nothing else would work out of the box. I don't even know how to keep a shared calendar with other DDs, without using a 3rd party cloud service that I don't want to trust with my life's personal schedule. I need to do a code review of every vim plugin I need to use, because you can only get them by cloning GitHub repositories over plain http, and they don't even provide signed tags. That code is run with my own privileges every time I start a text editor, which is, like, all the time. I'm frightened at the idea of how many people blissfully don't think about what that means. Vim users. Programmers. Cool chaps. Yet the most important thing in Debian today seems to be yet another crusade between upstart and systemd. But we haven't had a lengthy discussion on why, although the excellent OpenStreetMap exists and many of us contribute to it, it seems to still be more immediate to hit Google Maps to get a route computed. How can we change that? We haven't had a lengthy discussion on what can we offer to allow anyone to set up some social platform that won't get swamped with spam the first day and cracked open the second; that would allow people to share some photos with their friends only, and some with the rest of the world; that won't required a full-time paid person to maintain. That won't be obsolete and require a migration to a new platform in a year. That isn't Facebook or Google Plus. I stopped taking photos because it's too much work to show them to people. Other people use Instagram. Whatever the hipster trend is for photo sharing today, October 25, 2013, I'm pretty sure it's not a Free platform. But we can do something. We technology leaders. We are those who drive technological change! For example, today I invested two hours of hard effort trying to figure out why libbuffy's test suite fails on kfreebsd. All while wondering why I was doing that, since I know all buffy's users personally, and none of them uses kfreebsd. And I will take a day off work to study the library symbols file specification, so that next time I'll know right away if the new version of a C++ compiler decides that a template-generated symbol isn't worth adding to a library anymore. What is this effort really about? It sometimes feel like micromanaging to me. It's good to have excellent quality standards. But not without a vision. Not until "reliable network printing with all PDF viewers and print servers we ship" is among our release goals. Not until we commit to making sure that "sharing files between Debian users" will work out of the box, without the need of going through a 3rd party website, or email. I'm not interested in spending energy discussing init systems. I'm interested in spending energy sharing stories of what cool stuff we can do in Debian today, out of the box. And what cool stuff we'll be able to do tomorrow. Let's spend time on IRC, on mailing lists, and at the next Debian events, talking about why we are really into this. Talking about a vision! Note: Please don't spend time telling me how to fix the problems I mentioned above. I'm not interested in help fixing some problems for me today. I'm interested in asking help fixing problems for everybody, right in the next stable release. Remember, remember, the 5th of November, 2014.

Next.

Previous.