Hello The last few weeks, I ve been banging my head to use uscan, Pithub and JIRA::Client::Automated behind a corporate firewall. They are all written in Perl and use LWP::UserAgent to fetch information from Internet. At $work, using a proxy is mandatory to connect to Internet. But LWP::UserAgent https connection does not work through a proxy. This bug was reported 10 years ago but is still not fixed. Here s my plan to fix (or work-around) this issue, at least for Debian. Before going further, let s step back to explain briefly what is https and how proxies work. https is not a protocol by itself. https is plain http over a socket encrypted with SSL (aka TLS). To create a https connection, the agent must first setup the SSL socket with the server. Then the agent uses http protocol to communicate with the server. All the traffic is encrypted on agent side and decrypted on server side by the SSL layer. When creating a connection through a proxy, things get a little more complicated. Plain http requests (like GET, POST and so on ) are sent to the proxy as if the proxy was the http server. Then, the proxy forwards the request to the actual server. From what I ve read, most proxy servers refuse to plainly forward encrypted data to a web server. First a negotiation to create a tunnel towards the web server must be done by sending a http CONNECT request to the proxy server. The encrypted socket is then set up between the user and the web server through the proxy. Once this encrypted tunnel is set up, the usual http communication can be done. Let s go back to LWP::UserAgent. To create a connection over SSL, LWP::UserAgent will use a SSL library to setup the socket. This SSL library can be IO::Socket::SSL or Net::SSL. Direct https connection with LWP::UserAgent works fine with either library. When trying to setup a https connection through a proxy, LWP::UserAgent will try to use the proxy like a regular http proxy without going through the CONNECT phase. This does not work. Net::SSL is able to setup correctly an SSL socket through a proxy provided the proxy server is specified in an https_proxy environment variable. But LWP::UserAgent will also try to (badly) setup the proxy when this variable is set. One can find various work-around to play with %ENV to bypass LWP::UserAgent. So I ve proposed a patch to make LWP::UserAgent skip the https_proxy variable. Then Net::SSL can handle the connection to the proxy. I hope this blog will push upstream to either apply this path or tell me why it s a bad idea (which is possible, I ve some trouble to believe that a 10 years old bug can be fixed by such a simple patch) [ Edited ] I was right to worry. Turns out that relying on Net::SSL is a bad idea: SSL certificates are not checked correctly with this library. For more details, see https://github.com/libwww-perl/libwww-perl/pull/51 and https://bugzilla.redhat.com/show_bug.cgi?id=705044 . I m going to push JGMYERS set of patches on libwww-perl repositories Anyway, this patch works provided:

• Net::SSL is loaded by the application
• the proxy is setup with https_proxy environment variable
• LWP user agent is created this way:

  my $ua = LWP::UserAgent->new(
      env_proxy => 1,
      ssl_opts =>   verify_hostname => 0 
  );

  The verify_hostname parameter is used by IO::Socket::SSL.
  LWP::UserAgent bails out if Net::SSL is used while this parameter is not set to 0.

Even though this patch is probably not ideal, it does fix issues with uscan so I ll push it to Debian. On the other hand, the patch does not work if https_proxy is specified by calling:

$ua->proxy( https => ...);

JGMYERS has proposed a set of patches to fix this problem. But these patches are more invasive and I do not dare to push them to Debian. Last but not least, this blog avoids the issue of host name verification or peer authentication. Net::SSL and Crypt::SSLeay are able to perform such a verification, but I did not verify this point. Next step is to provide simple patches to let uscan, Pithub and JIRA::Client::Automated correctly connect through proxies without jumping through hoops. All the best
Tagged: debian, https_proxy, Perl

I feel like I m taking a big personal risk writing this, even though I know the internet is large and probably no-one will read this . So, dear reader, please be gentle. As we grow as people, as developers, as professionals some lessons are are hard to learn (e.g. you have to keep trying and trying to learn the task), and some are hard to experience (they might still be hard to learn, but just being there is hard itself ) I want to talk about a particular lesson I started learning in late 2008/early 2009 while I was at Canonical sadly one of those that was hard to experience. At the time I was one of the core developers on Bazaar, and I was feeling pretty happy about our progress, how bzr was developing, features, community etc. There was a bunch of pressure on to succeed in the marketplace, but that was ok, challenges bring out the stubborn in me . There was one glitch though we d been having a bunch of contentious code reviews, and my manager (Martin Pool) was chatting to me about them. I was as far as I could tell doing precisely the right thing from a peer review perspective: I was safeguarding the project, preventing changes that didn t fit properly, or that reduced key aspects- performance, usability from landing until they were fixed. However, the folk on the other side of the review were feeling frustrated, that nothing they could do would fix it, and generally very unhappy. Reviews and design discussions would grind to a halt, and they felt I was the cause. [They were right]. And here was the thing I simply couldn t understand the issue. I was doing my job; I wasn t angry at the people submitting code; I wasn t hostile; I wasn t attacking them (but I was being shall we say frank about the work being submitted). I remember saying to Martin one day look, I just don t get it can you show me what I said wrong? and he couldn t. Canonical has a 360 review system every 6 months / year (it changed over time) you review your peers, subordinate(s) and manager(s), and they review you. Imagine my surprise I was used to getting very positive reports with some constructive suggestions when I scored low on a bunch of the inter-personal metrics in the review. Martin explained that it was the reviews thing folk were genuinely unhappy, even as they commended me on my technical merits. Further to that, he said that I really needed to stop worrying about technical improvement and focus on this inter-personal stuff. Two really important things happened around this time. Firstly, Steve Alexander, who was one of my managers-once-removed at the time, reached out to me and suggested I read a book Getting out of the box and that we might have a chat about the issue after I had read it. I did so, and we chatted. That book gave me a language and viewpoint for thinking about the problem. It didn t solve it, but it meant that I got it , which I hadn t before. So then the second thing happened we had a company all hands and I got to chat with Claire Davis (head of HR at Canonical at the time) about what was going on. To this day the sheer embarrassment I felt when she told me that the broad perception of me amongst other teams managers was and I paraphrase a longer, more nuance conversation here technically fantastic but very scary to have on the team will disrupt and cause trouble . So, at this point about 6 months had passed, I knew what I wanted I wanted folk to want to work with me, to find my presence beneficial and positive on both technical and team aspects. I already knew then that what I seek is technical challenges: I crave novelty, new challenges, new problems. Once things become easy, it call all too easily slip into tedium. So at that time my reasoning was somewhat selfish: how was I to get challenges if no-one wanted to work with me except in extremis? I spent the next year working on myself as much as specific projects: learning more and more about how to play well with others. In June 2010 I got a performance review I could be proud of again I was in no way perfect, but I d made massive strides. This journey had also made huge improvements to my personal life a lot of stress between Lynne and I had gone away. Shortly after that I was invited to apply for a new role within Canonical as Technical Architect for Launchpad and Francis Lacoste told me that it was only due to my improved ability to play well with others that I was even considered. I literally could not have done the job 18 months before. I got the job, and I think I did pretty well in fact I was awarded an internal Spotlight on Success award for what we (it was a whole Launchpad team team effort) achieved while I was in that role. So, what did I change/learn? There s just a couple of key changes I needed to make in myself, but a) they aren t sticky: if I get overly tired, ye old terrible Robert can leak out, and b) there s actually a /lot/ of learnable skills in this area, much of which is derived lots of practice and critical self review is a good thing. The main thing I learnt was that I was Selfish. Yes capital S. For instance, in a discussion about adding working tree filter to bzr, I would focus on the impact/risk on me-and-things-I-directly-care-about: would it make my life harder, would it make bzr slower, was there anything that could go wrong. And I would spend only a little time thinking about what the proposer needed: they needed support and assistance making their idea reach the standards the bzr community had agreed on. The net effect of my behaviours was that I was a class A asshole when it came to getting proposals into a code base. The key things I had to change were:

1. I need to think about the needs of the person I m speaking to *and not my own*. [Thats not to say you should ignore your needs, but you shouldn't dwell on them: if they are critical, your brain will prompt you].
2. There s always a reason people do things: if it doesn t make sense, ask them! [The crucial conversations books have some useful modelling here on how and why people do things, and on how-and-why conversations and confrontations go bad and how to fix them.]

Ok so this is all interesting and so forth, but why the blog post? Firstly, I want to thank four folk who were particularly instrumental in helping me learn this lesson: Martin, Steve, Claire and of course my wife Lynne I owe you all an unmeasurable debt for your support and assistance. Secondly, I realised today that while I ve apologised one on one to particular folk who I knew I d made life hard for, I d never really made a widespread apology. So here it is: I spent many years as an ass, and while I didn t mean to be one, intent doesn t actually count here actions do. I m sorry for making your life hell in the past, and I hope I m doing better now. Lastly, if I m an ass to you now, I m sorry, I m probably regressing to old habits because I m too tired something I try to avoid, but it s not always possible. Please tell me, and I will go get some sleep then come and apologise to you, and try to do better in future. Less-assily-yrs,
Rob

Review: Fantasy & Science Fiction, May/June 2011

Editor:	Gordon van Gelder
Issue:	Volume 120, No. 5 & 6
ISSN:	1095-8258
Pages:	258

The editorial in this issue is about the investigation into the troubling death of long-time contributor F. Gwynplain MacIntyre (which was not his real name). It's disturbing, but to me it underscores one of the things that I love about the Internet: people for whom life isn't working very well can still find an outlet, make friendships, and control how they choose to present themselves to the world on-line. That's something quite valuable, and part of why the pushes for "real names" always gives me pause. Somewhat thematically related, this issue also features a non-fiction essay by Maria E. Alonzo about her investigation of Jesse Francis McComas, her great-uncle but better known to the SF community as one of the founding editors of F&SF and co-editor of the famous classic anthology Adventures in Time and Space. This is mostly a curiosity, but it's fun to read about the sense of triumph in tracking down lost family history. This issue also features a Chris Moriarty book review column, always a plus, as well as a few positive reviews of obscure superhero movies by Kathi Maio (plus the required grumbling about a more mainstream film). "The Final Verse" by Chet Williamson: This is more of a horror story than I would normally like, but I got pulled into the investigation of an old bluegrass song and the guesswork and footwork required to track down where it came from. Williamson does a good job with the tone and first-person narration, and the degree to which the protagonist cares about the song to the exclusion of the horrific happenings of the story blunts the horror. Not quite my thing, but I thought it was well-done and played well with the possible meanings of song lyrics. (6) "Stock Photos" by Robert Reed: This is well-written, like nearly all Reed stories, but it lacked enough clues for the reader for me. It's a very short story about a man who's out mowing his lawn when approached by two strangers who apparently want to take photographs of him for stock image collections. Then things get rather weird, but without any explanation, and the ending lost me completely. Frustrating. (It is partially explained by the later "The Road Ahead" story in this same issue.) (4) "The Black Mountain" by Albert E. Cowdrey: From one of F&SF's most reliable story-tellers to another, and this is a more typical story. Cowdrey offers an abandoned and very strange cathedral for an obscure religion, a conflict over a development project, and some rather creepy results, all told in Cowdrey's entertaining fashion. Some places you just don't mess with. (6) "Agent of Change" by Steven Popkes: Told Dos-Passos-style with news excerpts, web sites, and the transcript of an emergency committee, this story shows the discovery of Godzilla, or something akin to Godzilla, in the Pacific, where it's destroying whaling vessels. I do like this style of storytelling, and here it mixes well with humor and a bit of parody as Popkes shows how each different news outlet puts its own recognizable spin on the story. The story isn't particularly memorable, and it doesn't end so much as just stop, but it was fun. (7) "Fine Green Dust" by Don Webb: This story is dedicated to Neal Barrett, which will give SFF short story readers a warning of weirdness to come. In a near future where global warming as continued to make summers even more miserable, the protagonist happens across a naked woman painted green. The green turns out to be a sun block that claims to assist humans in metamorphosis into animals. Most of the story is the protagonist trying to decide what to think of that, interspersed with staring at his neighbor's naked daughter. It's mildly amusing if you don't think about it too much and don't mind the rather prominent male gaze. (5) "Rampion" by Alexandra Duncan: The novella of the story, this is set in Muslim Spain some time during the long fights between Muslims and Christians in the north. It's told as two parallel stories: one telling the protagonist's first meeting with his love, and the second following him as a blind man, some time later, deciding whether, and how, to re-engage with the world. The style feels like fantasy, but there's very little overt fantasy here, and the story could be read as historical adventure. It's good adventure, though; conventional in construction, but with some romance and some drama and a good ending. (7) "Signs of Life" by Carter Scholz: This is to science fiction what "Rampion" is to fantasy: not really SF in the classic sense, but fiction about the process of science. The protagonist works on gene sequencing and is mildly obsessed with a visualization of junk DNA in an attempt to find patterns in it. Like a lot of fiction about science, it's primarily concerned with office politics, grant funding, and an awful boss. There is a faint touch of the supernatural, but that strand of the story doesn't amount to much. There's a happy ending of sorts, but the story left me with a bad taste in my mouth, and I'd completely forgotten it by the time I sat down to write this review. (4) "Starship Dazzle" by Scott Bradfield: I've never seen much in Bradfield's ongoing series of stories about Dazzle, the talking dog. In this one, he's sent via rocket on a one-way trip into outer space and ends up making a bizarre sort of first contact. Like the other Dazzle stories, it's full of attempts at humor that don't really work for me, even though you'd think I'd be sympathetic to the mocking of our commercialization of everything. The ending is just silly, and not in a good way. (3) "The Old Terrologist's Tale" by S.L. Gilbow: I love the setup for this story. It's set in some sort of far future in which terraforming has become routine, and a group of people are telling each other stories over drinks. The first-person protagonist is a terrologist, someone who designs planets (and the technology is available to do this almost from scratch). The conversation is taking a turn towards the humiliating, with a politician belittling the work of terrologists, when an old terrologist who has been listening quietly starts telling a story about designing worlds, both mundane and dangerously beautiful. Gilbow does a great job here capturing blithe self-importance, the habit of belittling other people's technical work, and revenge via storytelling with a nasty barb. This was my favorite story of the issue. (7) "Altogether Elsewhere, Vast Herds of Reindeer" by Ken Liu: This is a rather odd but quite touching story about mothers, daughters, nature, connection, and uploading. It's set after a singularity, in a time when all humans are uploaded into computers and exploring higher dimensions, digital natives in a much deeper sense than is meant today. But Rene 's mother is an Ancient, from before the singularity and still three-dimensional, and she wants to spend some time with her daughter. That leads to a memorable moment of connection, without pulling Rene entirely out of her father's world. Well done. (7) "The Road Ahead" by Robert Reed: Two Reed stories in one issue! And this one is a sequel to "Stock Photos" from earlier, since apparently I wasn't the only one who found it hopelessly confusing. It provides some backstory and makes a bit more sense of the first story, and that also makes it a more interesting story in its own right. The stock photo concept wasn't entirely a lie, as I had thought it was after the first story. There is analysis, anticipation, and trends behind who the pair take pictures of. But this story explores some internal tension, some conflict between them and some knowledge that the woman has that the man doesn't. And in the process it makes everything creepier, but also more interesting, and provides a hint at a really dark way of viewing the news media. I would say that this salvages "Stock Photos," except that I don't think "Stock Photos" is necessary now that one can read this story. (7) "Music Makers" by Kate Wilhelm: This is another story about investigation of the history of music, mingled with the supernatural, but unlike the story that opened this issue, it's not horror. Rather, it's a gentle and sweet fantasy about the power of music and benevolent ghosts and a community coming together. It's a positive and happy note on which to end the issue. (6) Rating: 6 out of 10

The SPICE protocol for remote display access is the preferred solution with oVirt and RedHat Enterprise Virtualization, and I was sad to discover the other day that the browser plugin needed to use these systems seamlessly was missing in Debian. The request for a package was from 2012-04-10 with no progress since 2013-04-01, so I decided to wrap up a package based on the great work from Cajus Pollmeier and put it in a collab-maint maintained git repository to get a package I could use. I would very much like others to help me maintain the package (or just take over, I do not mind), but as no-one had volunteered so far, I just uploaded it to NEW. I hope it will be available in Debian in a few days. The source is now available from http://anonscm.debian.org/gitweb/?p=collab-maint/spice-xpi.git;a=summary.

Today I'm recuperating, and almost back to full health. Unfortunately I made the mistake of online-shopping, oops. Good job I stopped myself from registaring all the domains, but I did get two that I liked: spare.io & edinburgh.io. I've updated my database to record them, but I wonder what do other people use to remind them about expiration dates of domains, SSL-certificates, & etc? I googled and didn't find a definitive free/paid service, but it seems like something lots of people need to be reminded about.. Maybe people just rely on registrars sending strident emails. (Of course the redemption period for domains make it reasonably safe to forget for a day or two, until your customers complain and your emails start to bounce..)

Spam trap Dear lazyweb, I am considering to implement spam traps and evaluate their efficiency. The idea as rather simple:

1. publish some real-looking email addresses on websites, in ways that no human would use them to send legitimate mail, for instance in hidden texts, or in texts clearly stating they should not be used;
2. when my mail server receives a message for one of these address, blacklist the originating server for some time so it cannot spam real recipients.

Practical implementation Now, I have to decide on how to implement that with my Postfix server. I am thinking of using fail2ban to simply detect attempts to send mail to my spam traps and blocking these spammers at iptables-level, but I do not think that will give me much tuning possibilities. Do you have other suggestions? Implementation details I also have to think about some specific details. Here is what I identified so far, do you think I am missing something that may be useful?

• Blacklisting time: since no legitimate mail can ever be sent to an explicit spam trap address, everyone sending mail to these addresses is by definition a spammer, so I can block them for months, if not forever. There is, however, a risk of blocking large scale services mostly used by regular users but sometimes abused by spammers, like Hotmail or Gmail
• Server response: a spam trap address is only efficient if it gets included in a spammer's address list and if it stays there, so I should make sure my server answers the first attempt in a way that does not make him think that address is problematic and should be removed from his list. Defining a /dev/null alias should do it.

Wired has an interesting article by David Samuels about the Skybox, a small satellite (about the size of a bar fridge) that is being developed to provide cheap photographs of the Earth from low orbit [1]. Governments of major countries will probably try to limit what they do, but if they can prove that it s viable then someone else from a different jurisdiction will build similar satellites. Alice Dreger gave an interesting TED talk about the various ways that people can fall outside the expected genetic sex binary [2]. The short film Love is All You Need has an interesting way of showing the way that non-straight kids are treated [3]. The Guardian has an interesting article by Ranjana Srivastava about doctors and depression [4]. Don Marti wrote an interesting post about believing bullshit as a way of demonstrating group loyalty [5]. Zacqart Adam Green wrote an interesting article for the Falkvinge blog about the way that the Ouya gaming console can teach children about free software and political freedom [6]. Read more at www.ouya.tv [7]. It s a pity that the Ouya is not conveniently sold outside the US and the UK, with shipping it would probably cost a lot more than $99 in Australia. Tim Chevalier wrote an interesting post for Geek Feminism about the unintended consequences of some codes of conduct [8]. Tim Chevalier wrote an interesting Geek Feminism post about Wikipedia describing how the Neutral Point Of View is a way of representing the views of people in power [9]. Ramin Shokrizade wrote an interesting article for Gamasutra about the Free 2 Play (F2P) techniques [10]. The concept of F2P games is that the game can be installed for free but requires regular small payments to make the game easier, apparently some people pay $3000 per year or more. The TED blog has an interesting interview with Jack Andraka, a teenager who invented a new test for pancreatic cancer (and also ovarian and lung cancer) that is cheaper, faster, and less invasive than other tests [11]. The blog post also has a link to Jack s TED talk.

• [1] http://www.wired.com/wiredscience/2013/06/startup-skybox/
• [2] http://www.ted.com/talks/alice_dreger_is_anatomy_destiny.html
• [3] http://www.youtube.com/watch?v=3ROXTFfkcfo
• [4] http://www.theguardian.com/commentisfree/2013/oct/09/doctors-depressed
• [5] http://zgp.org/~dmarti/misc/believing-bullshit/
• [6] http://falkvinge.net/2013/06/27/oh-boy-more-ouya-hype/
• [7] https://www.ouya.tv/
• [8] http://tinyurl.com/q4lfp27
• [9] http://geekfeminism.org/2013/07/11/when-who-you-are-is-off-topic/
• [10] http://tinyurl.com/nbto4vm
• [11] http://tinyurl.com/l384e2s

The point of switching to a new init system is to get rid of huge init scripts. Well, here s an example OpenRC runscript, rewriting /etc/init.d/rsyslog:

#!/sbin/runscript
command=/usr/sbin/rsyslogd
name="enhanced syslogd"
depend()
 
        provide rsyslogd syslog
        need $remote_fs $time
 

Pretty minimalistic to me... For the record, the original sysv-rc script was 126 lines, and the above is just 10 lines.

Many protocols today allow you to upgrade to TLS from within a cleartext version of the protocol. This often falls under the rubric of "STARTTLS", though different protocols have different ways of doing it. I often forget the exact steps, and when i'm debugging a TLS connection (e.g. with tools like gnutls-cli) i need to poke a remote peer into being ready for a TLS handshake. So i'm noting the different mechanisms here. lines starting with C: are from the client, lines starting with S: are from the server. many of these are (roughly) built into openssl s_client, using the -starttls option. Sometimes this doesn't work because the handshake needs tuning for a given server; other times you want to do this with a different TLS library. To use the techniques below with gnutls-cli from the gnutls-bin package, just provide the --starttls argument (and the appropriate --port XXX argument), and then hit Ctrl+D when you think it's ok to start the TLS negotiation. SMTP The polite SMTP handshake (on port 25 or port 587) that negotiates a TLS upgrade looks like:

C: EHLO myhostname.example
S: [...]
S: 250-STARTTLS
S: [...]
S: 250 [somefeature]
C: STARTTLS
S: 220 2.0.0 Ready to start TLS
<Client can begin TLS handshake>

IMAP The polite IMAP handshake (on port 143) that negotiates a TLS upgrade looks like:

S: OK [CAPABILITY IMAP4rev1 [...] STARTTLS [...]] [...]
C: A STARTTLS
S: A OK Begin TLS negotiation now
<Client can begin TLS handshake>

POP The polite POP handshake (on port 110) that negotiates a TLS upgrade looks like:

S: +OK POP3 ready
C: STLS
S: +OK Begin TLS 
<Client can begin TLS handshake>

XMPP The polite XMPP handshake (on port 5222 for client-to-server, or port 5269 for server-to-server) that negiotiates a TLS upgrade looks something like (note that the domain requested needs to be the right one):

C: <?xml version="1.0"?><stream:stream to="example.net"
C:  xmlns="jabber:client" xmlns:stream="http://etherx.jabber.org/streams" version="1.0">
S: <?xml version='1.0'?>
S: <stream:stream
S:  xmlns:db='jabber:server:dialback'
S:  xmlns:stream='http://etherx.jabber.org/streams'
S:  version='1.0'
S:  from='example.net'
S:  id='d34edc7c-22bd-44b3-9dba-8162da5b5e72'
S:  xml:lang='en'
S:  xmlns='jabber:server'>
S: <stream:features>
S: <dialback xmlns='urn:xmpp:features:dialback'/>
S: <starttls xmlns='urn:ietf:params:xml:ns:xmpp-tls'/>
S: </stream:features>
C: <starttls xmlns="urn:ietf:params:xml:ns:xmpp-tls" id="1"/>
S: <proceed xmlns='urn:ietf:params:xml:ns:xmpp-tls'/>
<Client can begin TLS handshake>

NNTP RogerBW (in the comments below) points out that NNTP has TLS support:

C: CAPABILITIES
S: [...]
S: STARTTLS
S: [...]
S: .
C: STARTTLS
S: 382 Continue with TLS negotiation

PostgreSQL I got mail from James Cloos suggesting how to negotiate an upgrade to TLS over the PostgreSQL RDBMS. He points to the protocol docs, and in particular, to multiple protocol flow documents, and SSLRequest and StartupMessage chunks of the protocol spec (and clarification that data is sent in network byte order). It won't work in a text-mode communication, but it's worth noting here anyway: The client starts by sending these eight octets:

0x00 0x00 0x00 0x08 0x04 0xD2 0x16 0x2F

and the server replies with 'S' for secure or 'N' for not. If the reply is S, TLS negotiation follows. The message represents int32(8) specifying that there are 8 octets and int16(1234),int16(5678). All sent in network order. (The non-TLS case starts with a similar message with int16(3),int16(0) for protocol version 3.0. Starttls is essentially pg protocol version 1234.5678.) what else? I don't know (but would like to) how to do:

• mysql TLS negotiation
• other reasonable network protocols capable of upgrade
• other free TLS wrapping tools like openssl s_client or gnutls-cli that can start off in the clear and negotiate to TLS. I am trying to get libNSS's tstclnt into the libnss3-tools package, but that hasn't happened yet.

If you know other mechanisms, or see bugs with the simple handshakes i've posted above, please let me know either by e-mail or on the comments here. Other interesting notes: RFC 2817, a not-widely-supported mechanism for upgrading to TLS in the middle of a normal HTTP session.Tags: gnutls, imap, pop, postgresql, smtp, starttls, tls, xmpp

This could have been a blog post about toothbrushes for monkeys, which I seem to have dreamed about this morning. But then I was vaguely listening to the FAIFCast in the car, and I came up with something even more esoteric to write about! License monads would allow separating parts of code that are under different licenses, in a rigorous fashion. For example, we might have two functions in different license monads: Perhaps foo needs to use bar to calculate its value. It can only do so if there's a way to lift code in the BSD monad into the GPL monad. Which we can legally write, since the BSD license is upwards-compatable with the GPL: On the other hand, there should be no way provided to lift the GPL monad into the BSD monad. So bar cannot be written using code from foo, which would violate foo's GPL license. Perhaps the reason I am thinking about this is that the other day I found myself refactoring some code out of the git-annex webapp (which is AGPL licensed) and into the git-annex assistant (which is GPL licensed). Which meant I had to relicense that code. Luckily that was easy to do legally speaking, since I am the only author of the git-annex webapp so far, and own the whole license of it. (Actually, it's only around 3 thousand lines of code, and another thousand of html.) It also turned out to be easy to do the refactoring, technically speaking because looking at the code, I realized I had accidentially written it in the wrong monad; all the functions were in the webapp's Handler monad, but all of them used liftAnnex to actually do their work in the Annex monad. If that had not been the case, I would not have been able to refactor the code, at least not without entirely rewriting it. It's as if I had accidentially written: Which can be generalized to: I don't think that license monads can be realistically used in the current world, because lawyers and math often don't mix well. Lawyers have, after all, in the past written laws re-defining pi to be 3. Still, license monads are an interesting way to think about things for myself. They capture how my code and licenses are structured and allow me to reason about it on a more granular level than the licences of individual files. (I have a vague feeling someone else may have written about this idea before? Perhaps I should have been blogging about monkey toothbrushes after all...)

A strange problem Yesterday, I was reported a funny problem with the dokuwiki Debian package's postinst script, which contains a piece of shell script similar to that:

# Check the destination does not already exist
if [ ! -e /the/destination ]
then
    ln -s /some/file /the/destination
fi

It was failing with that message: ln: failed to create symbolic link '/the/destination': File exists. Even though I had just tested it did not exist! The explanation The reason of that failure is quite simple, but easy to forget: test(1), aka [, uses stat(2), which dereferences symlinks when testing for file existence! So, test ! -e /the/destination does not mean check there is no file there but rather check there is either no file or only a dead symlink . So, the correct way to write such a piece of code is:

# Check there is not already a file or a (dead) symlink
if [ ! -e /the/destination -a ! -h /the/destination ]
then
    ln -s /some/file /the/destination
fi

I have always pretended that OpenRC would be very easy to port to Debian GNU/kFreeBSD. Well, now I stop pretending: http://youtu.be/zoNoi8BgQjs This was done within a few hours working with upstream. Now, next-up: Debian GNU/Hurd. Hoping that porters will volunteer to do the work.

In a comment to yesterday s post on why your Puppet module sucks, Warren asks what can be done about the problem of multiple modules needing to include the same package (gcc being the most common example I m yet to come across). As I stated previously, there is no sane, standardised way for multiple independent modules to cooperate to ensure that certain packages are installed for their cooperative use. If it wasn t already obvious, I consider this to be a significant failing in Puppet, and one which essentially renders any attempt at public, wide-scale reuse of Puppet modules impossible. Software packages are such a fundamental part of system management today that it is rare for a module not to want to interact with them in some way. Without strong leadership from Puppet Labs, or someone core to the project who s willing to be very noisy and obnoxious on the subject, the issue is fundamentally unsolveable, because it requires either deep (and non-backwards-compatible) changes to how Puppet works at a core level, or it needs cooperation from everyone who writes Puppet modules for public consumption (to use a single, coordinated approach to common packages). The approaches that I ve seen people advocate using, or that I ve considered myself, roughly fall into the following camps.

Put your package into a class in your module Absofrickinglutely useless, because the elementary package resources you end up creating will still conflict with other modules package resources. Anyone who suggests this can be permanently ignored, as they don t understand the problem being solved.

Use a globally common module of classes representing common packages This could work, if you could get everyone who writes modules to use it (see someone willing to be very noisy and obnoxious ). Practically speaking, anyone who suggests this doesn t understand human nature. I don t see this actually happening any time soon.

Use the defined() function everywhere By wrapping all your package resources in if !defined(Package["foo"]) blocks, you can at least stop your module from causing explosions. What it doesn t do is make sure that the various definitions of the package resource are congruent (imagine if one was package "gcc": ensure => absent ). In order to safely avoid explosion, everyone would have to follow this approach, which (in the worst case) reduces the problem to the globally common module of classes. However, it s the least-worst approach I can practically consider. At least your module won t be the cause of explosions, and realistically no module s going to ask for gcc to be removed (or, if they are, you ll quickly find them and kill them). I ve seen calls from Puppet core dev team members to deprecate and remove the defined() function entirely. Given the complete lack of alternatives, this is a disturbing illustration of just how out of touch they are from the unfortunate realities of practical Puppetry.

Use virtual resources This is yet another variant of the common class technique that would require everyone, everywhere, to dance to the same tune. It has all the same problems, and hence gets a big thumbs-down from me.

Use singleton_packages This module is a valiant attempt to work around the problem, but practically speaking is no better than using defined() because again, if a module that isn t using singleton_packages specifies package "gcc": ensure => absent , you re going to end up with assplosion. The mandatory dependency on hiera doesn t win me over, either (I m a hiera-skeptic, for reasons which I might go into some other time).

Use only modules from a single source This is the solution that I use myself, if that s any recommendation As a result of pretty much every publically-available Puppet module sucking for one reason or another, I do not, currently, have any externally-written Puppet modules in the trees that I use. Every module (158, at current count) has been written by someone in the sysadmin team at $DAYJOB, to our common standards. This means that I can coordinate use of package resources, using our module of common classes where appropriate, or refactor modules to separate concerns appropriately. If you re wondering why we don t have all of these 158 modules available publically, well, we have a few of them up, but yes, the vast majority of them aren t publically available. Some of them suck mightily, while many others are just too intertwined with other modules to be able to use on their own, and we don t want to release crap code there s far too much of that out there already.

In a comment to yesterday s post on why your Puppet module sucks, Warren asks what can be done about the problem of multiple modules needing to include the same package (gcc being the most common example I m yet to come across). As I stated previously, there is no sane, standardised way for multiple independent modules to cooperate to ensure that certain packages are installed for their cooperative use. If it wasn t already obvious, I consider this to be a significant failing in Puppet, and one which essentially renders any attempt at public, wide-scale reuse of Puppet modules impossible. Software packages are such a fundamental part of system management today that it is rare for a module not to want to interact with them in some way. Without strong leadership from Puppet Labs, or someone core to the project who s willing to be very noisy and obnoxious on the subject, the issue is fundamentally unsolveable, because it requires either deep (and non-backwards-compatible) changes to how Puppet works at a core level, or it needs cooperation from everyone who writes Puppet modules for public consumption (to use a single, coordinated approach to common packages). The approaches that I ve seen people advocate using, or that I ve considered myself, roughly fall into the following camps.

Put your package into a class in your module Absofrickinglutely useless, because the elementary package resources you end up creating will still conflict with other modules package resources. Anyone who suggests this can be permanently ignored, as they don t understand the problem being solved.

Use a globally common module of classes representing common packages This could work, if you could get everyone who writes modules to use it (see someone willing to be very noisy and obnoxious ). Practically speaking, anyone who suggests this doesn t understand human nature. I don t see this actually happening any time soon.

Use the defined() function everywhere By wrapping all your package resources in if !defined(Package["foo"]) blocks, you can at least stop your module from causing explosions. What it doesn t do is make sure that the various definitions of the package resource are congruent (imagine if one was package "gcc": ensure => absent ). In order to safely avoid explosion, everyone would have to follow this approach, which (in the worst case) reduces the problem to the globally common module of classes. However, it s the least-worst approach I can practically consider. At least your module won t be the cause of explosions, and realistically no module s going to ask for gcc to be removed (or, if they are, you ll quickly find them and kill them). I ve seen calls from Puppet core dev team members to deprecate and remove the defined() function entirely. Given the complete lack of alternatives, this is a disturbing illustration of just how out of touch they are from the unfortunate realities of practical Puppetry.

Use virtual resources This is yet another variant of the common class technique that would require everyone, everywhere, to dance to the same tune. It has all the same problems, and hence gets a big thumbs-down from me.

Use singleton_packages This module is a valiant attempt to work around the problem, but practically speaking is no better than using defined() because again, if a module that isn t using singleton_packages specifies package "gcc": ensure => absent , you re going to end up with assplosion. The mandatory dependency on hiera doesn t win me over, either (I m a hiera-skeptic, for reasons which I might go into some other time).

Use only modules from a single source This is the solution that I use myself, if that s any recommendation As a result of pretty much every publically-available Puppet module sucking for one reason or another, I do not, currently, have any externally-written Puppet modules in the trees that I use. Every module (158, at current count) has been written by someone in the sysadmin team at $DAYJOB, to our common standards. This means that I can coordinate use of package resources, using our module of common classes where appropriate, or refactor modules to separate concerns appropriately. If you re wondering why we don t have all of these 158 modules available publically, well, we have a few of them up, but yes, the vast majority of them aren t publically available. Some of them suck mightily, while many others are just too intertwined with other modules to be able to use on their own, and we don t want to release crap code there s far too much of that out there already.

A new maintenance release 0.10.6 of Rcpp is now on the CRAN network for GNU R; binaries for Debian have been uploaded as well. This version ties up a number of smaller loose ends, but also adds a few new things, particularly John's new exposeClass. Details are as usual in the NEWS extract below, and also in the ChangeLog file in the package and on the Rcpp Changelog page.

Changes in Rcpp version 0.10.6 (2013-10-27)

• Changes in Rcpp API:
  • The function exposeClass takes a description of the constructors, fields and methods to be exposed from a C++ class, and writes C++ and R files in the package. Inherited classes can be dealt with, but require data type information. This approach avoids hand-coding module files.
  • Two missing is<>() templates for CharacterVector and CharacterMatrix have been added, and some tests for is_na() and is_finite() have been corrected thanks to Thomas Tse.
• Changes in R code:
  • Export linking helper function LdFlags as well as RcppLdFlags.
  • Function Rcpp.package.skeleton() no longer passes a namespace argument on to package.skeleton()
• Changes in R setup:
  • Raise requirement for R itself to be version 3.0.0 or later as needed by the vignette processing
• Changes in Rcpp attributes:
  • sourceCpp now correctly binds to Rtools 3.0 and 3.1

Thanks to CRANberries, you can also look at a diff to the previous release 0.10.5. As always, even fuller details are on the Rcpp Changelog page and the Rcpp page which also leads to the downloads, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Sometimes building the same code on multiple architectures is useful to detect horrors like:

ExEnv::err0() < < sprintf("AtomInfo: invalid name: %s\n",name.c_str());
This code builds using -Werror=format-security with only a few warnings on most architectures, while GCC is correctly detects the issue on some others. This has been reported as bug#728249.

FOSDEM is one of the largest gatherings of Free Software contributors in the world and happens each February in Brussels (Belgium). One of the tracks will be the Desktops DevRoom (formerly known as CrossDesktop DevRoom ), which will host Desktop-related talks. We are now inviting proposals for talks about Free/Libre/Open-source Software on the topics of Desktop development, Desktop applications and interoperability amongst Desktop Environments. This is a unique opportunity to show novel ideas and developments to a wide technical audience. Topics accepted include, but are not limited to: Enlightenment, Gnome, KDE, Unity, XFCE/Razor, Windows, Mac OS X, general desktop matters, applications that enhance desktops and web (when related to desktop). Talks can be very specific, such as developing mobile applications with Qt Quick; or as general as predictions for the fusion of Desktop and web in 5 years time. Topics that are of interest to the users and developers of all desktop environments are especially welcome. The FOSDEM 2013 schedule might give you some inspiration. Please include the following information when submitting a proposal:

• Your name
• The title of your talk (please be descriptive, as titles will be listed with around 250 from other projects)
• Short abstract of one or two paragraphs
• Short bio (with photo)
• Requested time: from 15 to 45 minutes. Normal duration is 30 minutes. Longer duration requests must be properly justified. You may be assigned LESS time than you request.

The deadline for submissions is December 14th 2013. FOSDEM will be held on the weekend of 1-2 February 2014. Please use the following website to submit your proposals: https://penta.fosdem.org/submission/FOSDEM14 You can also join the devroom s mailing list, which is the official communication channel for the DevRoom: desktops-devroom@lists.fosdem.org (subscription page for the mailing list) The Desktops DevRoom 2014 Organization Team

Dear all, we are proud to announce the 3.0 release of the Shogun Machine-Learning Toolbox. This release features the incredible projects of our 8 hard-working Google Summer of Code students. In addition, you get other cool new features as well as lots of internal improvements, bugfixes, and documentation improvements. To speak in numbers, we got more than 2000 commits changing almost 400000 lines in more than 7000 files and increased the number of unit tests from 50 to 600. This is the largest release that Shogun ever had! Please visit http://shogun-toolbox.org/ to obtain Shogun. News Here is a brief description of what is new, starting with the GSoC projects, which deserve most fame:

• Gaussian Process classification by Roman Votjakov
• Structured Output Learning of graph models by Shell Hu
• Estimators for log-determinants of large sparse matrices by Soumyajit De
• Feature Hashing and random kitchen sinks by Evangelos Anagnostopoulos
• Independent Component Analysis by Kevin Hughes
• A web-based demo framework by Liu Zhengyang
• Metric learning with large margin nearest neighbours by Fernando Iglesias
• Native support for various popular file formats by Evgeniy Andreev

Screenshots Everyone likes screenshots. Well, we have got something better! All of the above projects (and more) are now documented in the form of IPython notebooks, combining machine learning fundamentals, code, and plots. Those are a great looking way that we chose to document our framework from now on. Have a look at them and feel free to submit your use case as a notebook! The web-demo framework has been integrated into our website, go check them out. Other changes We finally moved to the Shogun build process to CMake. Through GSoC, added a general clone and equals methods to all Shogun objects, and added automagic unit-testing for serialisation and clone/equals for all classes. Other new features include multiclass LDA, and probability outputs for multiclass SVMs. For the full list, see the NEWS. Workshop Videos and slides In case you missed the first Shogun workshop that we organised in Berlin last July, all of the talks have been put online. Shogun in the Cloud As setting up the right environment for shogun and installing it was always one of the biggest problems for the users (hence the switching to CMake), we have created a sandbox where you can try out shogun on your own without installing shogun on your system! Basically it's a web-service which give you access to your own ipython notebook server with all the shogun notebooks. Of course you are more than welcome to create and share your own notebooks using this service! *NOTE*: This is a courtesy service created by Shogun Toolbox developers, hence if you like it please consider some form of donation to the project so that we can keep up this service running for you. Try shogun in the cloud. Thanks The release has been made possible by the hard work of all of our GSoC students, see list above. Thanks also to Thoralf Klein and Bj rn Esser for the load of great contributions. Last but not least, thanks to all the people who use Shogun and provide feedback. S ren Sonnenburg on behalf of the Shogun team (+ Viktor Gal, Sergey Lisitsyn, Heiko Strathmann and Fernando Iglesias)

Shogun goes cloud. Try out http://cloud.shogun-toolbox.org to interactively play with machine learning algorithms or try any of the interactive demos. The SHOGUN Machine Learning Toolbox is a collection of algorithms designed for unified learning for a broad range of feature types and learning settings, like classification, regression, or explorative data analysis. For more information visit http://www.shogun-toolbox.org.