DebConf 13 report (by Andreas Tille) General impression I'm beginning my DebConf report in an unofficial "Scenic Hacklab" right at the edge of the lake in Yverdon. This is the right place to memorise the last days. When I started from this place cycling to Le Camp 12 days ago I was full of great expectations and what should I say - the reality has even beaten these. Once it comes about comparing DebConfs even if it is an unfair comparison due all the differences my secret long term favourite was Helsinki very closely followed by Argentina and also very closely followed by all the other great DebConfs I joined (and I joined all in Europe). Would Le Camp be able to beat it? The short answer is: Yes, it is now my favourite DebConf while I think I do not suffer from the last-Debconf-was-the-best-DebConf-syndrome (and I realised there are others thinking the same). As you might probably know I'm a bit addicted to swimming. While Helsinki had admittedly the better conditions I was at least able to fix the distance issue using my bicycle. (Hey, those Le Camp photographers did a great job in hiding the fact that you can not actually touch the lake right from the meadow of Le Camp.) Being able to have my bicycle at DebConf scored some extra points. However, the really great view of the lake, the inspiring "Scenic Hacklab" which was my favourite place has bumped DebConf13 at first place in my personal ranking. So it comes quite natural to say: "Kudos to the great organisation team!" They did a Swiss-like precise work and perfectly succeeded in hiding any problems (I assume there were some as always) from the attendees so everything went smooth, nice and shiny for the attendees. The local team was even precise in setting up great weather conditions for DebConf. While saying thanks to the local team I would like to also explicitly thank Luca Capello who has quite some share that this DebConf was possible at all (while I have to decrease my DebConf score one point because he was not really there - Luca to bad that you were not able to come full time!) Also thanks to Gunnar and Gannef who helped remotely (another score down because I were missing them this year as well). Even if it was my favourite DebConf I was not able to work down my todo list fully (which was not only uploading one package per day which I at least statistically fullfilled). But that's probably a general feature of todo lists anyway. One item was definitely done: Doing my daily swimming BoF. I actually was able to do the other parts of the triathlon which was skipped by Christian and have done in summary about 150km cycling with 3500m elevation and estimated 7-8km swimming (0m elevation ;-)). Considering the great view at sunrise over the lake I was not hating my "Senile bed escape" disease too much (I was every day waking up at sunset) - it was simply a great experience. I will never forget seeing water drips glimmering like gold inside the morning sun while seeing the Alps panorama in the distant. I hope I was able to help all interested swimmers with the DebConf Beach Map which was just a by-product of my activities in DebCamp. Speaking about OSM: I was astonished that the area was way less covered than I expected. Thanks to several DebConf attendees the situation became better and the map does not only show random trees in the wild but also the tracks leading to these. (Remark: It was no DebConf attendee who is responsible for plastering the map with single trees.) While I had my mapping focus basically close to the edge of the lake I was also able to even map my very own street. :-) I clearly remember one specific mapping tour when I was invited by the DPL: He convinced me to join him on a bicycle tour and since I was afraid to get fired I joined him instead to keep on hacking. Also Sorina was brave enough to join us on the tour and she did quite well. (Sorina, do you remember the agreement about your work on the installer? ;-)) Lucas described the tour as: going uphill on only asphalted roads. Sorina and me were witnessing the mighty DPL powers when we left the wood around Le Camp to reach the described road: The asphalt was just put onto the road - no doubt that it was done on the immediate demand of mighty DPL. :-) DebCamp time was flying like nose dive and a lot of known (and unknown) faces arrived at Le Camp. What I really liked a lot this year was that several really young children has pulled down the average age of DebConf attendees. I clearly remember all the discussion one year ago what to do about children. As always the issue was solved in a typical Debian way: Just do it and bring your children - they had obviously a great time as well. I think the youngest child was 2 months and the oldest "child" above 20. ;-) Actually Baptiste Perrier did great in making the C&W party a success and had obviously a nice time. (I wished my son would have been able to come as well but he needs to write his bachelor s thesis in physics. :-() It was nice to see the kids using all playing facilities and communicating with geeks. Also I would like to point out that even the very young attendees had their share at the success of DebConf: Just think of the three "bell ringing assistants" who helped me ringing the bells for lunch and dinner. I've got this cool job from Didier in the beginning of DebCamp. I must say having some real bells ringing is by far nicer than just the "lunch / dinner starts in 10 minutes" from IRC bot. The only thing I did not understand was that people did not considered ringing the bells at 8:00 for breakfast as a good idea. Regarding the food in general I would also like to send kudos to the kitchen: It was tasty, freshly prepared, regional food with a good change rate. I really liked this. Extra points for having the chance to sit outside when eating. Talks But lets have a look into the conference programme. I'd really recommend watching the videos of the talks Bits from the DPL (video) and Debian Cosmology (video). I considered both talks as entertaining and interesting. I also really hope that the effort Enrico Zini started in Debian Contributors (video) will be successful. I had some talks and BoFs myself starting with Why running a Blend (video) and I admit that (as usual) the number of attendees was quite low even if I think there is some proof (see below) that it is interesting for way more people who should consider working more "blendish" in their team. Do you know how to recruit one developer per year and relax the man power problem in your team? Feel free to watch the video. We have confirmation that ten DDs of our team have considered to join Debian only because Debian Med exists. Admittedly biology and medicine are really leaf topics inside the Debian universe. So if even this topic that has a very tiny share of the Debian users is able to attract this level of attention - how many more people could we win for multimedia, games, GIS and others? So if you feel you are quite overworked with your packaging and you have no time this is most probably wrong. The amount of time is basically a matter of priorities you set for your tasks. Try to put some higher priority onto using the just existing Blends tools I explained in my talk to attract more users and developers to your team and by doing so spread the workload over more people. It works, the prove was given in my main talk. So before you start working on a specific package you should wonder who else could have an even stronger interest to get this work done and provide him with some additional motivation and help to get the common goal done. The interesting thing is that my BoF about How to attract new developers for your team (video) - which was a simple report about some by-product of the Blends work - made it into the main talk room and got way more attention. For me this is the proof that the Blends concept itself is probably badly perceived as something like "a few outsiders are doing damn specific stuff which is not really interesting for anybody else" instead of what is really is: Smoothing the way from specific upstream applications to the end user via Debian. Once you see the video of this BoF you can observe how my friend Asheesh Laroia became more and more excited about the Blends concept and admitted what I said above: We should have more Blends for different fields. Funnily enough Asheesh asked me in his excitement to talk more about Blends. This would have been a really good suggestion ten years ago. At DebConf 3 in Oslo I had my very first talk about Blends (at this time under the name "Debian Internal Projects"). I continuously kept on talking about this (MiniDebConf Peking 2005, DebConf 5, Helsinki (video), DebConf 7, Edinburgh (video), DebConf 8, Mar del Plata (video), DebConf 9, C ceres (video), MiniDebConf Berlin 2010 (video in German), MiniDebConf Paris 2010 (not video recorded), DebConf 11, Banja Luka (video) ... and these are only (Mini)DebConfs my talks page is full of this topic) and every new year I try different ways to communicate the idea to my fellow Debianistas. I'm wondering how I could invent a title + abstract avoiding the term Blends, put "Git", "release" and "systemd versus upstart" in and being able to inform about Blends reasonably by not becoming to off topic with the abstract. I also registered the Debian Science round table. I admit we were lacking some input from remote via IRC which used to be quite helpful in the past. The attendees agreed upon the handling of citations in debian/upstream files which was invented by Debian Med team to create even stronger bounds to our upstream developers by giving their work extra reward and providing users with even better documentation (see my summary in Wiki). As usual I suggested to create some Debian Science offsprings like "Debian Astronomy", "Debian Electronics", "Debian Mathematics", "Debian Physics" etc. who could perfectly leave the Debian Science umbrella to get a more fine grained structure and a more focused team to enhance the contact to our users. Unfortunately there is nobody who volunteers to take over the lead for such Blends. I have given a short summary about this BoF on the Debian Science mailing list. In the Debian Med meeting I have given some status report. No other long term team members were attending DebConf and so I gave some kind of introduction for newcomers and interested people. I touched also the DebiChem topic which maintains some packages that are used by biologists frequently and so we have a good connection to this team. Finally I had registered three BoFs in Blends I'm actually not (or not yet) active part of. My motivation was to turn the ideas I have explained in my main talk into specific application inside these teams and helping them to implement the Blends framework. In the first BoF about Debian GIS I have shown the usual team metrics graphs to demonstrate, that the one packaging team Pkg-OSM is in danger to become MIA. There are only three persons doing actual uploads. Two of them were at DebConf but did not joined the BoF because they do not consider their contribution to Pkg-OSM as a major part of their general Debian work. I will contact the main contributor David Paleino about his opinion to move the packages step by step into maintenance of Debian GIS packaging team to try to overcome the split of two teams that are sharing a good amount of interest. At least if I might become an Uploader for one of the packages currently maintained by Pkg-OSM I will move this to pkg-grass-devel (which is the name of the packaging team of Debian GIS for historical reasons). The attendees of the BoF have considered this plan as sensible. Moreover I talked about my experiences with OSGeo Live - an Ubuntu derivative that tries to provide a full tool chain to work on GIS and OSM problems ... basically the same goal as Debian GIS has just provided by the OSGeo project. I'm lurking on OSGeo mailing list when I asked explicitly I've got the answer that they are working together with Debian GIS and are using common repository (which is IMHO the optimal way of cooperation). However, it seems that several protagonists of OSGeo Live are underestimating the resources provided by Debian. For instance there was a question about Java packaging issues but people were not aware about the existence of the debian-java mailing list. I was able to give an example how the Debian Med team managed to strengthen its ties to BioLinux that is also an Ubuntu derivative for biologists. At our first Debian Med sprint in 2011 we invited developers from BioLinux and reached a state where they are using the very same VCS on Alioth where we are maintaining our packages. At DebConf I was able to upload two packages where BioLinux developers did certain changes for enhancing the user experience. My "work" was just bumping the version number in changelog and so we did profit from the work of the BioLinux developers as well as they are profiting from our work. I plan to dive a bit more into Debian GIS and try to strengthen the connection to OSGeo Live a bit. The next BoF was the Debian Multimedia meeting. It was nice that the current leader of Ubuntu Studio Kaj Ailomaa joined the meeting. When I was explaining my ideas about cooperation with derivatives I repeated my detailed explanation about the relation with BioLinux. It seems every topic you could cover inside Debian has its related derivative. So to me it seems to be quite natural to work together with the developers of the derivative to join forces. I actually consider a Blend a derivative done the right way = inside Debian. The final work for the derivers that might be left for them is doing some shiny customising of backgrounds or something like this - but all the hard work could and should be done in common with the relevant Debian team. My dream is to raise such relevant teams inside Debian ... the Blends. Finally the last BoF of this series was the Debian Games meeting. As always I presented the team metrics graphs and the Debian Games team members who attended the BoF were quite interested. So it seems to be some unknown fact that team metrics are done for several teams in side Debian and so I repeat the link to it for those who are not yet aware of it. As a result of the BoF Debian Games team members agreed to put some more effort into maintaining their Blends tasks. Moreover Miriam Ruiz wants to put some effort into reviving Debian Jr. Regarding Debian Jr. there was an interesting talk about DouDouLinux - in case you might want to watch the video I'd recommend skipping the first 30min and rather watch the nice live demo. There was also an ad hoc BoF about Debian Jr scheduled to bring together all people interested into this cute project and Per Anderson volunteered to take over the lead. I have given a summary about this specific BoF at the Debian Jr list. For some other talks that I'd regard as remarkable for some reasons: I'd regard the talk "Debian-LAN" by Andreas Mundt as some hidden pearl because it did not got a lot of attention but after having seen the video I was quite impressed - specifically because it is also relevant for the Blends topic. Memories I also liked "Paths into Debian" by Moray Allan (and I was only able to enjoy the latter talks thanks to the great work of the video team!) because it also scratched the same topic I was concerned about in my mentoring talk. Related to this was in my opinion also "Women in Debian 2013" were we tried to find out reasons for the lack of woman compared to other projects and how to overcome this issue. Besides the talks I will probably never forget two specific moments that make DebConf so special. One of these moments is recorded on an image that clearly needs no words - just see Geert hovering over the grass. Another strong moment in my personal record was in the DebConf Newbies BoF "First time at DebConf" that unfortunately was not recorded but at least for this statement it would have been very great if we would have some reference better than personal memory. Aarsh Shah a GSoC student from India suddenly raised up and said: "Four months ago I was not even aware that Free Software exists. Now I'm here with so many people who are totally equal. If I will tell my mother at home that I was standing in the same queue where the Debian Project Leader was queuing up for food she will never believe me." He was totally excited about things we are regarding as normal. IMHO we should memorise moments like this that might be part of the key to success in cultures, where Debian is widely unknown and very rarely in use. Amongst these not scheduled great moments the scheduled day trip was also a great thing. I had a really hard time to decide what tour I might join but ended up in the "long distance walking (or should I say running) group". Inspired by the "running Bubulle" who was flashing between the walking groups we went uphill with 5.4km/h which was a nice exercise. Our destination the large cliff was an exciting landscape and I guess we all enjoyed the dinner organised by the "Trout cabal". ;-) So I had a hard time to leave Le Camp and tried hard to make sure my memories will remain as long as possible. Keeping some signs attached to my bicycle, conserving the "Scenic Hacklab" sign for my private "scenic hacklab @ home" was one part. I also have cut some branches of the Buxus sempervirens in Le Camp and have put them in my garden at home (where I create some hedgerow from places where I spent some great time). These will probably build a great part of the hedgerow ... Thanks for reading this longish report. Looking forward to see you all in Germany 2015 (or earlier) Andreas.

A couple of days ago, the upstream maintainer of autodie, Paul Jamieson Fenwick (PJF), released autodie 2.21. It includes all of the performance improvements I had proposed in Optimizing autodie (plus a few extra ones authored after I wrote the post) and a small fix to one of the tests (due to changes in Carp). As mentioned in my earlier post, these improvements gives about a factor 4 load time improvement in most cases. For the sake of it, I have decided to use Lintian as an example of how good this improvement is. First off, Lintian loads uses autodie in about 75 unique files in the current git master branch.

$ grep -lr 'use autodie' lib/ checks/ collection/ frontend/lintian   wc -l
75

As a simple benchmark, lets consider the following Perl one-liner:

$ /usr/bin/time -f "%es %Us %Ss" perl -Mautodie \
  -e 'print $autodie::VERSION, " => ";' \
  -e 'eval join("\n", map  "package X$_; use autodie;"  (1..74));'

A quick-n-dirty comparison gives:

2.20 => 0.91s 0.90s 0.01s
2.21 => 0.15s 0.15s 0.00s
# NB: the times are "wall, user and sys" in that order, in case you
# are too lazy to read time(1).   

So, 2.21 is now closer to a factor 6 rather than the original factor 4. I have to admit that the timings seem to be very machine dependent as my laptop shows a factor 7 improvement (from ~1.39 to ~0.22s). With at least 370 tests cases running in the Lintian testsuite, I can look forward to a reduction in user time on about 4 minutes.

Yesterday, I pushed my first sizable contribution to GHC, the Haskell compiler. The feature solves the problem that newtypes are not always free: If we have newtype Age = MkAge Int, then we have all learned that the Age function has zero run-time cost. But if the Int that we want to convert is inside another type, the conversion is no longer free: Converting a Maybe Int to a Maybe Age using, for example, fmap Age, will cause time and space overhead at runtime, and there was no way around it. Well, there is unsafeCoerce, but really, that ought to be avoided. So after some discussion with and encouragement of Simon Peyton Jones at RDP in Eindhoven this year I worked on a design (which was developed, as far as I know, by Simon, Roman Cheplyaka, Stephanie Weirich, Richard Eisenberg and me). In GHC 7.8, there will be a function coerce :: Coercible a b => a -> b that works, from the user point of view, like unsafeCoerce (i.e. no run-time cost), but with the big difference that it will only typecheck if the compiler can infer that it indeed is safe to coerce between a and b. So it will derive Coercible Age Int, and Coercible Int Age, and Coercible (Maybe Age) (Maybe Int) and even stuff like Coercible (Int -> Age) (Age -> Int), but not Coercible Int Bool. It will also not coerce between Age and Int if the constructor MkAge is not exported, to respect module boundaries. Under the hood this relies on the also new feature of roles, which were solved to make the previously unsafe GeneralizedNewtypeDeriving feature safe again, and which also guarantee that coerce is indeed as safe as the name suggests. The feature will come with 7.8, but not fully advertised , so things might change again for 7.10, and bugs with the feature may not necessarily qualify to be fixed in further 7.8.x releases, so beware. It also does not automatically convert fmap Age into coerce, but it is a step in that direction.

I signed up to take part of an online, massive course Arte y cultura en circulaci n: crear y compartir en tiempos digitales (The circulation of arts and culture: Creating and sharing in digital times). As the activities of us the attendees are to be published by each on their personal blog/space, I will be publishing them here. I hope they are interesting to some of my other regular readers. I am late in joining, and I should already be posting the second activity. Anyway, this text goes towards the first week's lesson: Qu es un autor: la (de)construcci n hist rica del concepto de autor a (What is an author? Historic (de)construction of the concept of authorship). Our homework is to find an example of a current discussion where the notion of authorship is discussed, and share it this way. And I'm somewhat in a hurry right now, so this will be hasty. But I didn't want to delay my (late) submission anymore! In the Debian Ruby mailing list, Hleb Valoshka asks basically how deep should copyright recognition be. Because, yes, while copyright attribution is simple (simpler, at least) in literary / artistic works, computer programs (and even more those developed following the Free Software distributed fashion) are harder to properly attribute. Is a two-line patch to a tens-of-thousands-of-lines project enough to warrant mention in the copyright file? Most of us would agree it is not. Few would contend this were the amount of changes to make up, say, even 1% of that scale of project. In the coding projects where I am the main author (which, yes, are usually very small I cannot recognize myself as a good programmer, and the size of my successful works proves me), I try to take care to mention explicitly each of the contributors, even if their contribution is quite small. But were I to lead a larger development work, with enough following to generate on the order of one submitted patch a day, would it make sense for me to follow up with such detail the authorship information? And as a minor side note: How much does the law require me? A very small patch can fix important functionality issues (think security or performance). If somebody sends a very small patch fixing an intellectually hard to grasp problem... I am sure it should be properly acknowledged! Anyway, I have to flee now. I am just dropping some ideas on the table. Of course, if anybody is willing to discuss them further, I'm interested in any debate that springs off it! (Meanwhile... I expect to spam you with this topic a couple of times. But it should not bug you much, as it's one of my usual blog topics!)

Many thanks to Vincent Meurisse who solved my node.js callback woe. Some history of the blogspam service:

Back in 2008 I was annoyed by the many spam-comments that were being submitted to my Debian Administration website. I added some simple anti-spam measures, which reduced the flow, but it was a losing battle. In the end I decided I should test comments, as the users submitted them, via some kind of external service. The intention being that any improvements to that central service would benefit all users. (So I could move to testing comments on my personal blog too, for example). Ultimately I registered the domain-name "blogspam.net", and set up a simple service on it which would test comments and judge them to be "SPAM" or "OK". The current statistics show that this service has stopped 20 million spam comments, since then. (We have to pretend I didn't wipe the counters once or twice.)

I've spent a while now re-implementing most of the old plugins in node.js, and I think I'll be ready to deploy the new service over the weekend. The new service will have to handle two different kinds of requests:

New Requests

These will be submitted via HTTP POSTed JSON data, and will be handled by node.js. These should be nice and fast.

Legacy Requests

These will come in via XML-RPC, and be proxied through the new node.js implementation. Hopefully this will mean existing clients won't even notice the transition.

I've not yet deployed the new code, but it is just a matter of time. Hopefully being node.js based and significantly easier to install, update, and tweak, I'll get more contributions too. The dependencies are happily very minimal:

• A redis-server for maintaining state:
  • The number of SPAM/OK comments for each submitting site.
  • An auto-expiring cache of blacklisted IP adddresses. (I cache the results of various RBL results for 48 hours).
• node.js

The only significant outstanding issue is that I need to pick a node.js library for performing CIDR lookups - "Does 10.11.12.23 lie within 10.11.12.0/24?" - I'm surprised that functionality isn't available out of the box, but it is the only omission I've missed. I've been keeping load & RAM graphs, so it will be interesting to see how the node.js service competes. I expect that if clients were using it, in preference to the XML-RPC version, then I'd get a hell of a lot more throughput, but with it hidden behind the XML-RPC proxy I'm less sure what will happen. I guess I also need to write documentation for the new/preferred JSON-based API... https://github.com/skx/blogspam.js

Here s some documentation about a hack I ve been working on to allow publication of RDF views for Tastypie resources (in Django applications). While working on implementing support for RDF meta-data for descriptions of Debian packages for the Debian Package Tracking System rewrite, I ve tried and idetify which libraries/frameworks would allow to create some RDF views for Django model objects with a minimal effort. Ideally, this would save the hassle of writing code, and could just be a matter of mapping some Django model fields to proper ontology attributes. I haven t found an existing tool to do so, but it seemed to me that Tastypie could offer a nice starting point. Tastypie (I focused on v. 0.9.15 which is currently in Debian testing) offers some REST content-negociation support, and other niceties. This post is an attempt at documenting an initial implementation for my problem. I ve implemented it as some code in the example blog application described in the Tastypie documentation. Unfortunately, it s not yet a patch that could be applied to Tastypie, to add this as a standard feature. The code is available in my git clone of Tastypie, based on 0.9.15, and lies in the commit(s) between the example_myapp and rdfviews branches. It needs an up to date RDFLib to work (Debian s is too old, btw). It basically relies on the addition of a _rdf_mapping dict in the Django model objects, and an RdfModelResource as a base class for Tastypie ModelResources. To provide an RDF view for a ModelResource, it also requires to declare a particular MetaClass, and adding a few bits to its Meta subclass. The principle is that the dehydratation steps of the Tastypie system will replace fields values by their RDF objects counterparts, as either RDFLib Literals or URIRefs. Then the resuting Bundle will be converted to some RDFLib Graph. It is then just a matter of serialization of that Graph to turtle (I ve not added another format, but it should be pretty straightforward). There s a bit of hackish drak magic in the code, in particular when introspecting the Tastypie / Django resource / model objects to be able to dynamically add some dehydrate_FOO methods to perform the conversion to Literals or URIRefs. Custom dehydrate_Foo methods can still be written to process the fields in particular ways. I ve added some LDP style paging along the way, so I hope it will be quite usable by LDP compatible tools. Here s the result : First, listing all resources : $ curl -H 'Accept: text/turtle' "http://localhost:8000/api/v1/entry/"
@prefix dcterms: <http://purl.org/dc/terms/> .
@prefix ldp: <http://www.w3.org/ns/ldp#> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix sioc: <http://rdfs.org/sioc/ns#> .
@prefix xml: <http://www.w3.org/XML/1998/namespace> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> . </api/v1/entry/#list> dcterms:hasPart </api/v1/entry/1/#post>,
</api/v1/entry/2/#post>,
</api/v1/entry/3/#post> . Then, details of one particular post : $ curl -L -H 'Accept: text/turtle' "http://localhost:8000/api/v1/entry/1"
@prefix dcterms: <http://purl.org/dc/terms/> .
@prefix owl: <http://www.w3.org/2002/07/owl#> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix sioc: <http://rdfs.org/sioc/ns#> .
@prefix xml: <http://www.w3.org/XML/1998/namespace> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> . </api/v1/entry/1/#post> a sioc:Post ;
dcterms:created "2011-05-21T22:46:38+00:00"^^xsd:dateTime ;
dcterms:title "Another Post" ;
sioc:content "MESSAGE: This will prbbly be my lst post. /MESSAGE" ;
sioc:has_creator </api/v1/user/1/> ;
owl:sameAs </posts/another-post#post> .

I hope this is usable, and welcome any feedback.

Today I started hacking on a re-implementation of my BlogSpam service - which tests that incoming comments are SPAM/HAM - in node.js (blogspam.js) The current API uses XML::RPC and a perl server, along with a list of plugins, to do the work. Having had some fun and success with the HTTP+JSON mstore toy I figured I'd have a stab at making BlogSpam more modern:

• Receive a JSON body via HTTP-POST.
• Deserialize it.
• Run the body through a series of Javascript plugins.
• Return the result back to the caller via HTTP status-code + text.

In theory this is easy, I've hacked up a couple of plugins, and a Perl client to make a submission. But sadly the async-stuff is causing me .. pain. This is my current status:

shelob ~/git/blogspam.js $ node blogspam.js
Loaded plugin: ./plugins/10-example.js
Loaded plugin: ./plugins/20-ip.js
Loaded plugin: ./plugins/80-sfs.js
Loaded plugin: ./plugins/99-last.js
Received submission:  "body":"This is my body ..","ip":"109.194.111.184","name":"Steve Kemp" 
	plugin 10-example.js said next :next
	plugin 20-ip.js said next :next
	plugin 99-last.js said spam SPAM: Listed in StopForumSpam.com

So we've loaded plugins, and each has been called. But the end result was "SPAM: Listed .." and yet the caller didn't get that result. Instead the caller go this:

shelob ~/git/blogspam.js $ ./client.pl
200 OK 99-last.js

The specific issue is that I iterate over every loaded-plugin, and wait for them to complete. Because they complete asynchronously the plugin which should be last, and just return "OK" , has executed befure the 80-sfs.js plugin. (Which makes an outgoing HTTP request). I've looked at async, I've looked at promises, but right now I can't get anything working. Meh. Surprise me with a pull request ;)

It has been nearly six years with a netbook and five since I last wrote about wifi roaming from the bus to stay on irc without a costly celluar link during the daily commute. Since then, some readers have asked me to share my refinements to the method in a followup post. So here it is. The software On the server:

• openssh-server
• screen
• irssi

On the client:

• screen
• wpasupplicant
• isc-dhcp-client
• openssh-client
• openbox
• sudo & gksudo (optional)
• urxvt
• wavemon (optional)
• three shell scripts (provided below)

Putting it together: on the client Make sure if you have a wireless manager installed (such as NetworkManager) it is configured to skip your wireless interface, disabled entirely, or if possible, removed. Set up /etc/wpa_supplicant/wpa_supplicant.conf and /etc/network/interfaces for roaming, as per the instructions in /usr/share/doc/wpasupplicant/README.modes.gz. Don t forget to add yourself to the netdev group if you are not in it already. In /etc/wpa_supplicant/wpa_supplicant.conf, list common names of open networks. Normally the catch-all network that associates with any essid, i.e. the first stanza below, works well. However, occasionally the strongest signal is neither one of the common networks nor an easily accessible network (e.g. web portals), so having a list of common open networks helps to quickly select from among those instead. The more you travel, the more of these will discover and add. Just use reconfigure from wpa_cli to reload your edited list each time you add a new one.

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
network= 
        key_mgmt=NONE
 
network= 
        ssid="default"
        key_mgmt=NONE
 
network= 
        ssid="linksys"
        key_mgmt=NONE
 
...

Since you ll be using ssh repeatedly to connect and it has to be fast, make sure your server is set up to accept your key and use ssh-add so that you only have to enter your ssh key password once. You can tweak isc-dhcp-client to make connections faster. In /etc/dhcp/dhclient.conf, use:

backoff-cutoff 1;
initial-interval 1;

Here are a few scripts I wrote to facilitate quick roaming from one open AP to another and reconnect to irssi running in screen, to break a connection and try the next one, and to recover from occasional lockups (more about that later). ~/bin/screen_reconnect This is a script to reconnect continuously via ssh to a screen session:

#!/bin/sh
reset
while ! ssh -t 10.9.8.7 'screen -UDr' 2>/dev/null ; do echo -n "." ; sleep .1 ; done

Just substitute the IP of your own server here. Using an IP instead of domain name makes the connection faster because a DNS lookup is not required. ~/bin/wifi_reassociate This script closes any open ssh sessions and informs wpa_supplicant to attempt to connect again.

#/bin/sh
/sbin/wpa_cli rea
killall ssh >/dev/null 2>&1

~/bin/wifi_killall This optional, somewhat ugly script addresses an issue I hope you never have. On my ASUS Eee PC 1001PX, occasionally scanning stops. When this happens, and I have never figured out why, apparently ACPI events are blocked. At this point wifi becomes unusable and ACPI sleep is inhibited. By trial and error I found that if you bring down the interface, kill all network-related processes, and bring it up again, ACPI events are unblocked and wifi is usable once more (and any pending request to sleep will finally happen). The script requires sudo, and to use the openbox key binding, gksudo.

#/bin/sh
sudo ifdown wlan0
# in case any of these are hung
sudo killall dhclient3
sudo killall wpa_cli
sudo killall wpa_action
sudo killall wpa_supplicant
# in case any of these are *really* hung
sleep 1
sudo killall -9 dhclient3
sudo killall -9 wpa_cli
sudo killall -9 wpa_action
sudo killall -9 wpa_supplicant
sudo dhclient -r
sudo ifup wlan0

Openbox Since certain actions need to be performed repeatedly and quickly, it is useful to have hotkeys bound in your window manager to the scripts. In ~/.config/openbox/rc.xml, key bindings for <alt>-r to reassociate and <alt>-d to disconnect a hung connection would look like:

  
<keyboard>
  <!-- My keybindings -->
  <keybind key="A-R">
    <action name="Execute">
        <execute>/home/synrg/bin/wifi_reassociate</execute>
    </action>
  </keybind>
  <keybind key="A-D">
    <action name="Execute">
        <execute>gksudo /home/synrg/bin/wifi_killall</execute>
    </action>
  </keybind>
</keyboard>

Putting it together: on the server There is very little to do here. Just start screen, and start irssi in screen. Running screen on the client as well as the server means you should either bind the screen meta keys to a different key sequence on each system, or else learn to press meta twice to pass through meta to the server screen as needed. I use the latter approach. Alternatively, you could use a tabbed terminal on the client, or separate terminals per client process instead of screen. This is a matter of personal taste. Ready to roam Here is a typical setup for roaming on the bus: In a terminal (I use urxvt), first ssh-add, then start screen with these three processes running in separate virtual terminals:

• /sbin/wpa_cli
• screen_reconnect
• wavemon (optional)

March of the dots Most of the commute, just enjoy watching the dots march by, waiting for a new connection. If you estimate a connection is unusable, press <alt>-r to reassociate immediately, giving the next network a chance. If the connection is already firmly established, this might not work on the first try. If the dots don t resume immediately, wait a bit and press it again. This might take a few tries. Changing selected networks on the fly Use wpa_cli when you need to do some fine-tuning of network selections on the fly. While normally you can just watch the march of the dots until a connection is acquired, sometimes you can improve your chances of connecting to a good network by manually controlling the selected candidate networks here. For example, by watching the speed of the bus relative to known good APs, you can predict which networks are more likely to succeed. Rather than connect to any arbitrary network, you might select a specific one by id, and then later when it goes out of range, revert to the original configuration, e.g.

> select_network 5
...
> reconfigure

You can use tab-completion in wpa_cli to type these commands quickly or else just abbreviate the commands. Another common scenario is when you pass through a business area with many captive portal hotspots. These rarely make good choices because they either require a password not known to you or else you can t click through I agree in time before the bus moves on. In this case, you might just disable the catch-all stanza and let the common open network stanzas you listed ( default , linksys , etc.) do the work:

> disable_network 1

Become a type ahead wizard While running, a continuous stream of periods fills the screen, which provides you with a highly visible cue that no available APs are in range. When the movement stops, you know a connection is being attempted. While waiting to connect, you can type ahead any comments you want to make in the current irssi window (taking care to remember which one you are in!) While having periods interspersed in what you type may be disorienting at first, you get used to it. There is a point when a connection is first established and ssh is accepting input, but anything you type can no longer be seen while you re typing. Depending on whether the connection was completely successful or not, what you type now may or may not finally be sent. For best results, only type ahead before the dots stop moving. Eventually you can become skilled enough at this to type ahead a comment in one channel, switch channels with /win # and continue typing ahead in the new channel, all buffered until the next few seconds (or even fraction of a second) of connection time. Fine-tune antenna direction with wavemon When the bus has come to a standstill, you may find wavemon useful to pull in a weak signal. Because wavemon has continuously updated signal level and link quality bars, you can use it to fine-tune the antenna position. Just turn your laptop until the bars are at their maximum. Captive portals I have not figured out how to do any automation for this, so it really is a crapshoot, as it is likely the bus has moved on by the time you ve managed to manually navigate the login through a captive portal. But in rush hour, you may have the luxury of time to connect to these as you pass them. I have recently learned about the CoovaFX Firefox plugin which automates logins to captive portals. I m going to give it a try to see if it helps. Update: I can t recommend this plugin, as it is not compatible with Iceweasel >= 23.0. Also, the standard it is based on, WISPr, appears to have an uncertain future. That, coupled with the fact that the plugin appears to not be open source means I m still looking for alternatives. Summary If all of this sounds a bit nuts to you, well, it probably is. But after half a decade enjoying free access to irc from the bus, it all seems perfectly natural to me! If you try this method and like it, please let me know in the comments. Likewise, if you have any improvements to the process or scripts, please share them!

When it comes to provide redundant services, several options are available:

• The service can be hosted behind a set of load-balancers. They will detect any faulty node. However, you need to ensure that this new layer is also fault-tolerant.
• The nodes providing the service can rely on IP failover to share a set of IP using protocols like VRRP¹ or CARP. The IP address of a faulty node will be assigned to another node. This requires all nodes to be part of the same IP subnet.
• The clients of the service can ask a third-party for available nodes. Usually, this is achieved through round-robin DNS where only working nodes are advertised in a DNS record. The failover can be quite long because of caches.

A common setup is a combination of those solutions: web servers are behind a couple of load-balancers achieving both redundancy and load-balancing. The load-balancers use VRRP to ensure redundancy. The whole setup is replicated to another datacenter and round-robin DNS is used to ensure both redundancy and load-balacing of the datacenters. There is a fourth option which is similar to VRRP but relies on dynamic routing and therefore is not limited to nodes in the same subnet:

• The nodes advertise their availability with BGP to announce the set of service IP addresses they are able to serve. Each address is weighted such that IP addresses are balanced among the available nodes.

We will explore how to implement this fourth option using ExaBGP, the BGP swiss army knife of networking, in a small lab based on KVM. You can grab the complete lab from GitHub. ExaBGP 3.2.5 is needed to run this lab.

Environment We will be working in the following (IPv6-only) environment:

BGP configuration BGP is enabled on ER2 and ER3 to exchange routes with peers and transits (only R1 for this lab). BIRD is used as a BGP daemon. Its configuration is pretty basic. Here is a fragment:

router id 1.1.1.2;
protocol static NETS   #  
  import all;
  export none;
  route 2001:db8::/40 reject;
 
protocol bgp R1   #  
  import all;
  export where proto = "NETS";
  local as 64496;
  neighbor 2001:db8:1000::1 as 64511;
 
protocol bgp ER3   #  
  import all;
  export all;
  next hop self;
  local as 64496;
  neighbor 2001:db8:1::3 as 64496;
 

First, in , we declare the routes that we want to export to our peers. We don t try to summarize them from the IGP. We just unconditonnaly export the networks that we own. Then, in , R1 is defined as a neighbor and we export the static route we declared previously. We import any route that R1 is willing to send us. In , we share everything we know with our pal, ER3, using internal BGP.

OSPF configuration OSPF will distribute routes inside the AS. It is enabled on ER2, ER3, DR6, DR7 and DR8. For example, here is the relevant part of the configuration of DR6:

router id 1.1.1.6;
protocol kernel  
   persist;
   import none;
   export all;
 
protocol ospf INTERNAL  
  import all;
  export none;
  area 0.0.0.0  
    networks  
      2001:db8:1::/64;
      2001:db8:6::/64;
     ;
    interface "eth0";
    interface "eth1"   stub yes;  ;
   ;
 

ER2 and ER3 inject a default route into OSPF:

protocol static DEFAULT  
  import all;
  export none;
  route ::/0 via 2001:db8:1000::1;
 
filter default_route  
  if proto = "DEFAULT" then accept;
  reject;
 
protocol ospf INTERNAL  
  import all;
  export filter default_route;
  area 0.0.0.0  
    networks  
      2001:db8:1::/64;
     ;
    interface "eth1";
   ;
 

Web nodes The web nodes are pretty basic. They have a default static route to the nearest router and that s all. The interesting thing here is that they are each on a separate IP subnet: we cannot share an IP using VRRP². Why are those web servers on different subnets? Maybe they are not in the same datacenter or maybe your network architecture is using a routed access layer. Let s see how to use BGP to enable redundancy of those web nodes.

Redundancy with ExaBGP ExaBGP is a convenient tool to plug scripts into BGP. They can then receive and advertise routes. ExaBGP does the hard work of speaking BGP with your routers. The scripts just have to read routes from standard input or advertise them on standard output.

The big picture Here is the big picture: Let s explain it step by step:

1. Three IP addresses will be allocated for our web service: 2001:db8:30::1, 2001:db8:30::2 and 2001:db8:30::3. Those are distinct from the real IP addresses of W1, W2 and W3.
2. Each web node will advertise all of them to the route servers we added in the network. I will talk more about those route servers later.
3. Each route comes with a metric to help the route server choose where it should be routed. We choose the metrics such that each IP address will be routed to a distinct web node (unless there is a problem).
4. The route servers (which are not routers) will then advertise the best routes they learned to all the routers in our network. This is still done using BGP.
5. Now, for a given IP address, each router knows to which web node the traffic should be directed.

Here are the respective metrics announced routes for W1, W2 and W3 when everything works as expected:

Route	W1	W2	W3	Best	Backup
2001:db8:30::1	102	101	100	W3	W2
2001:db8:30::2	101	100	102	W2	W1
2001:db8:30::3	100	102	101	W1	W3

ExaBGP configuration The configuration of ExaBGP is quite simple:

group rs  
  neighbor 2001:db8:1::4  
    router-id 1.1.1.11;
    local-address 2001:db8:6::11;
    local-as 65001;
    peer-as 65002;
   
  neighbor 2001:db8:8::5  
    router-id 1.1.1.11;
    local-address 2001:db8:6::11;
    local-as 65001;
    peer-as 65002;
   
  process watch-nginx  
      run /usr/bin/python /lab/healthcheck.py -s --config /lab/healthcheck-nginx.conf --start-ip 0;
   
 

A script is entailed to check if the service (an nginx web server) is up and running and advertise the appropriate IP addresses to the two declared route servers. If we run the script manually, we can see the advertised routes:

$ python /lab/healthcheck.py --config /lab/healthcheck-nginx.conf --start-ip 0
INFO[healthcheck] send announces for UP state to ExaBGP
announce route 2001:db8:30::3/128 next-hop self med 100
announce route 2001:db8:30::2/128 next-hop self med 101
announce route 2001:db8:30::1/128 next-hop self med 102
[...]
WARNING[healthcheck] Check command was unsuccessful: 7
INFO[healthcheck] Output of check command:  curl: (7) Failed connect to ip6-localhost:80; Connection refused
WARNING[healthcheck] Check command was unsuccessful: 7
INFO[healthcheck] Output of check command:  curl: (7) Failed connect to ip6-localhost:80; Connection refused
WARNING[healthcheck] Check command was unsuccessful: 7
INFO[healthcheck] Output of check command:  curl: (7) Failed connect to ip6-localhost:80; Connection refused
INFO[healthcheck] send announces for DOWN state to ExaBGP
announce route 2001:db8:30::3/128 next-hop self med 1000
announce route 2001:db8:30::2/128 next-hop self med 1001
announce route 2001:db8:30::1/128 next-hop self med 1002

When the service becomes unresponsive, the healthcheck script detect the situation and retry several times before acknowledging that the service is dead. Then, the IP addresses are advertised with higher metrics and the service will be routed to another node (the one advertising 2001:db8:30::3/128 with metric 101). This healthcheck script is now part of ExaBGP.

Route servers We could have connected our ExaBGP servers directly to each router. However, if you have 20 routers and 10 web servers, you now have to manage a mesh of 200 sessions. The route servers are here for three purposes:

1. Reduce the number of BGP sessions (from 200 to 60) between equipments (less configuration, less errors).
2. Avoid modifying the configuration on routers each time a new service is added.
3. Separate the routing decision (the route servers) from the routing process (the routers).

You may also ask yourself: why not use OSPF? . Good question!

• OSPF could be enabled on each web node and the IP addresses advertised using this protocol. However, OSPF has several drawbacks: it does not scale, there are restrictions on the allowed topologies, it is difficult to filter routes inside OSPF and a misconfiguration will likely impact the whole network. Therefore, it is considered a good practice to limit OSPF to network equipments.
• The routes learned by the route servers could be injected into OSPF. On paper, OSPF has a next-hop field to provide an explicit next-hop. This would be handy as we wouldn t have to configure adjacencies with each router. However, I have absolutely no idea how to inject BGP next-hop into OSPF next-hop. What happens is that BGP next-hop is resolved locally using OSPF routes. For example, if we inject BGP routes into OSPF from RS4, RS4 will know the appropriate next-hop but other routers will route the traffic to RS4.

Let s look at how to configure our route servers. RS4 will use BIRD while RS5 will use Quagga. Using two different implementations will help with resiliency by avoiding a bug to hit the two route servers at the same time.

Bird configuration There are two sides for BGP configuration: the BGP sessions with the ExaBGP nodes and the ones with the routers. Here is the configuration for the later:

template bgp INFRABGP  
  export all;
  import none;
  local as 65002;
  rs client;
 
protocol bgp ER2 from INFRABGP  
  neighbor 2001:db8:1::2 as 65003;
 
protocol bgp ER3 from INFRABGP  
  neighbor 2001:db8:1::3 as 65003;
 
protocol bgp DR6 from INFRABGP  
  neighbor 2001:db8:1::6 as 65003;
 
protocol bgp DR7 from INFRABGP  
  neighbor 2001:db8:1::7 as 65003;
 
protocol bgp DR8 from INFRABGP  
  neighbor 2001:db8:1::8 as 65003;
 

The AS number used by our route server is 65002 while the AS number used for routers is 65003 (the AS numbers for web nodes will be 65001). They are reserved for private use by RFC 6996. All routes known by the route server are exported to the routers but no routes are accepted from them. Let s have a look at the other side:

# Only import loopback IPs
filter only_loopbacks   #  
  if net ~ [ 2001:db8:30::/64 128,128  ] then accept;
  reject;
 
# General template for an EXABGP node
template bgp EXABGP  
  local as 65002;
  import filter only_loopbacks; #  
  export none;
  route limit 10; #  
  rs client;
  hold time 6; #  
  multihop 10;
  igp table internal;
 
protocol bgp W1 from EXABGP  
  neighbor 2001:db8:6::11 as 65001;
 
protocol bgp W2 from EXABGP  
  neighbor 2001:db8:7::12 as 65001;
 
protocol bgp W3 from EXABGP  
  neighbor 2001:db8:8::13 as 65001;
 

To ensure separation of concerns, we are being a bit more picky. With and , we only accept loopback addresses and only if they are contained in the subnet that we reserved for this use. No server should be able to inject arbitrary addresses into our network. With , we also limit the number of routes that a server can advertise. With , we reduce the hold time from 240 to 6. It means that after 6 seconds, the peer is considered dead. This is quite important to be able to recover quickly from a dead machine. The minimal value is 3. We could have use a similar setting with the session with the routers.

Quagga configuration Quagga s configuration is a bit more verbose but should be strictly equivalent:

router bgp 65002 view EXABGP
 bgp router-id 1.1.1.5
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor R peer-group
 neighbor R remote-as 65003
 neighbor R ebgp-multihop 10
 neighbor EXABGP peer-group
 neighbor EXABGP remote-as 65001
 neighbor EXABGP ebgp-multihop 10
 neighbor EXABGP timers 2 6
!
 address-family ipv6
 neighbor R activate
 neighbor R soft-reconfiguration inbound
 neighbor R route-server-client
 neighbor R route-map R-IMPORT import
 neighbor R route-map R-EXPORT export
 neighbor 2001:db8:1::2 peer-group R
 neighbor 2001:db8:1::3 peer-group R
 neighbor 2001:db8:1::6 peer-group R
 neighbor 2001:db8:1::7 peer-group R
 neighbor 2001:db8:1::8 peer-group R
 neighbor EXABGP activate
 neighbor EXABGP soft-reconfiguration inbound
 neighbor EXABGP maximum-prefix 10
 neighbor EXABGP route-server-client
 neighbor EXABGP route-map RSCLIENT-IMPORT import
 neighbor EXABGP route-map RSCLIENT-EXPORT export
 neighbor 2001:db8:6::11 peer-group EXABGP
 neighbor 2001:db8:7::12 peer-group EXABGP
 neighbor 2001:db8:8::13 peer-group EXABGP
 exit-address-family
!
ipv6 prefix-list LOOPBACKS seq 5 permit 2001:db8:30::/64 ge 128 le 128
ipv6 prefix-list LOOPBACKS seq 10 deny any
!
route-map RSCLIENT-IMPORT deny 10
!
route-map RSCLIENT-EXPORT permit 10
  match ipv6 address prefix-list LOOPBACKS
!
route-map R-IMPORT permit 10
!
route-map R-EXPORT deny 10
!

The view is here to not install routes in the kernel³.

Routers Configuring BIRD to receive routes from route servers is straightforward:

# BGP with route servers
protocol bgp RS4  
  import all;
  export none;
  local as 65003;
  neighbor 2001:db8:1::4 as 65002;
  gateway recursive;
 
protocol bgp RS5  
  import all;
  export none;
  local as 65003;
  neighbor 2001:db8:8::5 as 65002;
  multihop 4;
  gateway recursive;
 

It is important to set gateway recursive because most of the time, the next-hop is not reachable directly. In this case, by default, BIRD will use the IP address of the advertising router (the route servers).

Testing Let s check that everything works as expected. Here is the view from RS5:

# show ipv6  bgp  
BGP table version is 0, local router ID is 1.1.1.5
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, R Removed
Origin codes: i - IGP, e - EGP, ? - incomplete
   Network          Next Hop            Metric LocPrf Weight Path
*  2001:db8:30::1/128
                    2001:db8:6::11         102             0 65001 i
*>                  2001:db8:8::13         100             0 65001 i
*                   2001:db8:7::12         101             0 65001 i
*  2001:db8:30::2/128
                    2001:db8:6::11         101             0 65001 i
*                   2001:db8:8::13         102             0 65001 i
*>                  2001:db8:7::12         100             0 65001 i
*> 2001:db8:30::3/128
                    2001:db8:6::11         100             0 65001 i
*                   2001:db8:8::13         101             0 65001 i
*                   2001:db8:7::12         102             0 65001 i
Total number of prefixes 3

For example, traffic to 2001:db8:30::2 should be routed through 2001:db8:7::12 (which is W2). Other IP are affected to W1 and W3. RS4 should see the same thing⁴:

$ birdc6 show route 
BIRD 1.3.11 ready.
2001:db8:30::1/128 [W3 22:07 from 2001:db8:8::13] * (100/20) [AS65001i]
                   [W1 23:34 from 2001:db8:6::11] (100/20) [AS65001i]
                   [W2 22:07 from 2001:db8:7::12] (100/20) [AS65001i]
2001:db8:30::2/128 [W2 22:07 from 2001:db8:7::12] * (100/20) [AS65001i]
                   [W1 23:34 from 2001:db8:6::11] (100/20) [AS65001i]
                   [W3 22:07 from 2001:db8:8::13] (100/20) [AS65001i]
2001:db8:30::3/128 [W1 23:34 from 2001:db8:6::11] * (100/20) [AS65001i]
                   [W3 22:07 from 2001:db8:8::13] (100/20) [AS65001i]
                   [W2 22:07 from 2001:db8:7::12] (100/20) [AS65001i]

Let s have a look at DR6:

$ birdc6 show route
2001:db8:30::1/128 via fe80::5054:56ff:fe6e:98a6 on eth0 * (100/20) [AS65001i]
                   via fe80::5054:56ff:fe6e:98a6 on eth0 (100/20) [AS65001i]
2001:db8:30::2/128 via fe80::5054:60ff:fe02:3681 on eth0 * (100/20) [AS65001i]
                   via fe80::5054:60ff:fe02:3681 on eth0 (100/20) [AS65001i]
2001:db8:30::3/128 via 2001:db8:6::11 on eth1 * (100/10) [AS65001i]
                   via 2001:db8:6::11 on eth1 (100/10) [AS65001i]

So, 2001:db8:30::3 is owned by W1 which is behind DR6 while the two others are in another part of the network and will be reached through the appropriate link-local addresses learned by OSPF. Let s kill W1 by stopping nginx. A few seconds later, DR6 learns the new routes:

$ birdc6 show route
2001:db8:30::1/128 via fe80::5054:56ff:fe6e:98a6 on eth0 * (100/20) [AS65001i]
                   via fe80::5054:56ff:fe6e:98a6 on eth0 (100/20) [AS65001i]
2001:db8:30::2/128 via fe80::5054:60ff:fe02:3681 on eth0 * (100/20) [AS65001i]
                   via fe80::5054:60ff:fe02:3681 on eth0 (100/20) [AS65001i]
2001:db8:30::3/128 via fe80::5054:56ff:fe6e:98a6 on eth0 * (100/20) [AS65001i]
                   via fe80::5054:56ff:fe6e:98a6 on eth0 (100/20) [AS65001i]

Demo For a demo, have a look at the following video (it is also available as an Ogg Theora video).

---

1. The primary target of VRRP is to make a default gateway for an IP network highly available by exposing a virtual router which is an abstract representation of multiple routers. The virtual IP address is held by the master router and any backup router can be promoted to master in case of a failure. In practice, VRRP can be used to achieve high availability of services too.
2. However, you could deploy an L2 overlay network using, for example, VXLAN and use VRRP on it.
3. I have been told on the Quagga mailing-list that such a setup is quite uncommon and that it would be better to not use views but use --no_kernel flag for bgpd instead. You may want to look at the whole thread for more details.
4. The output of birdc6 shows both the next-hop as advertised in BGP and the resolved next-hop if the route have to be exported into another protocol. I have simplified the output to avoid confusion.

Pretend I run a cluster, for hosting a site. Pretend that I have three-six web-nodes, and each one needs to know which database host to contact. How do I control that? Right now I have a /etc/settings.conf file, more or less, deployed by Slaughter. That works. Another common pattern is to use a hostname - for example pmaster.example.org. However failover isn't considered here. If I wanted to update to point to a secondary database I'd need to either:

• Add code to retry the second host on failure.
  • Worry about divergence if some hosts used DB1, then DB2, then DB1 came back online.
  • Failover is easy. Fail-back is probably best avoided.
• Worry about DNS caches and TTL.

In short I'm imagining there are several situations where you want to abstract away the configuration in a cluster-wide manner. (A real solution is obviously floating per-service IPs. Via HAProxy, Keepalived, ucarp, etc. People do that quite often for database specifically, but not for redis-servers, etc.) So I'm pondering what is essentially a multi-cast accessible key-value storage system. Have a deamon on the VLAN which will respond to multicast questions like "get db", or "get cache", with a hostname/IP/result. Suddenly your code would read:

• Send mcast question ("which db?").
• Get mcast reply ("db1").
• Connect to db1.

To me that seems like it should be genuinely useful. But I'm unsure if I'm trading one set of problems for another. I can't find any examples of existing tools/deamons in this area, which either means I'm being novel, innovate, and interesting. Or I'm over thinking...

Kim Collins was perhaps thinking more about physical improvement but his advice holds well for software.

A lot has been written about the problems around software engineers wanting to rewrite a codebase because of "legacy" issues. Experience has taught me that refactoring is a generally better solution than rewriting because you always have something that works and can be released if necessary.

Although that observation applies to the whole of a project, sometimes the technical debt in component modules means they need reimplementing. Within NetSurf we have historically had problems when such a change was done because of the large number of supported platforms and configurations.
HistoryA year ago I implemented a Continuous Integration (CI) solution for NetSurf which, combined with our switch to GIT for revision control, has transformed our process. Making several important refactor and rewrites possible while being confident about the overall project stability.

I know it has been a year because the VPS hosting bill from Mythic turned up and we are still a very happy customer. We have taken the opportunity to extend the infrastructure to add additional build systems which is still within the NetSurf projects means.

Over the last twelve months the CI system has attempted over 100,000 builds including the projects libraries and browser. Each commit causes an attempt to build for eight platforms, in multiple configurations with multiple compilers. Because of this the response time to a commit is dependant on the slowest build slave (the mac mini OS X leopard system).

Currently this means a browser build, not including the support libraries, completes in around 450 seconds. The eleven support libraries range from 30 to 330 seconds each. This gives a reasonable response time for most operations. The worst case involves changing the core buildsystem which causes everything to be rebuilt from scratch taking some 40 minutes.

The CI system has gained capability since it was first set up, there are now jobs that:

• Perform and publish the results of a static analysis for each component using the llvm/clang project scan-build tool.
• Run coverage reports on modules which support gcov.
• Build and install full toolchains for the cross compiled target platforms.
• Run components automated tests on native platforms
• Generate release sources and packages on a correctly tagged git commit.
• Generates and publishes the automated Doxygen documentation.

The first alpha of syslog-ng 3.4 was released more than a year ago, and the first stable release earlier this year, development on syslog-ng 3.5 has been going on steadlily since then, with a lot of interesting features and changes pouring in. While the 3.5 release will not bring as many overwhelming changes as the previous two releases did, the sheer number of improvements is still substantial. Our focus was on smaller developments all over the place this time, instead of big sweeping changes. In this post, we will go over most of these features, with use-cases, examples and a few words about why they are so useful and awesome.Multi-line support, new destinations, new template functions, type hinting, unit suffixes, and other things. Read on, and be amazed!These are only bigger features, there has been a lot of small tweaking going on, and of course, all of the fixes that went into prior versions will also be in 3.5 too, and there's still a chance that something else may find its way in before the feature freeze.Multi-line supportOne of the major new features in the 3.5 release will be support for multi-line messages, a feature that has been available in syslog-ng PE for a good while, and which has been ported to and improved upon to the open source edition. Two variants of multi-line are supported, which will be detailed below. Both of them are available for the file() and pipe() sources only.Indented multi-lineThe easiest variant is indented multi-line, where each line can be followed by others, indented by whitespace, and the message continues until the first non-indented line. This is the format used by the Linux kernel too, from version 3.5, for /dev/log. This type of multi-line can be used as follows:

source s_multiline   file("/path/to/file" multi-line-mode(indented));  ;

Consider that /path/to/file has the following content:

First line
 Continuation 1;
 Continuation 2;
Second line

With the indented multi-line-mode() setting, this would turn into two log messages:

First line\n Continuation 1;\n Continuation 2;
Second line

Regexp-based multilineIf multi-line input is not based on indentation, one can use the regexp multi-line-mode() instead, which makes two new settings available: multi-line-prefix() and multi-line-garbage(). These can be used to define the start and the end of a log message: any string between a the beginning matching prefix and a matching garbage will be considered a single message. That is, the prefix will be included, the garbage will not be: it will be discarded.To illustrate:

source s_multiline  
 file("/path/to/file" multi-line-mode(regexp)
                      multi-line-prefix("^prefix")
                      multi-line-garbage(" garbage$"));
 ;

If the source contains these lines:

prefix message
continuing garbage

This will turn into the following log message:

prefix message\ncontinuing

New destinationsIt would be pretty hard to do a syslog-ng release without new destinations, and with 3.5, we will have three of them!STOMPWhile we had AMQP before, we now have a STOMP destination too, so we can stream logs to any STOMP server, with a few simple lines:

stomp(
 host("localhost")
 port(61613)
 destination("/topic/syslog")
 routing_key("")
 persistent(yes)
 ack(no)
 username("someone")
 password("something")
 body("This is the body! YAY! Here's a message: $MESSAGE!\n")
 value-pairs(scope(nv-pairs, syslog-proto, selected-macros))
)

Of course, none of these settings need to be set, you can just use the defaults, and it will just work!RiemannDid you know that syslog-ng is far more than a log collector and processor? No? You do now. When you have access to a lot of logs, and a tremendous amount of power in parsing them, you can use these tools for monitoring easily! And when we're talking monitoring, riemann is a great asset in our toolbox, and with the new destination, we can easily forward metrics to it:

riemann(
 server("localhost")
 port(5555)
 host("$HOST")
 service("$PROGRAM")
 description("$PROGRAM pids")
 metric(int("$PID"))
 ttl(300)
);

Of course, the above is a very silly example, it would make much more sense to extract data from a log message instead, but this will serve as an example.New template functions

$(upper-case TEXT...), $(lower-case TEXT...)

To turn some text into *upper-* or *lower-case*, these two template functions come in handy. They simply turn all their arguments into the respecitve case.

$(delimit DELIMITERS NEW-DELIMITER TEXT)

Sometimes one has a delimited string, where one wishes to replace the delimiters. This function does just that: give it a list of delimiters (you may need to quote it, if it contains whitespace), a replacement, and a text to replace delimiters in, and it does the rest. For example, to replace tabs and spaces with a vertical bar, one could write a template like this:

template("$(delimit \"\t \" \" \" $MESSAGE)\n")

$(env VARIABLE...)

Have you ever felt the need to check an environment variable from within syslog-ng? I did, and now I can.

Type hintingWhile syslog-ng supported sending log messages into various data stores and message queues for a while (SQL at first, MongoDB, JSON and AMQP later), even when those supported different types of data than strings, we could not do anything else. Until now. It is now possible - at places where it makes sense - to annotate templates with type hints, which the destination driver can optionally use. Type hinting is implemented for the mongodb() destination and the $(format-json) template function for now. When no type hint is specified, syslog-ng defaults to string.To add type hints, simply wrap the respective template with the hinted type, like this:

mongodb(
 value-pairs(pair("date", datetime("$UNIXDATE"))
             pair("pid", int64("$PID")))
);

Currently the following type hints exist: boolean (anything that begins with a t or 1 is true, anything that begins with f or 0 is false, everything else is an error), string, literal (same as string, but not quoted if it would be quoted otherwise), int32 (int is an alias for this), int64, and datetime. Only UNIX timestamps can be type-hinted to datetime, anything else will likely result in a casting error.It is also possible to control what happens when type casting fails: syslog-ng can drop the whole message, drop the property, or fall back to string. It can also do all of these silently:

options  
 typecast(on-error(silently-drop-property));
 ;

Using this feature with $(format-json) is very similary too:

$(format-json date=datetime("$UNIXDATE") pid=int64("$PID"))

With this feature in place, you can now store your non-string values with their proper types!Unit suffixesUnit suffixes make it considerably easier to set limits and describe numbers within the syslog-ng configuration. We no longer need to spell out sizes to the byte precious, it is now enough to write: log-fifo-size(200MiB). Now, syslog-ng will understand suffixes for kilo-, mega-, and giga-bytes, (K, M, G, respectively) either in base-10 or base-2 (with an extra i after the suffix). One can also omit the trailing b from the end.So, to set the log-fifo-size() to 2097152 bytes, one can simply use 2MiB. Or, to set it to 2000000, 2Mb. That's a whole lot easier, isn't it? No more counting zeros, no more silly typos in a ten-digit number, no more pain, but easily readable units!Miscellaneous featuresApart from the features above, there have been a lot of other changes and improvements in the code base:

• We now have a (mostly) non-recursive, quiet build system, which is not only much faster than what we had before, but more reliable too, and easier to glance through its logs, too.
• A few old settings that were never used were removed (the username() and password() settings of the mongodb() driver), some were renamed and deprecated: the replace() key transformation function of value-pairs() was renamed to replace-prefix(), as that makes the intent clearer. In this latter case, the old name is still valid, but obsoleted.
• Our integration with systemd is much tighter now: syslog-ng can notify systemd when it is ready. This also means that when systemd support is enabled, certain systemd libraries will have to be installed: syslog-ng no longer carries a convenience copy.
• A new filter was implemented: in-list, with which one can implement efficient white- or blacklists. To use it, you will need a file with one value a line, and do something along these lines:

  filter f_whitelist  
    in-list("/path/to/file.list", value("PROGRAM"));
   ;

  To do a blacklist, just negate the filter:

  filter f_blacklist  
    not in-list("/path/to/file.list", value("HOST"));
   ;

• syslog-ng now supports Linux 3.5+-style /dev/kmsg, and will use that instead of /proc/kmsg when a sufficiently recent kernel is detected (assuming one is using the system() source). The new kernel log format supports structured messages, and syslog-ng is smart enough to parse them, and make them accessible like all other message properties (with a .linux. prefix).

Did you know that you can get a user's home directory with the "~username" tilde expansion? You probably do, but how about other tilde expansions?

Thanks to a few bashisms you can make your script even more difficult to read by using "~+" to get $PWD, and "~-" to get $OLDPWD.

Are you using bash and want to access the directories stack created when using the pushd and popd bashisms? The ~i (where i is an integer) expansion can give you that. It gives you forward (~+i) and backward (~-i) access to the directories stack.

But beware, if the directories stack is smaller than the number you used, there won't be any expansion.

When using tildes in shell scripts, make sure you quote to avoid unwanted expansions. Note that the posh shell in wheezy and older do support those non-POSIX expansions.

in the first examples the tilde is quoted for easier reading; the expansion doesn't occur if the tilde is quoted.

The Garmin 910XT is the premier GPS watch for triathlon training and racing.

1. You'll probably need a newer version of pyusb:

   $ git clone git://github.com/walac/pyusb
   

   There is no need to install it systemwide - you can simply leave it checked out somewhere.

2. Get the Garmin-Forerunner-610-Extractor project from GitHub:

   $ git clone git://github.com/Tigge/Garmin-Forerunner-610-Extractor.git
   

3. Turn your watch on and plug in the USB adaptor.

4. You can now sync your activities:

   $ sudo env PYTHONPATH=/path/to/pyusb ./garmin.py
   Request basic information...
     ANT version:   AJK1.04RAF
     Capabilities:  array('B', [8, 3, 0, 186, 54, 0])
     Serial number: 3480039
   Starting system...
   Searching...
   Authenticating with Forerunner 910XT (3863343203)
    - Passkey: OK
   Downloading 1 file(s)
   Downloading 2013-09-03_21-09-32-80-19364.fit [..........]
   

   The first time you sync, you will have to authenticate on the watch itself. Downloading the activities will be much slower than you expect.

5. After the syncing has finished, don't forget to turn the watch off. If you are used to the Garmin Edge 800 (which turns itself off when you insert the USB cable) you will probably not have the habit of turning off the unit.

6. Rather confusingly, your .FIT files are available in the Xdg configuration folder. For example:

   ls -l ~/.config/garmin-extractor/3863343203/activities/
   -rw-r--r-- 1 root root   2688 Jun  9 15:23 2013-06-08_12-33-24-80-2688.fit
   -rw-r--r-- 1 root root   1171 Jun  9 15:23 2013-06-08_12-37-14-80-1171.fit
   -rw-r--r-- 1 root root   6111 Jun  9 15:23 2013-06-08_22-19-28-80-6111.fit
   -rw-r--r-- 1 root root 170327 Jun  9 15:26 2013-06-09_11-09-44-80-170327.fit
   

7. Strava supports these .fit files through the From File upload option.

The Garmin 910XT is the premier GPS watch for triathlon training and racing.

1. You'll probably need a newer version of pyusb:

   $ git clone git://github.com/walac/pyusb
   

   There is no need to install it systemwide - you can simply leave it checked out somewhere.

2. Get the Garmin-Forerunner-610-Extractor project from GitHub:

   $ git clone git://github.com/Tigge/Garmin-Forerunner-610-Extractor.git
   

3. Turn your watch on and plug in the USB adaptor.

4. You can now sync your activities:

   $ sudo env PYTHONPATH=/path/to/pyusb ./garmin.py
   Request basic information...
     ANT version:   AJK1.04RAF
     Capabilities:  array('B', [8, 3, 0, 186, 54, 0])
     Serial number: 3480039
   Starting system...
   Searching...
   Authenticating with Forerunner 910XT (3863343203)
    - Passkey: OK
   Downloading 1 file(s)
   Downloading 2013-09-03_21-09-32-80-19364.fit [..........]
   

   The first time you sync, you will have to authenticate on the watch itself. Downloading the activities will be much slower than you expect.

5. After the syncing has finished, don't forget to turn the watch off. If you are used to the Garmin Edge 800 (which turns itself off when you insert the USB cable) you will probably not have the habit of turning off the unit.

6. Rather confusingly, your .FIT files are available in the Xdg configuration folder. For example:

   ls -l ~/.config/garmin-extractor/3863343203/activities/
   -rw-r--r-- 1 root root   2688 Jun  9 15:23 2013-06-08_12-33-24-80-2688.fit
   -rw-r--r-- 1 root root   1171 Jun  9 15:23 2013-06-08_12-37-14-80-1171.fit
   -rw-r--r-- 1 root root   6111 Jun  9 15:23 2013-06-08_22-19-28-80-6111.fit
   -rw-r--r-- 1 root root 170327 Jun  9 15:26 2013-06-09_11-09-44-80-170327.fit
   

7. Strava supports these .fit files through the From File upload option.

Some of you might have noticed there is a new nagios-plugins version in the row. To support upstream in evaluating bugs and bringing the code to a wider audience, I uploaded the latest development version into Debian unstable. Since this package version contrib/ is dropped by upstream. But we stopped shipping the latest plugins of contrib/ with Debian wheezy. We also removed perlmods/ from the upstream tarball, as this was never used by our package. The packaging changes are:

* New upstream git snapshot (b15adb7762)
  - Adjust debian/copyright
* Droped the following upstream integrated patches
  - 05_fix_gets_undefined_in_iso_c11.dpatch
  - 10_check_apt_perfdata.dpatch
  - 11_check_nt_npe.dpatch
  - 12_check_smtp_double_threshold.dpatch
* Fix removing empty /usr/include
* Add new check_dbi into nagios-plugins-standard
* Remove whois from Recommands, as check_bgpstate is removed
* Cleanup debian/rules from unused stuff
* Provide information about repackaging
  - Extend debian/README.source
  - Add debian/bin/repack.sh
* Updating standards version to 3.9.4 (no changes needed)
* Add libfreeradius-client-dev to build-deps and remove libradiusclient-ng-dev
  (Closes: #721621)
* Remove traces of contrib/ from debian/copyright
* Remove command.cfg from examples in nagios-plugins-common, it was removed
  upstream

The Debian Nagios Maintainer and Upstream Developer are asking you to try out this new package on non production systems. Not that we think there might be a high risk for damage, but you know .. :) The latest version of the plugins are running on several systems already, but it is nearly impossible to test all scenarios even automated. If you find bugs, please report all packaging bugs into the BTS. Upstream bugs you might consider to report in the upstream bugtracker. If you want to give it a try on Debian wheezy, you could install the packages from our wheezy-backports repository:

    # wget "http://ftp.cyconet.org/debian/sources.list.d/wheezy-backports-cyconet.list" \
        -O /etc/apt/sources.list.d/wheezy-backports-cyconet.list
    # aptitude update
    # aptitude -t wheezy-backports install nagios-plugins

For adding our archive key, you can just install the package "debian-cyconet-archive-keyring"

So that forum I mentioned? I've setup a test-installation at:

• http://spam-box.co.uk

What does this forum offer? A cross between hacker news and reddit. If the admin of the forums enables it you can create arbitrary tags, and then view them. For example:

• http://example.com/view/tag-name

It's also very fast, and reasonably easy to customize. Which is good, because the current layout is nasty. Things I like:

• Everything is stored in Redis.
• The code is made of simple primitives which are joined together in a web-application. Which means most of the logic is outside the core.
• The templates are pretty basic, which means a real designer can do good things.

Not much more to say really; except I've setup a test install and if you wish to login/register and post spam feel free.