Search Results: "tar"

19 September 2013

Luca Falavigna: So long, and thanks for all the packages!

I ve been running several Deb-o-Matic instances on several servers since ages. This service, meant to provide a way to automate the build of Debian source packages, has been the starting point for a lot of Italian developers to start packaging, and eventually become Debian or Ubuntu developers. I can t exactly tell how many packages Deb-o-Matic servers have built so far, but it has been the primary build and test environment for at least three Debian Developers since four years. This service was hosted for free since the beginning, but the good things always come to an end. My provider decided they cannot keep our servers running (its core business is totally unrelated to hosting servers), and asked me to shut them down. I started with the two kfreebsd-* machines, and soon I will shut down i386 and amd64 as well. When all services will be down, I ll take care of removing the debian.net domains I created for them. Deb-o-Matic development won t stop, but developers who used Deb-o-Matic daily will have to find a different solution to build and test their packages.

17 September 2013

Julien Danjou: Python 3.4 single dispatch, a step into generic functions

I love to say that Python is a nice subset of Lisp, and I discover that it's getting even more true as time passes. Recently, I've stumbled upon the PEP 443 that describes a way to dispatch generic functions, in a way that looks like what CLOS, the Common Lisp Object System, provides. What are generic functions If you come from the Lisp world, this won't be something new to you. The Lisp object system provides a really good way to define and handle method dispatching. It's a base of the Common Lisp object system. For my own pleasure to see Lisp code in a Python post, I'll show you how generic methods work in Lisp first. To begin, let's define a few very simple classes.
(defclass snare-drum ()
())

(defclass cymbal ()
())

(defclass stick ()
())

(defclass brushes ()
())

This defines a few classes: snare-drum, symbal, stick and brushes, without any parent class nor attribute. These classes compose a drum kit, and we can combine them to play sound. So we define a play method that takes two arguments, and returns a sound (as a string).
(defgeneric play (instrument accessory)
(:documentation "Play sound with instrument and accessory."))

This only defines a generic method: it has no body, and cannot be called with any instance yet. At this stage, we only inform the object system that the method is generic and can be then implemented with various type of arguments. We'll start by implementing versions of this method that knows how to play with the snare-drum.
(defmethod play ((instrument snare-drum) (accessory stick))
"POC!")

(defmethod play ((instrument snare-drum) (accessory brushes))
"SHHHH!")

Now we just defined concrete methods with code. They also takes two arguments: instrument which is an instance of snare-drum and accessory that is an instance of stick or brushes. At this stage, you should note the first difference with object system as built into language like Python: the method isn't tied to any class in particular. The methods are generic, and any class can implement them, or not. Let's try it.
* (play (make-instance 'snare-drum) (make-instance 'stick))
"POC!"

* (play (make-instance 'snare-drum) (make-instance 'brushes))
"SHHHH!"

* (play (make-instance 'cymbal) (make-instance 'stick))
debugger invoked on a SIMPLE-ERROR in thread
#<THREAD "main thread" RUNNING 1002ADAF23 >:
There is no applicable method for the generic function
#<STANDARD-GENERIC-FUNCTION PLAY (2)>
when called with arguments
(#<CYMBAL 1002B801D3 > #<STICK 1002B82763 >).

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
0: [RETRY] Retry calling the generic function.
1: [ABORT] Exit debugger, returning to top level.

((:METHOD NO-APPLICABLE-METHOD (T)) #<STANDARD-GENERIC-FUNCTION PLAY (2)> #<CYMBAL 1002B801D3 > #<STICK 1002B82763 >) [fast-method]

As you see, the function called depends on the class of the arguments. The object systems dispatch the function calls to the right function for us, depending on the arguments classes. If we call play with instances that are not know to the object system, an error will be thrown. Inheritance is also supported and the equivalent (but more powerful and less error prone) equivalent of Python's super() is available via (call-next-method).
(defclass snare-drum () ())
(defclass cymbal () ())

(defclass accessory () ())
(defclass stick (accessory) ())
(defclass brushes (accessory) ())

(defmethod play ((c cymbal) (a accessory))
"BIIING!")

(defmethod play ((c cymbal) (b brushes))
(concatenate 'string "SSHHHH!" (call-next-method)))

In this example, we define the stick and brushes classes as subclass of the accessory class. The play method defined will return the sound BIIING! regardless of the accessory instance that is used to play the cymbal. Except in the case where it's a brushes instance; only the most precise method is always called. The (call-next-method) function is used to call the closest parent method, in this case that would be the method returning _"BIIING!".
* (play (make-instance 'cymbal) (make-instance 'stick))
"BIIING!"

* (play (make-instance 'cymbal) (make-instance 'brushes))
"SSHHHH!BIIING!"

Note that CLOS is also able to dispatch on object instances themself by using the eql specializer. But if you're really curious about all features CLOS provides, I suggest you read the brief guide to CLOS by Jeff Dalton as a starter. Python implementation Python implements a simpler equivalence of this workflow with the singledispatch function. It will be provided with Python 3.4 as part of the functools module. Here's a rough equivalence of the above Lisp program.
import functools

class SnareDrum(object): pass
class Cymbal(object): pass
class Stick(object): pass
class Brushes(object): pass

@functools.singledispatch
def play(instrument, accessory):
raise NotImplementedError("Cannot play these")

@play.register(SnareDrum)
def _(instrument, accessory):
if isinstance(accessory, Stick):
return "POC!"
if isinstance(accessory, Brushes):
return "SHHHH!"
raise NotImplementedError("Cannot play these")

We define our four classes, and a base play function that raises NotImplementedError, indicating that by default we don't know what to do. We can then write specialized version of this function with a first instrument, the SnareDrum. We then check for the accessory type that we get, and return the appropriate sound or raise NotImplementedError again if we don't know what to do with it. If we run it, it works as expected:
>>> play(SnareDrum(), Stick())
'POC!'
>>> play(SnareDrum(), Brushes())
'SHHHH!'
>>> play(Cymbal(), Brushes())
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/jd/Source/cpython/Lib/functools.py", line 562, in wrapper
return dispatch(args[0].__class__)(*args, **kw)
File "/home/jd/sd.py", line 10, in play
raise NotImplementedError("Cannot play these")
NotImplementedError: Cannot play these
>>> play(SnareDrum(), Cymbal())
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/jd/Source/cpython/Lib/functools.py", line 562, in wrapper
return dispatch(args[0].__class__)(*args, **kw)
File "/home/jd/sd.py", line 18, in _
raise NotImplementedError("Cannot play these")
NotImplementedError: Cannot play these

The singledispatch module looks through the classes of the first argument passed to the play function, and calls the right version of it. The first defined version of the play function is always run for the object class, so if our instrument is a class that we did not register for, this base function will be called. For whose eager to try and use it, the singledispatch function is provided Python 2.6 to 3.3 through the Python Package Index. Limitations First, as you noticed in the Lisp version, CLOS provides a multiple dispatcher that can dispatch on the type of any of the argument defined in the method prototype, not only the first one. Unfortunately, Python dispatcher is named singledispatch for this good reason: it only knows to dispatch on the first argument. Guido van Rossum wrote a short article about the subject that he called multimethod a few years ago. Then, there's no way to call the parent function directly. There's no equivalent of the (call-next-method) from Lisp nor the super() function that allows to do that in Python class system. This means you will have to use various trick to bypass this limitation. So while I am really glad that Python is going toward that direction, as it's a really powerful way to enhance an object system, it really lacks a lot of more advanced features that CLOS provides out of the box. Though, improving this could be an interesting challenge. Especially to bring more CLOS power to Hy. :-)
The Hacker Guide to Python
A book I'm writing that will be launched soon, talking about designing Python applications, state of the art, and various Python tips. If you want to be the first to hear about the launch, subscribe now.

Craig Small: jqGrid in TurboGears2 Admin Screens

I wanted to use the jqGrid for my admin pages as I liked that look and it matches some of the other screens outside the admin controller. The admin controller, or rather the CrudRestController out of tgext.crud, has a way of exporting results in json format. So surely its a matter of changing the TableBase to use jqGrid in the admin controller and we re done? Well, no. First you need to adjust the jsonReader options so that it lines up to the field names that the controller sends and this was one of the first (or last snags). The json output looks like:
 
  "value_list":  
    "total": 20,
    "items_per_page": 7,
    "page": 1,
    "entries": [(lots of entries)...]
   
 
Now this is a little odd because of the top-level dictionary that is being used here. Most of the examples have everything that is inside the value_list being returned. In fact adjusting the controller to return only those items in the value_list values works. To look inside this structure we need to adjust the jsonReader options. jqGrid documentation uses the format toptier>subtier for the XML reader so that was the intial attempt. It was also an intial fail, it doesn t work and you get your very familiar empty grid. The trick is close to this format, but slightly different for json. You change the options to toptier.subtier . In other words change the greater than symbol to a full stop for json access. The jqGridWidget now has the following options (amongst others):
options =  
  'datatype': 'json',
  'jsonReader':  
    'repeatitems': False,
    'root': 'value_list.entries',
    'total': 'value_list.total',
    'page': 'value_list.page',
   
 
There might be a way of saying all entries sit under value_list inside jqGrid, but I couldn t find it. Those options given above do give a working jqGrid on the admin screens.

Vincent Sanders: Ever tried. Ever failed. No matter. Try Again. Fail again. Fail better.


I seem to be adhering to Beckett's approach recently but for some reason, after my previous stool attempts, despite having a functioning design I felt I had to do just one more iteration.

Five legged stoolGoing back to a previous concept of having five legs instead of three while this did not improve the rotational problems with the 12mm thick material, it did make the design more stable overall and less prone to tipping.

The five legged solution realised in 18mm plywood resulted in my final design for this concept. As my friend Stephen demonstrates the design is pretty solid even for those of us with a more ample frame. There are now a couple of them in use at the space alongside the three legged earlier versions.

I was finally satisfied with the result and thought I was done with furniture making for a while. My adorable wife then came up with a challenge, she wanted a practical foldable chair her requirements were:
A frame folding chair, image from wikimedia
Things for individual humans to sit on raised off the floor, or chairs as we call them, have been around for a long time. My previous research for the stool indicated that chairs have a long history with examples still surviving from the ancient egyptians. From this I assumed there would be nothing novel in this project and to quote the song it's all been done before
Foldingchairs
Given this perspective my research started with an image search for "folding chair". The results immediately showed there were two common shapes. Either the chair base was formed with a pair of linked X shapes with the seat across the top, or an A frame style where the seat is across the centre.

My initial thoughts were to replicate the IKEA style A frame design until I noticed that some designs were made from a single sheet of timber with a small number of profile cuts. I searched for pre-existing design files but found none, perhaps I had found something novel to do after all.

A frame chair  1 - Front  2 - Rear  3- seat
Some quick measurements of chairs in shops (and more weird looks, mainly from my family) suggested 900mm is generally the highest a chair ought to stand. I selected a popliteal height of 420mm as a general use compromise which is a bit less than the 430mm of most mass produced chairs. I also decided there should be a front to rear drop across the seat which ought to improve comfort a little.

Material selection was based upon what I had to hand which consisted of a couple of sheets of structural plywood (1220x606x18mm temperate softwood - probably spruce although not specified) which had not become stools yet. Allowing for edge wastage and tool width this gave me a working area from a sheet of 1196x582mm.

I sketched the side view of the chair numbering the three parts. I decided part 1 would be 1000mm total height with top and bottom cross braces 80mm high, assuming the 900mm tall target, the A frame apex would be at 820mm height.

If we assert that the apex is immediately below the top cross brace on part 1 this gives 920mm at which point we have two sides of a right angled triangle (820 and 920) from which we can calculate the top angle is 26 . This also means part 2 will be 840mm long with the same 820 height this gives an angle of 10 .

I did the maths for the seat triangle using the 36 angle and 420mm popliteal height. Having experienced how flexible 18mm plywood was in the face dimension I decided to make the frame sides 50mm wide leaving, after 6mm tool path widths, enough space for a 358mm wide seat .

Once the dimensions were decided the design went pretty quickly producing something that looked similar to the designs I had seen earlier.

CNC routers can go wrong, spot the fail
Generating toolpaths for this design proved challenging with the CAM software we had available, mainly because arranging multiple "inner" and "outer" cuts along the same path and having the tabs line up was obviously not an anticipated feature.

Things had been going far too well for me and during the routing operation our machine decided to follow an uncommanded toolpath excursion mid job (it cut a dirty great hole through the middle of my workpiece where I did not want it) ruining the sheet of material.

Fortunately, once factory reset and reconfigured, the machine ran the job correctly the second time although we still do not know why it went wrong. The parts were cut free of their tabs and hinges fitted.

The resulting design worked with a couple of problems:
Folding chair in 24mm plywood with edges rounded off
So time for a second iteration, this time I selected a thicker plywood to reduce the bend under large loads (OK already, I mean me). After shopping around for several days I discovered that unlike 12mm and 18mm thick plywood 24mm thick is much harder to find at a sensible price especially if you want it cut into 1220x606 sheets from the full 1220x2440.

I eventually physically went to Ridgeons and looked at what they actually had in the warehouse and got a price on a sheet of brazilian elliotis pine structural plywood for 43.96 which they cut for me while I waited. Being physically present also gave me the opportunity to pick a "less bad" sheet from the stack much to the annoyance of the Ridgeons employee for not just taking the top one.

This time I reduced the apex height by 30mm, the length of part 1 by 80mm. I also reduced the frame width by 10mm relying on the increased thickness of the plywood to maintain the strength. In area terms it is actually an increase in material (50mm*17.7mm = 885 against 40mm*22.80 = 912). I also included rebates for the top hinges within the design allowing them to be mounted flush within the sheet width.

The new design was cut (without incident this time) and the edges rounded off. I also added some simple catches to keep the seat and back inline while being transported.

While this version worked I had made a couple of mistakes again. The first was simply a radius versus diameter error on the handles which meant too much material was removed from a high stress area causing increased bending.

Third version of folding chair ready for finishing
The second error was a stupid mistake that where I had worked the latch holes in the seat to the wrong side of my measure line meaning the apex angle was 35.5 not the expected 33.6 which meant the backrest felt "too far forward"

The third version rectified those two errors and improved the seat front curve a little to further reduce the unneeded material in the back rest.

The fabrication of this design was double sided allowing for the seat hinge to be fully rebated and become flush and also adding some text similar to the stools.

Alex sat on version 3 folding chair
This version still suffers from a backrest that is a bit too far forward and my sons complain it makes them sit up too straight. The problem is that the angle of the front face (23.6 ) means that to move the rest back 20mm means the piece needs to extend another 50mm beyond the frame apex which would probably make the chair feel tall again.

I guess this is a compromise that would take another iteration or two to solve, alas my wife has declared a moratorium on more chairs unless I find somewhere to put the failed prototypes other than her conservatory.

One other addition would be a better form of catch for transport. The ones in version 2 work but are ugly, magnets inset into the frame have been suggested but not actually implemented yet.

Folding chairs versions 1 to 3 and TERJE for comparison
In conclusion I think I succeeded with the original brief and am pretty pleased with the result. A neatly folding chair that can be stacked simply by having a pile of them and are easy to move and setup.

The price per chair is a little higher than I would like at 22 ( 11 for timber, 5 for hinges and screws , 5 for varnish and 1 for tool wear) plus about 2 hours labour (two sided routing plus roundover takes ages)

As previously the design files are available on github and there are plenty more images in the flikr set.

John Goerzen: Earthrise

Today I link you to a video narrated by the legendary Carl Sagan The Frontier is Everywhere. Partial quotes:
We were hunters and foragers. The frontier was everywhere. We were bounded only by the earth, and the ocean, and the sky. The open road still softly calls. Our little terraqueous globe as the madhouse of those hundred thousand millions of worlds. We, who cannot even put our own planetary home in order, riven with rivalries and hatreds; are we to venture out into space? By the time we are ready to settle even the nearest other planetary systems, we will have changed For all our failings, despite our limitations and fallibilities, we humans are capable of greatness How far will our nomadic species have wandered, by the end of the next century, and the next millennium? Our remote descendants, safely arrayed on many worlds through the solar system, and beyond, will be unified, by their common heritage, by their regard for their home planet, and by the knowledge that, whatever other life may be, the only humans in all the universe, come from Earth. They will gaze up and strain to find the blue dot in their skies. They will marvel at how vulnerable the repository of all our potential once was, how perilous our infancy, how humble our beginnings, how many rivers we had to cross, before we found our way.
Can you imagine seeing Earthrise? Earthrise, 1968 December 24 Incredible.

Cyril Brulebois: Fixing bugs

It s a somewhat strange feeling to spend time fixing things that broke instead of implementing new things, but it s not like I m a creative guy anyway For those last two points, the long term plan is: Hopefully getting this done this week (famous last words, eh?).

16 September 2013

Debian Med

DebConf 13 report (by Andreas Tille) General impression unofficial  Scenic Hacklab I'm beginning my DebConf report in an unofficial "Scenic Hacklab" right at the edge of the lake in Yverdon. This is the right place to memorise the last days. When I started from this place cycling to Le Camp 12 days ago I was full of great expectations and what should I say - the reality has even beaten these. Once it comes about comparing DebConfs even if it is an unfair comparison due all the differences my secret long term favourite was Helsinki very closely followed by Argentina and also very closely followed by all the other great DebConfs I joined (and I joined all in Europe). Would Le Camp be able to beat it? The short answer is: Yes, it is now my favourite DebConf while I think I do not suffer from the last-Debconf-was-the-best-DebConf-syndrome (and I realised there are others thinking the same). As you might probably know I'm a bit addicted to swimming. While Helsinki had admittedly the better conditions I was at least able to fix the distance issue using my bicycle. (Hey, those Le Camp photographers did a great job in hiding the fact that you can not actually touch the lake right from the meadow of Le Camp.) Being able to have my bicycle at DebConf scored some extra points. However, the really great view of the lake, the inspiring "Scenic Hacklab" which was my favourite place has bumped DebConf13 at first place in my personal ranking. So it comes quite natural to say: "Kudos to the great organisation team!" They did a Swiss-like precise work and perfectly succeeded in hiding any problems (I assume there were some as always) from the attendees so everything went smooth, nice and shiny for the attendees. The local team was even precise in setting up great weather conditions for DebConf. sunrise over  the lake While saying thanks to the local team I would like to also explicitly thank Luca Capello who has quite some share that this DebConf was possible at all (while I have to decrease my DebConf score one point because he was not really there - Luca to bad that you were not able to come full time!) Also thanks to Gunnar and Gannef who helped remotely (another score down because I were missing them this year as well). Even if it was my favourite DebConf I was not able to work down my todo list fully (which was not only uploading one package per day which I at least statistically fullfilled). But that's probably a general feature of todo lists anyway. One item was definitely done: Doing my daily swimming BoF. I actually was able to do the other parts of the triathlon which was skipped by Christian and have done in summary about 150km cycling with 3500m elevation and estimated 7-8km swimming (0m elevation ;-)). Considering the great view at sunrise over the lake I was not hating my "Senile bed escape" disease too much (I was every day waking up at sunset) - it was simply a great experience. I will never forget seeing water drips glimmering like gold inside the morning sun while seeing the Alps panorama in the distant. I hope I was able to help all interested swimmers with the DebConf Beach Map which was just a by-product of my activities in DebCamp. Speaking about OSM: I was astonished that the area was way less covered than I expected. Thanks to several DebConf attendees the situation became better and the map does not only show random trees in the wild but also the tracks leading to these. (Remark: It was no DebConf attendee who is responsible for plastering the map with single trees.) While I had my mapping focus basically close to the edge of the lake I was also able to even map my very own street. :-) I clearly remember one specific mapping tour when I was invited by the DPL: He convinced me to join him on a bicycle tour and since I was afraid to get fired I joined him instead to keep on hacking. Also Sorina was brave enough to join us on the tour and she did quite well. (Sorina, do you remember the agreement about your work on the installer? ;-)) Lucas described the tour as: going uphill on only asphalted roads. Sorina and me were witnessing the mighty DPL powers when we left the wood around Le Camp to reach the described road: The asphalt was just put onto the road - no doubt that it was done on the immediate demand of mighty DPL. :-) DebCamp time was flying like nose dive and a lot of known (and unknown) faces arrived at Le Camp. What I really liked a lot this year was that several really young children has pulled down the average age of DebConf attendees. I clearly remember all the discussion one year ago what to do about children. As always the issue was solved in a typical Debian way: Just do it and bring your children - they had obviously a great time as well. I think the youngest child was 2 months and the oldest "child" above 20. ;-) Actually Baptiste Perrier did great in making the C&W party a success and had obviously a nice time. (I wished my son would have been able to come as well but he needs to write his bachelor s thesis in physics. :-() It was nice to see the kids using all playing facilities and communicating with geeks. Also I would like to point out that even the very young attendees had their share at the success of DebConf: Just think of the three "bell ringing assistants" who helped me ringing the bells for lunch and dinner. I've got this cool job from Didier in the beginning of DebCamp. I must say having some real bells ringing is by far nicer than just the "lunch / dinner starts in 10 minutes" from IRC bot. The only thing I did not understand was that people did not considered ringing the bells at 8:00 for breakfast as a good idea. Regarding the food in general I would also like to send kudos to the kitchen: It was tasty, freshly prepared, regional food with a good change rate. I really liked this. Extra points for having the chance to sit outside when eating. Talks But lets have a look into the conference programme. I'd really recommend watching the videos of the talks Bits from the DPL (video) and Debian Cosmology (video). I considered both talks as entertaining and interesting. I also really hope that the effort Enrico Zini started in Debian Contributors (video) will be successful. I had some talks and BoFs myself starting with Why running a Blend (video) and I admit that (as usual) the number of attendees was quite low even if I think there is some proof (see below) that it is interesting for way more people who should consider working more "blendish" in their team. Do you know how to recruit one developer per year and relax the man power problem in your team? Feel free to watch the video. We have confirmation that ten DDs of our team have considered to join Debian only because Debian Med exists. Admittedly biology and medicine are really leaf topics inside the Debian universe. So if even this topic that has a very tiny share of the Debian users is able to attract this level of attention - how many more people could we win for multimedia, games, GIS and others? So if you feel you are quite overworked with your packaging and you have no time this is most probably wrong. The amount of time is basically a matter of priorities you set for your tasks. Try to put some higher priority onto using the just existing Blends tools I explained in my talk to attract more users and developers to your team and by doing so spread the workload over more people. It works, the prove was given in my main talk. So before you start working on a specific package you should wonder who else could have an even stronger interest to get this work done and provide him with some additional motivation and help to get the common goal done. The interesting thing is that my BoF about How to attract new developers for your team (video) - which was a simple report about some by-product of the Blends work - made it into the main talk room and got way more attention. For me this is the proof that the Blends concept itself is probably badly perceived as something like "a few outsiders are doing damn specific stuff which is not really interesting for anybody else" instead of what is really is: Smoothing the way from specific upstream applications to the end user via Debian. Once you see the video of this BoF you can observe how my friend Asheesh Laroia became more and more excited about the Blends concept and admitted what I said above: We should have more Blends for different fields. Funnily enough Asheesh asked me in his excitement to talk more about Blends. This would have been a really good suggestion ten years ago. At DebConf 3 in Oslo I had my very first talk about Blends (at this time under the name "Debian Internal Projects"). I continuously kept on talking about this (MiniDebConf Peking 2005, DebConf 5, Helsinki (video), DebConf 7, Edinburgh (video), DebConf 8, Mar del Plata (video), DebConf 9, C ceres (video), MiniDebConf Berlin 2010 (video in German), MiniDebConf Paris 2010 (not video recorded), DebConf 11, Banja Luka (video) ... and these are only (Mini)DebConfs my talks page is full of this topic) and every new year I try different ways to communicate the idea to my fellow Debianistas. I'm wondering how I could invent a title + abstract avoiding the term Blends, put "Git", "release" and "systemd versus upstart" in and being able to inform about Blends reasonably by not becoming to off topic with the abstract. I also registered the Debian Science round table. I admit we were lacking some input from remote via IRC which used to be quite helpful in the past. The attendees agreed upon the handling of citations in debian/upstream files which was invented by Debian Med team to create even stronger bounds to our upstream developers by giving their work extra reward and providing users with even better documentation (see my summary in Wiki). As usual I suggested to create some Debian Science offsprings like "Debian Astronomy", "Debian Electronics", "Debian Mathematics", "Debian Physics" etc. who could perfectly leave the Debian Science umbrella to get a more fine grained structure and a more focused team to enhance the contact to our users. Unfortunately there is nobody who volunteers to take over the lead for such Blends. I have given a short summary about this BoF on the Debian Science mailing list. In the Debian Med meeting I have given some status report. No other long term team members were attending DebConf and so I gave some kind of introduction for newcomers and interested people. I touched also the DebiChem topic which maintains some packages that are used by biologists frequently and so we have a good connection to this team. Finally I had registered three BoFs in Blends I'm actually not (or not yet) active part of. My motivation was to turn the ideas I have explained in my main talk into specific application inside these teams and helping them to implement the Blends framework. In the first BoF about Debian GIS I have shown the usual team metrics graphs to demonstrate, that the one packaging team Pkg-OSM is in danger to become MIA. There are only three persons doing actual uploads. Two of them were at DebConf but did not joined the BoF because they do not consider their contribution to Pkg-OSM as a major part of their general Debian work. I will contact the main contributor David Paleino about his opinion to move the packages step by step into maintenance of Debian GIS packaging team to try to overcome the split of two teams that are sharing a good amount of interest. At least if I might become an Uploader for one of the packages currently maintained by Pkg-OSM I will move this to pkg-grass-devel (which is the name of the packaging team of Debian GIS for historical reasons). The attendees of the BoF have considered this plan as sensible. Moreover I talked about my experiences with OSGeo Live - an Ubuntu derivative that tries to provide a full tool chain to work on GIS and OSM problems ... basically the same goal as Debian GIS has just provided by the OSGeo project. I'm lurking on OSGeo mailing list when I asked explicitly I've got the answer that they are working together with Debian GIS and are using common repository (which is IMHO the optimal way of cooperation). However, it seems that several protagonists of OSGeo Live are underestimating the resources provided by Debian. For instance there was a question about Java packaging issues but people were not aware about the existence of the debian-java mailing list. I was able to give an example how the Debian Med team managed to strengthen its ties to BioLinux that is also an Ubuntu derivative for biologists. At our first Debian Med sprint in 2011 we invited developers from BioLinux and reached a state where they are using the very same VCS on Alioth where we are maintaining our packages. At DebConf I was able to upload two packages where BioLinux developers did certain changes for enhancing the user experience. My "work" was just bumping the version number in changelog and so we did profit from the work of the BioLinux developers as well as they are profiting from our work. I plan to dive a bit more into Debian GIS and try to strengthen the connection to OSGeo Live a bit. The next BoF was the Debian Multimedia meeting. It was nice that the current leader of Ubuntu Studio Kaj Ailomaa joined the meeting. When I was explaining my ideas about cooperation with derivatives I repeated my detailed explanation about the relation with BioLinux. It seems every topic you could cover inside Debian has its related derivative. So to me it seems to be quite natural to work together with the developers of the derivative to join forces. I actually consider a Blend a derivative done the right way = inside Debian. The final work for the derivers that might be left for them is doing some shiny customising of backgrounds or something like this - but all the hard work could and should be done in common with the relevant Debian team. My dream is to raise such relevant teams inside Debian ... the Blends. Finally the last BoF of this series was the Debian Games meeting. As always I presented the team metrics graphs and the Debian Games team members who attended the BoF were quite interested. So it seems to be some unknown fact that team metrics are done for several teams in side Debian and so I repeat the link to it for those who are not yet aware of it. As a result of the BoF Debian Games team members agreed to put some more effort into maintaining their Blends tasks. Moreover Miriam Ruiz wants to put some effort into reviving Debian Jr. Regarding Debian Jr. there was an interesting talk about DouDouLinux - in case you might want to watch the video I'd recommend skipping the first 30min and rather watch the nice live demo. There was also an ad hoc BoF about Debian Jr scheduled to bring together all people interested into this cute project and Per Anderson volunteered to take over the lead. I have given a summary about this specific BoF at the Debian Jr list. For some other talks that I'd regard as remarkable for some reasons: I'd regard the talk "Debian-LAN" by Andreas Mundt as some hidden pearl because it did not got a lot of attention but after having seen the video I was quite impressed - specifically because it is also relevant for the Blends topic. Memories I also liked "Paths into Debian" by Moray Allan (and I was only able to enjoy the latter talks thanks to the great work of the video team!) because it also scratched the same topic I was concerned about in my mentoring talk. Related to this was in my opinion also "Women in Debian 2013" were we tried to find out reasons for the lack of woman compared to other projects and how to overcome this issue. Geert hovering  over the grass Besides the talks I will probably never forget two specific moments that make DebConf so special. One of these moments is recorded on an image that clearly needs no words - just see Geert hovering over the grass. Another strong moment in my personal record was in the DebConf Newbies BoF "First time at DebConf" that unfortunately was not recorded but at least for this statement it would have been very great if we would have some reference better than personal memory. Aarsh Shah a GSoC student from India suddenly raised up and said: "Four months ago I was not even aware that Free Software exists. Now I'm here with so many people who are totally equal. If I will tell my mother at home that I was standing in the same queue where the Debian Project Leader was queuing up for food she will never believe me." He was totally excited about things we are regarding as normal. IMHO we should memorise moments like this that might be part of the key to success in cultures, where Debian is widely unknown and very rarely in use. Amongst these not scheduled great moments the scheduled day trip was also a great thing. I had a really hard time to decide what tour I might join but ended up in the "long distance walking (or should I say running) group". Inspired by the "running Bubulle" who was flashing between the walking groups we went uphill with 5.4km/h which was a nice exercise. Our destination the large cliff was an exciting landscape and I guess we all enjoyed the dinner organised by the "Trout cabal". ;-) say goodby to  friends So I had a hard time to leave Le Camp and tried hard to make sure my memories will remain as long as possible. Keeping some signs attached to my bicycle, conserving the "Scenic Hacklab" sign for my private "scenic hacklab @ home" was one part. I also have cut some branches of the Buxus sempervirens in Le Camp and have put them in my garden at home (where I create some hedgerow from places where I spent some great time). These will probably build a great part of the hedgerow ... Thanks for reading this longish report. Looking forward to see you all in Germany 2015 (or earlier) Andreas. Scenic Hacklab  @ home

15 September 2013

Christoph Berg: Jessie, the HP 6715b, and Wifi

If you are upgrading your HP/Compaq 6715b to Debian Jessie, and suddenly Wifi stops working because the PCI device is gone, install the "rfkill" package:
# lspci   tail -2
02:04.1 FireWire (IEEE 1394): Ricoh Co Ltd R5C832 IEEE 1394 Controller (rev 02)
10:00.0 Ethernet controller: Broadcom Corporation NetLink BCM5787M Gigabit Ethernet PCI Express (rev 02)
# rfkill list 1
1: hp-wifi: Wireless LAN
    Soft blocked: yes
    Hard blocked: no
# rfkill unblock wifi
# rfkill list 1
1: hp-wifi: Wireless LAN
    Soft blocked: no
    Hard blocked: no
# lspci   tail -2
10:00.0 Ethernet controller: Broadcom Corporation NetLink BCM5787M Gigabit Ethernet PCI Express (rev 02)
30:00.0 Network controller: Broadcom Corporation BCM4311 802.11a/b/g (rev 02)
Reports on the internet say that the same could be done by going into the BIOS and selecting "Reset to default" - this makes the Wifi LED active until about udev is started on the next boot. To be done: figure out how to automate this.

14 September 2013

Gerfried Fuchs: Earthlings

If you have been in Vienna over the last few weeks, you probably have seen it: there was an URL crayoned throughout the city at several places. Noone standing next to it telling you anything about it, just the address. Plain and simple. The address said earthlings.com (or rather, earthlings.de, but it's actually the same). Curious as I am, I wondered what it's about. And obviously, it wasn't too hard to find out. It's a documentary. But I don't want to say too much about it, I guess I let your own curiosity lead you. All that I want to say is that I consider it worth seeing. And especially worth seeing it through to the end. Definitely got me thinking. Thanks to the unknown people over here in Vienna who made it possible for me to stumble upon it.

/misc permanent link Comments: 4 Flattr this

Russell Coker: Is Portslave Still Useful?

Portslave is a project that was started in the 90 s to listen to a serial port and launch a PPP or SLIP session after a user has been authenticated, I describe it as a project not a program because a large part of it s operation is via a shared object that hooks into pppd, so if you connect to a Portslave terminal server and just start sending PPP data then the pppd will be launched and use the Portslave shared object for authentication. This dual mode of operation makes it a little tricky to develop and maintain, every significant update to pppd requires that Portslave be recompiled at the minimum, and sometimes code changes in Portslave have been required to match changes in pppd. CHAP authentication was broken in a pppd update in 2004 and I never fixed it, as an aside the last significant code change I made was to disable CHAP support, so I haven t been actively working on it for 9 years. I took over the Portslave project in 2000, at the time there were three separate forks of the project with different version numbering schemes. I used the release date as the only version number for my Portslave releases so that it would be easy for users to determine which version was the latest. Getting the latest version was very important given the ties to pppd. When I started maintaining Portslave I had a couple of clients that maintained banks of modems for ISP service and for their staff to connect to the Internet. Also multi-port serial devices were quite common and modems where the standard way of connecting to the Internet. Since that time all my clients have ceased running modems. Most people connect to the Internet via ADSL or Cable, and when people travel they use 3G net access via their phone which is usually cheaper, faster, and more convenient than using a modem. The last code changes I made to Portslave were in 2010, since then I ve made one upload to Debian for the sole purpose of compiling against a new version of pppd. I have no real interest in maintaining Portslave, it s no longer a fun project for me, I don t have enough spare time for such things, and no-one is paying me to work on it. Currently Portslave has two Debian bugs, one is from a CMU project to scan programs for crashes that might indicate security flaws, it seems that Portslave crashes if standard input isn t a terminal device [1]. That one shouldn t be difficult to solve. The other Debian bug is due to Portslave being compiled against an obsolete RADIUS client library [2]. It also shouldn t be that difficult to fix, when I made it use libradius1 that wasn t a difficult task and it should be even easier to convert from one RADIUS library to another. But the question is whether it s worth bothering. Is anyone using Portslave? Is anyone prepared to maintain it in Debian? Should I just file a bug report requesting that Portslave be removed from Debian?

13 September 2013

Daniel Pocock: Calendar and Contact data with free software in the Smartphone era

Since the rise of smartphones, first Blackberry, then iPhone and now Android, the world at large has had a convenient option to fall back on for management of contact and calendar data. The data itself seems somewhat trivial: many of us software developers have worked with a plethora of obscenely more obscure schemas. It is unlikely that most people would ever have to perform an operation on their address book that requires a statistical toolkit like the R project or any of the other advanced tools we now have for data processing. In terms of data, addresses and phone numbers can appear quite boring. So why hasn't the free software community mastered it and cloud providers are currently running rampant hoovering up the data of billions of users? Ruling out LDAP One problem that people have faced is the existence of different standards. LDAP has quite a high profile and appears to be supported by default in many desktop applications. For this reason, it tends to grab people's attention but after a close look, many people find it is not the right solution. LDAP tends to work well for organisations where the data is centrally managed and in situations where fine-grained access control is not required. LDAP servers often need a capable UNIX administer to manage effectively. To top it off, LDAP itself does not provide an effective solution for two-way synchronization of data with remote and mobile users. The emergence of SyncML, CalDAV, GroupDAV and CardDAV Addressing this final problem (two-way mobile/portable device sync) has brought on the development of further standards. Initially, SyncML (now known as Open Mobile Alliance Data Synchronization and Device Management) grabbed significant attention, partly because it had been included in some of the first generation of smartphones from once-prominent vendors like Nokia. Open source server solutions for SyncML did not gain traction though. The most prominent is Funambol, however, it is not packaged in any major Linux distribution and it requires some basic Java knowledge to maintain it effectively. Like LDAP it appears to be most comfortable in enterprise use cases and doesn't appeal to the people who run a private mail and web server. SyncML has further suffered with the lack of support in more recent generations of smartphones. WebDAV was formalised in 1999 and there are now various derivative standards such as CalDAV and CardDAV which use WebDAV as a foundation. However, standards didn't arrive at this place directly: CalDAV only appeared as RFC 4791 in 2007 and CardDAV as RFC 6352 in 2011. In the meantime, some vendors experiemented with an alternative called GroupDAV. GroupDAV is largely obsolete now and GroupDAV v2 is a subset of CalDAV and CardDAV. Finally gaining traction The good news is, CardDAV and CalDAV are gaining traction, in no small part due to support from Apple and the highly proprietary iPhone. Some free software enthusiasts may find that surprising. It appears to be a strategic move from Apple, using open standards to compete with the dominance of Microsoft Exchange in large corporate networks. Both protocols are supported natively in iPhones and Mac OS X. More exciting, Apple releases their Contacts Server and Calendar Server as open source software, making it easier to look under the hood and observe any deviations from the standards. Apple's adoption of CardDAV and CalDAV has been accompanied by a similar effort in free software, however, support is more patchy. The Mozilla Lightning calendar plug-in for Thunderbird (known as the iceowl-extension plugin for icedove on Debian) has supported CalDAV for some time. In contrast, Thunderbird's address book lacks native support for CardDAV and has other limitations (discussed in detail below). Limited CardDAV support in Thunderbird can be achieved using the third-party SOGo Connector, which is not just for ScalableOGo server. The Evolution PIM appears to support both CardDAV and CalDAV relatively smoothly and appears to be more functional and easier to get started with. Free software servers are also emerging. DAViCal, despite the name, supports both CardDAV and CalDAV. It is a good solution for people who have an existing mail server and don't want to transition to an all-in-one groupware suite. Packages available in Debian make it easy to deploy and get started. It is a PHP solution, using a PostgreSQL database underneath. For people who do want a groupware suite, CardDAV and CalDAV are supported in products like ScalableOGo and Zimbra. For people who don't want a server at all and just want to sync between a desktop and a mobile device, there are options like Calypso. Mobile support: the missing link A number of people have blogged about using the Android apps CardDAV-Sync and CalDAV-Sync to complete the picture. I've tested these myself and verified that they work. However, there is one vital thing missing: the source code. Nobody knows what is inside these apps except the developer. That makes it impossible for other developers to troubleshoot them and improve them and it also makes it impossible to be certain that they do not leak information. An alternative is the aCal app for Android. Despite the name, aCal can synchronize the address book as well as the calendar. However, in my tests against DAViCal, I found that the addressbook can only be synced once and that on subsequent syncs, it fails to discover any new server-side contacts. aCal source code is available on gitorious.
aCal appears to be a compelling solution for Android users, hopefully the contact sync will be fixed soon. It only syncs once after installation and then fails to detect any new contacts. Thunderbird's limitations It has already been mentioned that the Thunderbird address book can be synchronized to a CardDAV server by adding the SOGo Connector plugin. However, this is not a perfect solution. The plugin itself is fine: the SOGo web site describes the connector (as opposed to the SOGo Integrator plugin) as a portable solution: in other words, it is vendor neutral, compatible with any standards-compliant server. The first limitation is that it is necessary to duplicate the setup process for both the calendar sync and the address book sync. This is just a question of convenience.
The SOGo Connector plugin configuration window is very trivial. I recommend using read-only mode to avoid the problems with multiple email addresses or telephone numbers. More serious limitations exist in the address book itself: if the CardDAV server has more than 2 email addresses, they will not appear in the Thunderbird address book. Changing an email address on the client side will result in loss of those additional email addresses on the server-side. The Mozilla team initially committed to rewrite the address book from the ground up but it is not clear whether that work is still in progress. DAViCal confusion I had been using DAViCal for quite some time as a CalDAV server before I recently decided to try and use it for CardDAV as well. As a calendar server, it had been working fine. I have a small number of users with some private calendars and also some shared calendars. When I installed the SOGo Connector plugin and went to setup the client-side configuration, it asked me for the server URL. Having used CalDAV already, I simply tried copying the existing CalDAV URL, which is of the form https://myserver.example.org/davical/caldav.php/daniel/home The URL was accepted and I could create address book entries on the client side. No errors appeared. I then tried setting up the same URL as an address book in Evolution. This time, I found I was able to create contacts and sync them to the server and access them from another copy of Evolution. It appeared DAViCal was working fine and I felt that Thunderbird or the SOGo Connector was somehow broken, because the Evolution clients were syncing but Thunderbird was not. The lack of any errors from the Thunderbird GUI further undermined my confidence in the SOGo Connector. Nonetheless, I decided to dig a little deeper, looking through the DAViCal source code. This brought me to the realisation that each URL could be flagged as either a calendar resource or an address book resource. This had not been obvious to me before. Evolution had effectively been storing the contact entries into a calendar URL and DAViCal had been happily accepting them. SOGo Connector, querying the URL, appears to have received both calendar and contact records from DAViCal and given up.
Here is where I configured the extra resource for an address book store under my user account in DAViCal In the DAViCal web management console, I created an extra resource for my user ID and set the appropriate checkbox to mark the resource as an address book. Then I put this resource URL into all the clients and found that they were all able to sync with each other. DAViCal's maintainer has explained that this is a limitation in the CardDAV/CalDAV protocols: technically, it is valid for the client to store the wrong type of objects in each URL. Clients need to be more resilient when they stumble across objects they don't expect in a particular resource. Conclusion A complete free software based contact-management solution can be easily deployed using any of the servers mentioned above, particularly if Evolution is the preferred client software. Using Thunderbird as a client, it can also work, but attention is needed to avoid data loss on sync. Thunderbird could also be used in a read-only mode to access address books maintained with Evolution. Mobile support is still patchy on Android. The closed source solution (CardDAV-Sync and CalDAV-Sync) appears to work and a promising open source close, aCal, appears to be "nearly there". Some people would obviously prefer to use a standalone contact/calendar server, while others may prefer an all-in-one groupware suite. Both approaches appear to be feasible, but the standalone contact/calendar server appears to offer a more compelling way to start quickly and have minimal administrative overhead, particularly due to the fact that DAViCal is available in a supported package.
This is the contact server configuration window in Evolution. It offers slightly more options than SOGo Connector in Thunderbird and appears to be a more finished solution overall.

MJ Ray: Upgrading to Debian 7 wheezy

So I have upgraded my main workstation at last, but there are a few things I wanted to figure out. Some of these I found an answer to, but others I haven t and some answers open more questions: Where did debian-menu.menu go and why doesn t lxlauncher have the Debian menu by default anyway? Answer: It seems this is a combination of debian bug 333848 and bug 722563 which I reported yesterday in that lxlauncher doesn t use the debian menus. Workaround for now: install menu-xdg and hack the lxlauncher menu file. Why don t my xterm shells load .profile any more? Did I bodge that before without making a note? How do I stop kernel or initrd or whatever it is configuring the network with the dhcp settings at boot time instead of the better wicd ones that are available once the system is up? Continuing with networking, how did network-manager (which doesn t cope with my network configuration) get installed again? Answer: debian bug 645656 explains that debian is simply following a silly decision taken upstream and refers to attempts to make network-manager optional as the Crusade . Workaround: remove gnome and install its dependencies like gnome-core directly instead. Drawback: any new dependencies of gnome will have to be installed manually. Why is the audio capture volume at 10% by default and what s the best way to change that? I suspect this might be 682731 but that bug is untouched in over a year. I m tempted to remove pulseaudio, but gnome-core Depends pulseaudio so this is yet more hardware-breaking caused by gnome dependency changes and the only way out of this one would be to remove gnome-core and install its dependencies. Or maybe I should just give up and finally remove gnome because its packages have jumped the shark. I installed gnome to make the system easier to use and it s broken it in at least two ways now. One question that makes me hesitate is: why is gdm3 faster to start than lightdm?

12 September 2013

Russ Allbery: AFS::PAG 1.00

One of the very long-standing problems with AFS is that it doesn't have a stable library interface or API. The situation with shared libraries is even worse. Despite that, there is a native XS AFS Perl module that tries to provide a native Perl interface to the various AFS libraries, despite all of their quirks. We've used that off and on for various things over the years. Unfortunately, the historic AFS code used an LWP (lightweight threading) implementation that relied on hairy C internals, and when Perl switched over to being threaded with POSIX pthreads by default, the loadable module started having trouble. Various people managed to keep it limping along through Debian's squeeze release, but as of wheezy it just no longer worked; the thread mismatches and the mismatches between the various AFS libraries were too severe. There's work in progress to fix this by building yet another profile of the AFS libraries, with all their warts, but in the meantime we needed access to the AFS PAG functions for various backend scripts. Everything else the Perl module did could be replaced by running the regular commands and parsing their output, but setpag has to be called in the process that should be affected to work safely. Hence this module. AFS::PAG provides a native Perl interface to the API exposed by libkafs and libkopenafs for PAG manipulation: hasafs, haspag, setpag, and unlog. (Eventually it might provide access to pioctl, but I haven't done that work yet.) It supports any platform that has a libkafs or libkopenafs, or Linux without either (by implementing the pioctl interface itself). You can get the latest version from the AFS::PAG distribution page.

11 September 2013

Brett Parker: Moving, Changing Jobs and the Bank Holiday Weekend...

So, in the last couple of weeks I've (mostly) moved out of Brighton, got a new job, and generally the world has gone crazy. The bank holiday weekend was spent at Debian UK BBQ, and was fantastic fun, including the reading corner on Sunday where we played through various arcs of To Be or Not To Be (small spoiler: PIRATES!). Since then I've been mostly back and forth from Brighton moving stuff, buying a car, and starting a new job... As mentioned in my new employers blog there was a small stumbling block with my new work laptop that stopped me from getting a sensible operating system on it from the get-go, but it's now running a shiny Debian Wheezy install and alllllllllll is well with the world... well, mostly. I have just removed gnome-keyring because I hate that I can't see a way of relocking it, or infact relocking any of the individual keys that it had. So, back to ssh-agent I went, and the world is good again. But that means that I haven't got a gpg agent running at the moment, though, unless I'm missing something, I can't see a way of forcing gpg-agent to forget a passphrase either (ARGH, why on earth wouldn't you have an option for that?!).

Olivier Berger: Publishing RDF views for tastypie/django resources

Here s some documentation about a hack I ve been working on to allow publication of RDF views for Tastypie resources (in Django applications). While working on implementing support for RDF meta-data for descriptions of Debian packages for the Debian Package Tracking System rewrite, I ve tried and idetify which libraries/frameworks would allow to create some RDF views for Django model objects with a minimal effort. Ideally, this would save the hassle of writing code, and could just be a matter of mapping some Django model fields to proper ontology attributes. I haven t found an existing tool to do so, but it seemed to me that Tastypie could offer a nice starting point. Tastypie (I focused on v. 0.9.15 which is currently in Debian testing) offers some REST content-negociation support, and other niceties. This post is an attempt at documenting an initial implementation for my problem. I ve implemented it as some code in the example blog application described in the Tastypie documentation. Unfortunately, it s not yet a patch that could be applied to Tastypie, to add this as a standard feature. The code is available in my git clone of Tastypie, based on 0.9.15, and lies in the commit(s) between the example_myapp and rdfviews branches. It needs an up to date RDFLib to work (Debian s is too old, btw). It basically relies on the addition of a _rdf_mapping dict in the Django model objects, and an RdfModelResource as a base class for Tastypie ModelResources. To provide an RDF view for a ModelResource, it also requires to declare a particular MetaClass, and adding a few bits to its Meta subclass. The principle is that the dehydratation steps of the Tastypie system will replace fields values by their RDF objects counterparts, as either RDFLib Literals or URIRefs. Then the resuting Bundle will be converted to some RDFLib Graph. It is then just a matter of serialization of that Graph to turtle (I ve not added another format, but it should be pretty straightforward). There s a bit of hackish drak magic in the code, in particular when introspecting the Tastypie / Django resource / model objects to be able to dynamically add some dehydrate_FOO methods to perform the conversion to Literals or URIRefs. Custom dehydrateFoo methods can still be written to process the fields in particular ways. I ve added some LDP style paging along the way, so I hope it will be quite usable by LDP compatible tools. Here s the result : First, listing all resources : $ curl -H 'Accept: text/turtle' "http://localhost:8000/api/v1/entry/"
@prefix dcterms: <http://purl.org/dc/terms/> .
@prefix ldp: <http://www.w3.org/ns/ldp#> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix sioc: <http://rdfs.org/sioc/ns#> .
@prefix xml: <http://www.w3.org/XML/1998/namespace> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
</api/v1/entry/#list> dcterms:hasPart </api/v1/entry/1/#post>,
</api/v1/entry/2/#post>,
</api/v1/entry/3/#post> .
Then, details of one particular post : $ curl -L -H 'Accept: text/turtle' "http://localhost:8000/api/v1/entry/1"
@prefix dcterms: <http://purl.org/dc/terms/> .
@prefix owl: <http://www.w3.org/2002/07/owl#> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix sioc: <http://rdfs.org/sioc/ns#> .
@prefix xml: <http://www.w3.org/XML/1998/namespace> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
</api/v1/entry/1/#post> a sioc:Post ;
dcterms:created "2011-05-21T22:46:38+00:00"^^xsd:dateTime ;
dcterms:title "Another Post" ;
sioc:content "MESSAGE: This will prbbly be my lst post. /MESSAGE" ;
sioc:has_creator </api/v1/user/1/> ;
owl:sameAs </posts/another-post#post> .

I hope this is usable, and welcome any feedback.

10 September 2013

Steve Kemp: node.js is kicking me

Today I started hacking on a re-implementation of my BlogSpam service - which tests that incoming comments are SPAM/HAM - in node.js (blogspam.js) The current API uses XML::RPC and a perl server, along with a list of plugins, to do the work. Having had some fun and success with the HTTP+JSON mstore toy I figured I'd have a stab at making BlogSpam more modern: In theory this is easy, I've hacked up a couple of plugins, and a Perl client to make a submission. But sadly the async-stuff is causing me .. pain. This is my current status:
shelob ~/git/blogspam.js $ node blogspam.js
Loaded plugin: ./plugins/10-example.js
Loaded plugin: ./plugins/20-ip.js
Loaded plugin: ./plugins/80-sfs.js
Loaded plugin: ./plugins/99-last.js
Received submission:  "body":"This is my body ..","ip":"109.194.111.184","name":"Steve Kemp" 
	plugin 10-example.js said next :next
	plugin 20-ip.js said next :next
	plugin 99-last.js said spam SPAM: Listed in StopForumSpam.com
So we've loaded plugins, and each has been called. But the end result was "SPAM: Listed .." and yet the caller didn't get that result. Instead the caller go this:
shelob ~/git/blogspam.js $ ./client.pl
200 OK 99-last.js
The specific issue is that I iterate over every loaded-plugin, and wait for them to complete. Because they complete asynchronously the plugin which should be last, and just return "OK" , has executed befure the 80-sfs.js plugin. (Which makes an outgoing HTTP request). I've looked at async, I've looked at promises, but right now I can't get anything working. Meh. Surprise me with a pull request ;)

Craig Small: Odd WordPress pingbacks

I m getting some odd log messages for the apache module modsecurity. Essentially its xml parser is breaking when random places are sending pingbacks. The requests go to xmlrpc.php and the response headers are ok, but the body is binary. The message in the modsecurity log looks like:
Message: XML parser error: XML: Failed parsing document.
After a bit of guessing and sending messages to and fro, I can now see that it is a gziped response. So I m not sure if it is modsecurity not realising that the response is gziped or wordpress not marking it correctly. In any case I can regularly get very similar binary strings using gzip and the usual xml response. So that s half the mystery solved. Most of the requests are spammers so I m not too worried. I think it also impacts legitimate pingbacks because I ve not had any, even from the usual automatic places.

Petter Reinholdtsen: Recipe to test the Freedombox project on amd64 or Raspberry Pi

I was introduced to the Freedombox project in 2010, when Eben Moglen presented his vision about serving the need of non-technical people to keep their personal information private and within the legal protection of their own homes. The idea is to give people back the power over their network and machines, and return Internet back to its intended peer-to-peer architecture. Instead of depending on a central service, the Freedombox will give everyone control over their own basic infrastructure. I've intended to join the effort since then, but other tasks have taken priority. But this summers nasty news about the misuse of trust and privilege exercised by the "western" intelligence gathering communities increased my eagerness to contribute to a point where I actually started working on the project a while back. The initial Debian initiative based on the vision from Eben Moglen, is to create a simple and cheap Debian based appliance that anyone can hook up in their home and get access to secure and private services and communication. The initial deployment platform have been the Dreamplug, which is a piece of hardware I do not own. So to be able to test what the current Freedombox setup look like, I had to come up with a way to install it on some hardware I do have access to. I have rewritten the freedom-maker image build framework to use .deb packages instead of only copying setup into the boot images, and thanks to this rewrite I am able to set up any machine supported by Debian Wheezy as a Freedombox, using the previously mentioned deb (and a few support debs for packages missing in Debian). The current Freedombox setup consist of a set of bootstrapping scripts (freedombox-setup), and a administrative web interface (plinth + exmachina + withsqlite), as well as a privacy enhancing proxy based on privoxy (freedombox-privoxy). There is also a web/javascript based XMPP client (jwchat) trying (unsuccessfully so far) to talk to the XMPP server (ejabberd). The web interface is pluggable, and the goal is to use it to enable OpenID services, mesh network connectivity, use of TOR, etc, etc. Not much of this is really working yet, see the project TODO for links to GIT repositories. Most of the code is on github at the moment. The HTTP proxy is operational out of the box, and the admin web interface can be used to add/remove plinth users. I've not been able to do anything else with it so far, but know there are several branches spread around github and other places with lots of half baked features. Anyway, if you want to have a look at the current state, the following recipes should work to give you a test machine to poke at. Debian Wheezy amd64
  1. Fetch normal Debian Wheezy installation ISO.
  2. Boot from it, either as CD or USB stick.
  3. Press [tab] on the boot prompt and add this as a boot argument to the Debian installer:
    url=http://www.reinholdtsen.name/freedombox/preseed-wheezy.dat
  4. Answer the few language/region/password questions and pick disk to install on.
  5. When the installation is finished and the machine have rebooted a few times, your Freedombox is ready for testing.
Raspberry Pi Raspbian
  1. Fetch a Raspbian SD card image, create SD card.
  2. Boot from SD card, extend file system to fill the card completely.
  3. Log in and add this to /etc/sources.list:
    deb http://www.reinholdtsen.name/freedombox wheezy main
    
  4. Run this as root:
    wget -O - http://www.reinholdtsen.name/freedombox/BE1A583D.asc   \
       apt-key add -
    apt-get update
    apt-get install freedombox-setup
    /usr/lib/freedombox/setup
    
  5. Reboot into your freshly created Freedombox.
You can test it on other architectures too, but because the freedombox-privoxy package is binary, it will only work as intended on the architectures where I have had time to build the binary and put it in my APT repository. But do not let this stop you. It is only a short "apt-get source -b freedombox-privoxy" away. :) Note that by default Freedombox is a DHCP server on the 192.168.1.0/24 subnet, so if this is your subnet be careful and turn off the DHCP server by running "update-rc.d isc-dhcp-server disable" as root. Please let me know if this works for you, or if you have any problems. We gather on the IRC channel #freedombox on irc.debian.org and the project mailing list. Once you get your freedombox operational, you can visit http://your-host-name:8001/ to see the state of the plint welcome screen (dead end - do not be surprised if you are unable to get past it), and next visit http://your-host-name:8001/help/ to look at the rest of plinth. The default user is 'admin' and the default password is 'secret'.

9 September 2013

Bastian Blank: Setting up Ceph the hard way

Components Ceph consists of two main daemons. One is the monitoring daemon, which monitors the health of the cluster and provides location information. The second is the storage daemon, which maintains the actual storage. Both are needed in a minimal setup.
Monitor The monitor daemons are the heart of the cluster. They maintain quorum within the cluster to check if everything can be used. They provide referrals to clients, to allow them to find the data they seek. Without a majority of monitors nothing will work within the cluster.
Storage The storage daemons maintain the actual storage. One daemon maintains one backend storage device.
Configuration The default config is understandable, but several things will just not work with it.
Monitor on localhost By default the monitor daemon will not work on localhost. There is an (undocumented) override to force it to work on localhost:
[mon.noname-admin]
 mon addr = [::1]:6789
The monitor will be renamed to mon.admin internaly.
IPv6 Ceph supports IP (IPv6) or legacy-IP (IPv4), but never both. I don't really use legacy-IP any longer, so I have to configure Ceph accordingly:
[global]
  ms bind ipv6 = true
One-OSD clusters For testing purposes I wanted to create a cluster with exactly one OSD. It never got into a clean state. So I asked and found the answer in #ceph:
[global]
 osd crush chooseleaf type = 0
Disable authentication While deprecated, the following seems to work:
[global]
 auth supported = none
Complete configuration
[global]
 auth supported = none
 log file = $name.log
 run dir =  
 osd pool default size = 1
 osd crush chooseleaf type = 0
 ms bind ipv6 = true
[mon]
 mon data =  /$name
[mon.noname-admin]
 mon addr = [::1]:6789
[osd]
 osd data =  /$name
 osd journal =  /$name/journal
 osd journal size = 100
[osd.0]
 host = devel
Installation This is currently based on my updated packages. And they are still pretty unclean from my point of view.
Setup All the documentation tells only about ceph-deploy and ceph-disk. This tools are abstractions that need root to mount stuff and do all the work. Here I show how to do a minimal setup without needing root.
Keyring setup For some reason even with no authentication the monitor setup wants a keyring. So just set one up:
$ ceph-authtool --create-keyring keyring --gen-key -n mon.
$ ceph-authtool keyring --gen-key -n client.admin
Monitor setup Monitor setup by hand is easy:
$ mkdir $mon_data
$ ceph-mon -c ceph.conf --mkfs --fsid $(uuidgen) --keyring keyring
After that just start it:
$ ceph-mon -c ceph.conf
$
OSD setup First properly add the new OSD to the internal state:
$ ceph -c ceph.conf osd create
$ ceph -c ceph.conf osd crush set osd.0 1.0 root=default
Then setup the OSD itself:
$ mkdir $osd_data
$ ceph-osd -c ceph.conf -i 0 --mkfs --mkkey --keyring keyring
And start it:
$ ceph-osd -c ceph.conf -i 0
starting osd.0 at :/0 osd_data $osd_data $osd_data/journal
$
Health check The health check should return ok after some time:
$ ceph -c ceph.conf health
HEALTH_OK
$

Joey Hess: a rope ladder to the dgit treehouse

Ian has said something along the lines of dgit being a private club; if you're not in the club the strange git-centric views of its members don't affect you. With that said, dgit is now a quite usable and nice tool and we would like more people in the club. To that end, here's one way to get into our little world where the Debian source repository is really a collection of git repositories. I recently made a new package and added it to Debian, and here is every command I typed to do it (other than vim, lintian, and dpkg -i). As you read through, notice that every command has to do with git in one way or another. git clone https://github.com/blake2-ppc/git-remote-gcrypt This software makes git be able to push to encrypted git remotes. Very neat stuff. git archive --format=tar origin/master -o gzip > ../git-remote-gcrypt_0.20130908.orig.tar.gz Like most software on github, this one has no upstream tarball; its git repository is its native source format. Needing to manually generate this tarball appendix is the only unpleasant part of the process. git add debian
git commit -m debianization Had to write 48 lines of control file and stuff to make the package build. dgit build -tc This just runs dpkg-buildpackage. But you need to commit all files before running it. dgit push --new The --new is needed because this is a new package. This created the git repository for the package on alioth and pushed my changes and the upstream history to it, as well as uploadng the package to incoming.

Next.

Previous.