Some of my Romanian readers might know that for the last two years I've got involved in the skeptical movement to such a degree that I am a co-producer of a bi-weekly podcast on science and skepticism (in Romanian) called Skeptics in Romania . Some might even be regular listeners of the show.

(There isn't much to see now visually on the site, but me and the other people behind the project have some ongoing plans to change that.)

In spite of our modest site, up until now we had some successes, one of them being the publication of an article on us in a known Romanian printed publication and another being the invitation to a live show face to face with Oreste Teodorescu, a well known Romanian mysticist and woo promoter.

During that live show we managed to show a demonstration (video below, in Romanian) of how astrology gives the impression of working, without actually working, and, taking into account we had no prior TV camera experience and that it was a live show, I think we managed an honourable presence.


We also have a series of interviews in English with some really interesting people: Dr. Eugenie Scott, Prof. Christopher French, Prof. Edzard Ernst, Samantha Stein and others. We did these interviews at Denkfest 2011, in Zurich, and we integrated the translated (voice over) interviews in our podcast. The conlusion is that most of our activities revolve around the podcast, so let me tell you more about that.

The podcast has a somewhat fixed structure, it starts with a conversation between ourselves, then we have a segment on the history of science, technology, skepticism and woo, and then we have a segment called The dangers of not being skeptical . In this segment we present cases of people who lost their lives, their health, their money or any combination of the former because they were duped into some scam, science-y sounding non-science, unfounded claim or some other woo.

Having lost recently my brother-in-law to a form of cancer known as Hodgkin lymphoma, I became especially sensitive about miracle-cure claims for cancer, and this section of the show has lately seen its fair share of such cases. Honestly, if there could be a way to prosecute the irresponsible, ignorant and/or cynical people promoting all sorts of quack "therapies", especially for cancer*, I would really like to see it happen. But there isn't, and we're trying the best accessible approach, informing the public.

During my brother-in-law's last two years of his life, he went through lots of chemotherapy and radiotherapy sessions, repeated periods of hospitalisation, and lots of drugs. This is the best of what we currently have for treating and curing most forms of cancer, and too many times this isn't enough. I can't even imagine how stressful and discouraging it must feel when the best of what we have doesn't help.

Here is where the desperation and hopes of patients and their families meet the purely irresponsible cynical or ignorant promoters of woo and quack therapies. Because it takes either an ignorant or a really cynical (I really feel this word isn't enough) person to prey on the suffering of other people to make easy money under the false pretence of offering a cure.

It almost happened to my brother-in-law and his family, because they almost went for some herbal concoction promoted as a cancer cure on some forum, blog or page of a seller of this fake therapy. It was really hard for me to make them understand why using such a product it not advisable, not even in parallel with the medical treatment due its possible counter effect or interactions with the real medical treatment, without them getting the wrong idea that I wasn't trying to help. While trying to be brief and informative not to lose their attention, I told them how "natural" doesn't necessarily mean "good" (uranium, lead and Irukandji's venom are all natural), and how plants are drugs because they all contain chemical substances (and no, "chemical" does not mean "human made" or "artificial") which could interact with the medical treatment.


But most people don't even have the chance of having close by a person with a more science-leaning thought process and a skeptical mind. Those unfortunate people are the most vulnerable people and constitute the biggest chunk of the victims of baseless pseudo-cures or pseudo-treatments.

On our last show, I presented the case of Yvonne Main, a cancer suffering patient who mistook an invasive carcinoma for a cyst, and irdologist Ruth Nelson for a real healthcare giver.

Yvonne Main, died from an invasive carcinoma after seeking help from a iridologist, and delaying real medical treament for 18 months

Yvonne, after seeking medical advice from a person that essentially promotes the dead idea of guessing diseases by looking at the eyes**, used natural treatments for about 18 months and, after all this time, her carcinoma grew to a size of 10 to 11 cm, eating through her skull and causing damage which was later attempted to be countered through bone transplant from her ribs.

Ruth Nelson wasn't prosecuted in any way and continues her practice of quackery unharmed.

This is not the only case, nor even one case from a select few where woo and quackery lead to grave consequences for patients. There are many, many more; they're so many that even after splitting them in categories they seem too many per category, especially when you realise these are only the findings of, essentially, a single man:




This is part of what I have been doing in the last few years, instead of working on Debian. Is it a good thing? Is it a bad thing? Maybe it's good. I want to know what do you think?


* you will, most likely, never hear such a promoter of non-therapies say that there isn't just one cancer, and that, in fact, cancer is a name for a certain family of diseases which are all called cancer - that's a first sign that you might be dealing with quack
** probably in the line of thought that the eyes are the gates to the soul so they must tell something significant about health

I was pointed at a great online course If you are into e-voting analysis (or, more broadly, into democratic processes' history, evolution and future), I strongly suggest you to take a look at Securing Digital Democracy . Just the name of the teacher should be enough to make it interesting: University of Michigan professor J. Alex Halderman, the guy who has analized/hacked several electronic booths, and one of the clearest, smartest voices to explain what should we require of a voting system and how electronic booths are the worst fit for any purpose. The course is delivered through Coursera; I have found Coursera to be an effective, usable, unobtrusive platform So much I even signed up for another course. I am not so happy with online courses requiring to wait so much between lessons, but after all, it tries to mimic what we see at "regular" (i.e. classroom) teaching settings. And, after all, we autodidacts are still a minority. The course in question started ten days ago, but you can still perfectly join. Each week has two lessons, worth of approximately 40 minutes of video each, and are "graded" through a quiz. Lets see how this evolves.

Here s a summary of what happened at this conference. Opening address. Keynote from Nick Hurd MP, @minforcivsoc. Lot less money around. Called canals and rivers trust the biggest social enterprise. Blames lack of large social enterprises on culture, leadership and access to capital. First Big Society Capital investments announced tomorrow. Questions (and responses) about what had changed in commissioning (determination?), something about accountability in health I think but I didn t hear (more public scrutiny), other models besides worker-led (gov is agnostic, but that s just the type of mutuals so far), and capital renewal (gov is challenging the banks with Big Society Bank). This is going to get a bit long, so click through to our site to read what the other the speakers had to say and what that block of flats in the picture has to do with it Still here? Good! Harriet Hounsell from John Lewis Partnership was next. What s the Right level of profit? Structure of the partnership. The registry seems like the member services function and a bit more. Partner suggestions make a real difference. Beliefs have been tested. Recession hit us. Vivian Woodall, the co-operative phone, back to how it started. He knew people working for NGOs with high international phone bills. Bulk buy. Solid steady growth. Paid all tax. Living our cooperative values. Employee council and profit-shared. Adoption of new brand. Questions to both on redundancies (debate openly, try to redeploy, ultimately division of responsibilities: managers manage staff levels, not members), and being rejected as too small to help other social enterprises (ask why? Try local level instead of head office?), plus praise for John Lewis. Then there was coffee, workshops, lunch, workshops and coffee. I don t think I should report all of that, so a quick comment on the venue at Anglia Ruskin Uni Cambridge: is the Lord Ashcroft it s named after the same one who said it was legal to attack Iraq? Anyway Based on the workshops, I don t think our co-op will do SROI accounting any time soon (needs too much people time) and I m little the wiser on leadership. Some people are intrigued by collective management and cooperative working, though. The closing keynote was Wayne Hemingway, formerly of Red or Dead and now of Hemingway Design. He gave a quick history of himself and of those two enterprises and a bit of an entertaining ramble around. He seemed very much from the business end and how to use business to benefit society, but I ve not really looked at his companies before. I probably should do, as they ve a regeneration project in Lynn (pictured: before Wayne). Then it was the closing address, the thanks (yes, thanks to the workers and sponsors) and then time to head for the exit. It s a shame that some people who made good points during the keynotes (coop party!) left quickly instead of staying to chat.

It happens occasionally that I need to validate an optimization by benchmarking, and the benchmark results are a bit noisy. Usually the situation does not ask for a heavy-duty statistical analysis, and I am too lazy for that anyway. However, sometimes having at least the fundamental statistics of a sample would be handy.

Obviously, calculating the standard deviation of a sample in the 21st century by itself is not a problem. There is LibreOffice, there is R, there's a bazillion statistical tools out there. They do feel a bit heavyweight for the task though, and the friction involved in starting them up and entering the data in the right format means that often I just do not bother. Google brings up some dynamic webpages for the task, a much more lightweight solution, but most of these are ugly and ad-riddled.

What's a programmer to do? Code up another weekend app, statsb.in, of course. And it is the way I like it: there is a textarea where it is easy to paste and edit inline snippets of text, say, shell script output in full; non-numeric data are ignored; statistics are computed on the fly (no stupid "Submit" buttons); and you can save the snippet on the server and get a short link (useful for footnotes pointing to data sources).

The service probably will not be expanded much further. After all, if those basic statistics are not enough, you are probably better off firing up a real statistics tool. However, I do plan to add another page for the most common task that I run into, namely, comparing the means of two populations (did my optimization have an effect?). In other words, calculating the p-value of the hypothesis that the means of two populations are the same. Stay tuned.

I'm maintaining a host under acaia.ca. I intend to share it with friends and encourage them to use non-evil services to host their stuff. For now I'm providing the following:

• http://bookmarks.acaia.ca using semanticscuttle
• https://clipperz.acaia.ca/beta/ using clipperz
• http://owncloud.acaia.ca/ using owncloud
• ikiwiki for personal website and blog

...and testing these:

• https://contacts.acaia.ca/ using PHP iAddressBook
• http://minigal.acaia.ca/ using a minigal
• https://money.acaia.ca/html/ using Personal Finance Manager

As I'm not happy with minigal I'm trying mediagoblin, Zenphoto and Plogger for media hosting. Any other idea? I want light and simple stuff, my VPS is cheap. (btw, I didn't renew my flickr pro last month so I'm very likely leaving this service. I used flickrbackup to have all my pictures and some of meta-data saved, as they didn't provide me any better way to do a backup.)

Seems I've forgotten to announce Enca 1.13 here, but I won't do the same mistake with 1.14, which has been released today. If you don't know Enca, iit is an Extremely Naive Charset Analyser. It detects character set and encoding of text files and can also convert them to other encodings using either a built-in converter or external libraries and tools like libiconv, librecode, or cstocs. It's code is currently available at Gitorious. The full changes for 1.14 release are short:

• Allow standard names for belarusian and slovenian languages, thanks to Branislav Ger o for suggestion.
• Reset strictness when check buffer less than file size, thanks to Sam Liao.
• Fixed typos in man page, thanks to A. Costa.

Still enca is in maintenance mode only and I have no intentions to write new features. However there is no limitation to other contributors :-). You can download from http://cihar.com/software/enca/.

Filed under: Enca English 0 comments Flattr this!

Lessons learned from trying to do some simple non-linear video editing on Linux: Kdenlive has reached a point where it's usable. Not good by any means there are glaring problems such as the color management support being very rudimentary, almost no multicore support, general lack of optimization, frequent near-crashes, and so on but usable to the point where you can actually create a video from start to end. I'd say it's very roughly comparable to Premiere 5 was, circa 1998, but it's not something I'd recommend to my mother. slowmoVideo, on the other hand, was not very useful to me for framerate up-conversions; the end results were simply too poor given the general complexity in my scenes. It can seemingly do an okay (but not perfect) job for down-conversion if you're willing to hack it a bit and live with general slowness and demand for disk space. So, well, behold the end result, I guess. If you like frisbee, that is.

Sarah is back in Australia attending the graduation ceremony for her Bachelor of Business degree, at the University of Southern Queensland, an Australia University that specialises in distance education. To say I'm immensely proud of her achievement is putting it lightly. Consider for a moment all of the challenges she has had to face in the past 6 years since she started it:

• studying remotely
• working full time (for a bit)
• losing a baby
• having major heart surgery (with a staph infection complication)
• getting her sternal wires removed (with another staph infection complication)
• having a baby
• looking after a baby at home while continuing to study

Me, I'd fail at "studying remotely". Heck, I've had two stabs at a Bachelor of Information Technology on campus, and what do I have to show for it? No, I'm so very very proud of her for sticking at it, and doing so very well (she twice received the Dean's Award for having a high GPA for the semester). She decided to embark on a business degree after having a really hard time finding meaningful work when we moved to the US. She had been working in business project management for the Australian Federal Government (and really enjoying the work) before we moved, and couldn't find any project management that wasn't high-tech, which wasn't really her thing, and everything wanted a degree. She briefly contracted at VMWare, in a marketing position, and Google, in a Security Vulnerability Management role, but neither of these were really up her alley, so to speak. So she went "bugger it" and enrolled in the degree. Ironically, mid-way through the course, she decided that nursing was more what she wanted to do, so she has also gone out and done short courses in health care. She has been licensed as a Certified Nurse Assistant and also a Phlebotomist, but never worked as either. She's just started an Emergency Medical Technician course at Mission College, and is keen to try and work in that field for a little while before starting on a post-graduate Nursing degree in Australia. Sarah had indicated to me that she'd really love to go back for the on-campus graduation ceremony, since she's barely set foot on campus in the six years she's been studying at the university. I thought that was an excellent idea, so I sprang for a Business Class flight back for just her, and am staying behind to look after Zoe. It's only a flying visit. She departed Wednesday night, returns on Monday. As I write, she's probably at the ceremony right now. Normally it would take 6 years to complete a degree externally, assuming two subjects a semester for 12 semesters. She's obviously had to take a few semesters off here and there because of some of the aforementioned challenges. To still finish the degree in 6 years, she's had to plow through the "summer" break (I say "summer" because it's during the Northern Hemisphere winter) and use the accelerated "summer semester", also known as "semester 3" to do an additional subject. That's meant that summers over here have been her shorter winter break. This summer has been the first summer where she hasn't had to worry about studying, and it's been really wonderful to be a whole family again for an extended period of time, and we've had a jam packed summer as a result. We bought a tent and did a small amount of camping. We've had weekends away with friends. We've been to Raging Waters. We've been to Disneyland (granted that was in between final exams). It's been go go go. So now that summer is almost over, I'm looking forward to the pace slowing down a bit more and things returning to "normal" (well the new "normal" is Sarah doing her EMT class a couple of nights a week until December). She's only just started, but she's really loving the on-campus aspect of it. I'm also looking forward to getting a bit more "me" time. Since Zoe was born, I have sacrificed my personal time to facilitate Sarah's study. I haven't had much time to tinker and I've barely had time for Debian. I'm looking forward to trying to carve out a few hours a week for some of both, plus a bit of reading.

This is not the kind of place I live, though I don't mind a visit. I'm not bothered by a more rural life, I thrive on peace and quiet, and am not bothered by needing to haul my water, split firewood, or spend an hour just mowing my driveway. I bought a GPS the other day, and it doesn't know where my driveway is..
or where the road leading to my driveway is..
or even about the highway leading to that road!
The only thing that really bothers me about living here off the map is that I'm in the middle of a food desert too. The nearest semi-decent grocery store and produce market are 40 minutes away. There's one restaurant worth eating at, and while a very congenial place to eat rustic food while watching the river flow by, it serves almost no vegetables. Adding to the problem, my fridge is a rather expensive propane model, which is about half a normal size, and has no freezer. Managing a healthy meal plan is a constant struggle, though I do have time to cook, at least, thanks to not having the kind of commute that is typical around here. Well, the food desert was my only real problem. Now I can add to that the UPS emergency. A week ago, a package left Shenzen China for here. It quickly passed through Anchorage Alaska, and with a few days had arrived an hour and a half from here. Getting it here has taken the whole rest of the week.
I only knew UPS attempted delivery because a neighbor told me. Seems they took one look at my driveway and turned around. They never tried again, and they didn't bother to leave a note on the mailbox, or anything. Now, the US postal service is happy to drive up this driveway; they drive a sensible 4x4 around here. I think DHL and Fedex have been here too. (Contrary to John Goerzen's experiences) My compact car is not bothered by my driveway either... But UPS's brand involves large brown trucks, even where inappropriate, and damage to them from such things as low-hanging branches apparently comes out of the driver's paycheck. And while normally UPS leaves small things in my mailbox, or with a neighbor, this was an important package from China, and it needed a signature. But really, a post card? "emergency"?! So, I bought the GPS to find UPS, because my package was in an industrial park in some town I've never been to. And on Tuesday I was within an hour of there and swung by. And it was closed. UPS's website had not mentioned this; other businesses are open the day after Memorial day. That turned out to be a really annoying day all around. When I got home, after a night away, I found my fridge's fire had blown out, and nearly all my food was only fit for the dog. Today I went out to try again (and buy groceries). When I reached the UPS depot.. it was closed! Until 3 pm. At this point I felt this story was reaching full-on farce. After cooling my heels in a library for three hours, I was finally able to open the most over-packed package I've probably ever seen. My kickstarter reward sample looks good.. I'll be ordering a lot more, but not to this address. As a coda to the UPS farce, arriving home today I checked my mailbox, and found the UPS postcard had finally arrived. It included mention of the 3 pm opening time, and noted that I had until tomorrow before they'd return my package to China. The coda to the coda is that on my way home from this second annoying day delivered courtesy of UPS, I stumbled on a restaurant overlooking a waterfall, just outside the town where I get my groceries. It serves food that satisfies both my Cajun and healthy cravings. Now I just need a larger fridge..

If there s one thing that I ve had missing in my life for way too long, it s hacking on small silly stupid amusing little things. When I was young I used to write dozens of little pointless programs a week and forget about them, only to found them a few months later and be incredibly amused by them. This afternoon I noticed that one of my machines didn t have my default .bashrc loaded, which colourises (or colorizes for Americans or colourizes for Canadians) my shell so that I know on which machine I am so that I don t accidentally do something silly like reboot the wrong host. One problem I ve had with this is that there are limited colours available that makes sense to use for a shell prompt. Right as I was thinking this, I was looking at my Byobu screen sessions and thinking Hmm! I like those logos, maybe I could use them in my shell! . So in just a few minutes I had a little script to help you configure those for your shell. Here is a screenshot of the shell logos in action. On my laptop I use a red background when I have a root shell: This is what the shello script looks like that sets it up: I might get bored by seeing it soon and get rid of it, but at least I had a little fun with it. If you d like to try it out, then you can get the script right here: https://github.com/highvoltage/shello It doesn t do anything like validating input yet, I d also like to add something that creates some kind of a checksum of your hostname and choose a unicode character based on that. Maybe I ll finish it up this weekend, or maybe I ll have new things to write about!

There's been good progress in Fedora's implementation of UEFI Secure Boot, so time for a quick update.
Boot process signingThe infrastructure for signing the bootloader binaries is now implemented. pesign is in the archive and being used to sign shim, grub2 and the kernel. At the moment they're all being signed by test keys, and the private key is actually in the pesign package. This is, obviously, not intended for production use - it's just to ensure that we can build correctly signed images. We've proof-of-concepted signing via cryptographic hardware and will shortly be deploying new build systems dedicated to building the signed binaries. These won't be general access systems and will have a lightly modified mock configuration to ensure that the crypto hardware is available to the build chroots, but otherwise there's nothing special about them.

As far as signing with the Microsoft keys goes, we're in the final round of legal review of the appropriate agreements and will be migrating to a version of shim signed with their key in the near future.
Kernel modificationsThere's two main sets of patches for the kernel. The first is to automatically enable the locking down of various bits of functionality when secure boot is enabled, in order to prevent users (including root) from being able to modify the kernel at runtime. They've just been posted upstream and haven't been dreadfully received, though we'll probably be changing the name of the capability. The main thing that users will notice is that any X drivers that still need direct hardware access (which, on secure boot systems, should be zero - we've implemented native kernel drivers for all the hardware we expect to see there) will fail to work, as will setpci. The plan is for kexec to require that kernel images be signed, but that's requiring rather more reworking of kexec than expected so for now kexec will just be disabled.

The other lump of kernel functionality is the support for module signatures. That had been somewhat blocked based on a lack of consensus between patch authors and upstream, but an agreement got hammered out at Kernel Summit last week and so we should see progress there shortly. Right now we're still looking at modules being signed with a throwaway key, but the plan is still for keys in db (and MOK, if using the Suse approach) to be imported into the kernel keyring and used. That's dependent upon that code being written in time, though. It's definitely still a goal for F18.
Key managementWe're planning on using Suse's approach of permitting local key management at the shim level, and I spent some time discussing this with Vojtech last week. In combination with the above, this should provide a workable mechanism for permitting the end-user to install module signing keys.
General UEFI support improvementsWe've added support to grub to use the kernel's built-in UEFI setup functionality, although there's still some discussion with upstream to ensure that it's implemented in an acceptable way. This approach also makes it easy for us to obtain PCI ROMs via UEFI, which improves radeon support on a bunch of Macs. Finally, we've been hammering on some remaining UEFI bugs and are definitely at the point where the machines that don't work are surprising and the ones that do are the tedious expectation. This is far better than any previous release.
DocumentationStill has to be written, and that's probably where much of the my next month is going.
SummaryFedora 18 is still on track to have full UEFI Secure Boot support out of the box, and the Beta should be fully signed although perhaps still with our test keys.

For those of you who get sent to surprisingly expensive conferences, I'll be discussing some of this as part of a presentation on improving Linux support for UEFI at the Intel Developer Forum in San Francisco next Tuesday morning. Feel free to say hi if you're in the area.

comments

Panama just underwent a nasty e-voting exercise: Electronic-mediated elections were held for the committee of the PRD party. It sounds simple - Even trivial! There were only 4100 authorized voters, it was geographically trivial (all set inside a stadium)... But it blew up in smoke. I won't reiterate all what happened, I'll rather direct you to our project's (the e-voting observatorium) page: News regarding Panama (for those coming from the future, search starting at 2012-08-27 and yes, it's all in Spanish, but there are free-as-in-beer translation services. Many e-vote proponents/sellers/pushers were very eagerly waiting for this election to brag about one more success... So much that they could not just ignore it, and started rationalizing it away. Anyway, while feeding the observatorium, I came across this opinion-article in the Voto Digital website, which makes quite a bit of pro-e-voting noise. I replied to it, and I think my analysis is worth sharing also with you:

So, lets make some simple numbers, rounding the numbers: The PRD vote in Panama was done for a universe of 4100 voters. It took 10 hours (instead of the planned 4), so 410 people were processed every hour. There were 40 voting (electronic) booths, so each processed 10 people per hour. This means, each person spent 6 minutes by the booth. A manual vote in this fashion is highly parallelizable: Each of the voters can be given ballots with anticipation, or many of them cna be allowed in to be given the ballots in situ (depending on the electoral scheme employed). The contention time is the time it takes to each voter to get near the booth and deposit his ballot (either folded or in an envelope) - And it will very rarely be more than a couple of seconds. So, given that using electronic booths parallelism cannot grow (there is a fixed number of machines) and the queues grew wildly, with traditional voting it would have surely fitted in the expected four hours (they were expected, also, based on their past experiences). As for counting, that's the slowest part of manual voting, it's also highly parallelizable: If each of the 40 booths has slightly over 100 ballots, the party personnel can easily count them in under 30 minutes. Capture and aggregation for the 40 partial results would take an extra 10 minutes, even being generous. Manual voting would have saved them around five hours, without demanding additional resources (and being thus much more economical than having to buy 40 specific-purpose computers). And as an additional advantage, the physical and tangible vote proofs would remain, in case they were ever again needed.

I've never really used facebook so it's not too hard for me to delete my account there. The only thing that's a pity is that I found (or others found me) many people that I once met and which I'd love to meet again once. So I'll go through my facebook contacts list and will try to copy all contact data especially the email adresses and plan to write emails to everybody every quarter or half a year about happenings in our lifes. I'd be glad to receive such emails from others too! They'll surely be read more carefully then my facebook timeline!

This gets me to another point: I still don't have a satisfying free (as in freedom) solution to manage my contacts across devices and available on my web server. The same for image galleries. So I can understand how convenient facebook for many people is. Still facebook is evil and it's an important and ongoing project to provide a better alternative.

It'll be harder for me to leave Twitter. This one comes next. If people would just switch to the free and privacy aware alternative identi.ca...

And then I've to finally provide a better alternative for my family to share baby pictures then Google Plus... Leaving Xing and Linkedin also doesn't worry me, I just have to collect the addresses of my important contacts there. Not so pressing is to leave Couchsurfing for bewelcome.org.

So if you want to hear news from me just pass by www.koch.ro from time to time or put my blogs feed into your feed reader. I'll also keep my identi.ca account.

update: One tiny example of facebooks evilness is, that they don't show me your real email address (anymore). They only show me an @facebook address. So if I don't know your real email address from any other source, I would not be able to send you messages without sending it over facebooks servers. I read that you can change the email address shown back to your real email address somewhere in the options.

We recently switched from Buildbot to Jenkins at work, for building Varnish on various platforms. Buildbot worked-ish, but was a bit fiddly to get going on some platforms such as Mac OS and Solaris. Where buildbot has a daemon on each node that is responsible for contacting the central host, Jenkins uses SSH as the transport and centrally manages retries if a host goes down or is rebooted. All in all, we are pretty happy with Jenkins, except for one thing: The job configurations are a bunch of XML files and the way you are supposed to configure this is through a web interface. That doesn't scale particularly well when you want to build many very similar jobs. We want to build multiple branches, some which are not public and we want to build on many slaves. The latter we could partially solve with matrix builds, except that will fail the entire build if a single slave fails with an error that works on retry. As the number of slaves increases, such failures become more common. To solve this, I hacked together a crude tool that takes a yaml file and writes the XML files. It's not anywhere near as well structured and pretty as liw's jenkinstool, but it is quite good at translating the YAML into a bunch of XML files. I don't know if it's useful for anybody else, there is no documentation and so on, but if you want to take a look, it's on github. Feedback is most welcome, as usual. Patches even more so.

On year ago, we (the mancoosi team) published a comparison study regarding the state of the art of dependency solving in debian. As few noticed, the data presented had few glitches that I promised to fix. So we've repeated our tests using exactly the same data we used one year ago, but now using the latest available versions of all package managers as available in debian unstable. During the last year, three out of four solver that we evaluated release a major upgrade so I expected many improvements in performances and accuracy.

• apt-get 0.8.10 -> 0.9.7
• aptitude 0.6.3 -> 0.6.7
• smart 1.3-1 -> 1.4
• cupt 1.5.14.1 -> 2.5.6

Mpm, our test-bench for new technologies, changed quite a bit under the wood as a consequence of the evolution of apt-cudf and the recent work done in apt-get to integrate external cudf dependency solvers. Overall the results of our study are not changed. All solvers but mpm, that is based on aspcud, are not scalable as the number of packages (and alternatives) grows. It seems that Smart is the solver that does not give up, incurring in a timeout (fixed at 60 seconds) most of the time. Aptitude is the solver that tried to give you a solution, doesn't matter what and as result providing solutions that do not satisfy the user request for one reason or the other. Apt-get does surprisingly well, but it gives up pretty often showing the incomplete nature of it's internal solver. Cupt sometimes timeout, sometimes gives up, but when it is able to provide an answer it is usually optimal and it is very fast ... Mpm consistently finds an optimal solution, but sometimes it takes really a long time to do it. Since mpm is written in python and not optimized for speed this is not a big problem for us. The technology used by mpm is now integrated in apt-get and I hope this will alleviate this problem. All the details of our study can be found one the Mancoosi Website as usual with a lot of details. For example here you can find the results when mixing four major releases : sarge-etch-lenny-squeeze. Comments are more then welcome.

I have been having quite costly experiments with (embeded) hardware in the last few weeks/months. To cover at least part of the expenses, I am trying out to use Google's AdSense with my blog. I plan on blogging more frequently with the results of the experiments, but it will take me a few days, as I want to offer people only the "meat" of what I actually did, pointing out some pitfalls that made me waste a lot of time. Anyway, back to AdSense, one interesting thing about trying it is that I applied to be part of the program for 3 or 4 times late in August and, due to unspecified reasons, my application was rejected for "not complying with their policies". I read the policies from left to side, from up and down and nowhere I saw anything that my blog infringed. This morning, I tried once again and I'm now in a sort of "probation period" for AdSense, which is progress I think. So far, I only put one advertisement at the top of my pages, but I am curious to know how "targeted" those ads would be to a blog that has articles that were written more than a decade ago, with posts in at least two languages. I guess that this is a weaker version of what Rapha l Hertzog is trying to do with his funding campaigns, except that I only expect to cover the costs of my hardware (even if I am skeptical that I will actually manage to reach that point of self-sufficiency).

Some months ago I have run into a pretty interesting bug while working on a Ubuntu-based remote desktop system. The whole OS was installed on a server somewhere and users could access their desktop remotely. Some call this stuff Desktop-as-a-Service. The operating system we chose was Ubuntu Oneiric (11.10) and the remote access part was implemented with x2go, which uses nxagent to provide NX transport of X sessions. Users could access their Ubuntu machines remotely, with sound, video, youtube, and all you would expect from a desktop machine. The whole thing was working quite well. Now, as I said that was in May. Ubuntu 12.04 LTS was available, and the choice of upgrading to it sounded pretty much obvious. So we upgraded our test system to Precise and everything seemed to work smoothly. Till we tried to open a PDF document, actually. It took evince about 50 seconds to display a relatively small document. Same story with images opened with the default image viewer, eog. The fix delivered to our users was simple: we have set shotwell-viewer as the default image viewer, and epdfview as the default PDF viewer. Everybody happy. In the meantime, obviously, I was interested in this issue so I ve tried to run run evince from a terminal, getting the following output:

(evince:15833): GRIP-WARNING **: Failed to initialize gesture manager.

Funny. On another test system running Debian Sid (unstable) everything was working smoothly. The diff between Ubuntu s version of evince and Debian s is a 6MB monster. Among other changes, I noticed that Ubuntu s version build-depends on libgrip-dev, which depends on libutouch-geis. Multitouch stuff. Why should multitouch support break my remote session? So on May the 10th I filed a bug on launchpad. How this issue got handled is in my opinion one of the many fine examples of the inherent superiority of free software, coupled with a we won t hide problems mindset. For an example of how bad is the proprietary approach, just check a random bug in Adobe s bug tracking system. But let s go back to the evince bug. Other users reported that their VNC sessions were also affected by the same problem. After a few days it was clear that the culprit was utouch-geis, and a patch appeared. Unfortunately it did not actually address the issue. Somebody else reported that RDP sessions were broken too. At the beginning of June Precise was still affected. Finally, on August the 6th a working patch was submitted by Bradley M. Froehle and included by Chase Douglas (thank you guys). End of August, fixed version of geis accepted into precise-updates, case closed. Now for some considerations. The problem was clearly of a certain importance. A Long Term Support, stable version of Ubuntu shipped with broken PDF and image viewing functionalities. It got fixed properly, even though 3 good months are quite a long time for such a bug to get solved. However, the issue only affected a pretty limited number of users, also certainly not Ubuntu s main target audience. This bug never affected Debian, simply because utouch-geis has not made its way into the archive yet. It takes longer to make changes like this in Debian, but for some categories of users stability is more important than new, cool features. Choice is a good thing.

ELKI is a data mining software project that I have been working on for the last years as part of my PhD research. It is open source (AGPL-3 licensed) and avilable as both a Debian package and Ubuntu package in the official repositories. So a simple aptitude install elki should get you going and give you a menu entry for ELKI. These packages come with the scripts elki to launch the MiniGUI and elki-cli to run from command line. The key feature that sets ELKI apart from existing open source tools used in data mining (e.g. Weka and R) is that it has support for index structures to speed up algorithms, and a very modular architecture that allows various combinations of data types, distance functions, index structures and algorithms. When looking for performance regressions and optimization potential in ELKI, I recently ran some benchmarks on a data set with 110250 images described by 8 dimensional color histograms. This is a decently sized dataset: it takes long enough (usually in the range of 1-10 minutes) to measure true hotspots. When including Weka and R in the comarison I was quite surprised: our k-means implementation runs at the same speed as Rs implementation in C (and around twice that of the more flexible "flexclus" version). For some of the key agorithms (DBSCAN, OPTICS, LOF) we are an order of magnitude faster than Weka and R, and adding index support speeds up the computation by another factor of 5-10x. In the most extreme case - DBSCAN in Weka vs. DBSCAN with R-tree in ELKI - the speedup was a factor of 330x, or 2 minutes (ELKI) as opposed to 11 hours (Weka).
The reason why I was suprised is that I expected ELKI to perform much worse. It is written in Java (as opposed to R's kmeans, which is in C), uses a very flexible architecture which for example does not assume distances to be of type double and just has a lot of glue code inbetween. However, obviously, the Java Hotspot compiler actually lives up to its expectations and manages to inline the whole distance computations into k-means, and then compiles it at a level comparable to C. R executes vectorized operations quite fast, but on non-native code as in the LOF example it can become quite slow, too. (I would not take Weka as reference, in particular with DBSCAN and OPTICS there seems to be something seriously broken. Judging from a quick look at it, the OPTICS implementation actually is not even complete, and both implementations actually copy all data out of Weka into a custom linear database, process it there, then feed back the result into Weka. They should just drop that "extension" altogether. The much newer and Weka-like LOF module is much more comparable.) Note that we also have a different focus than Weka. Weka is really popular for machine learning, in particular for classification. In ELKI, we do not have a single classification algorithm because there is Weka for that. Instead, ELKI focuses on cluster analysis and outlier detection. And ELKI has a lot of algorithms in this domain, I dare to say the largest collection. In particular, they are all in the same framework, so they can be easily compared. R does of course have an impressive collection in CRAN, but in the end they do not really fit together. Anyway, ELKI is a cool research project. It keeps on growing, we have a number of students writing extensions as part of their thesis. It has been extremely helpful for me in my own research, as I could quickly prototype some algorithms, then try different combinations and use my existing evaluation and benchmarking. You need some time to get started (largely because of the modular architecture, Java generics and such hurdles), but then it is a very powerful research tool. But there are just many more algorithms, published sometime, somewhere, but barely with source code available. We'd love to get all these published algorithms into ELKI, so researchers can try them out. And enhance them. And use them for their actual data. So far, ELKI was mostly used for algorithmic research, but it's starting to move out into the "real" world. More and more people that are not computer scientists start using ELKI to analyze their data. Because it has algorithms that no other tools have. I tried to get ELKI into the "Google Summer of Code", but it was not accepted. But I'd really like to see it gain more traction outside the university world. There are a number of cool projects associated with ELKI that I will not be able to do myself the next years, unfortunately.

• A web browser frontend would be cool. Maybe even connected to Google Refine, using Refine for preprocessing the data, then migrating it into ELKI for analysis. The current visualization engine of ELKI is using SVG - this should be fairly easy to port into the web browser. Likely, the web browers will even be faster than the current Apache Batik renderer.
• Visual programming frontend. Weka, RapidMiner, Orange: they all have visual programming style UIs. This seems to work quite well to model the data flow within the analysis. I'd love to see this for ELKI, too.
• Cluster/Cloud backend. ELKI can already handle fairly large data sets on a big enough system. If someone spends extra effort on the index structures, the data won't even need to fit into main memory anymore. Yet, everybody now wants "big data", and parallel computation probably is the future. I'm currently working on some first Hadoop YARN based experiments with ELKI. But this is a huge field, turning this into true "elk yarn". I will likely only lay some foundations (unless I get funding to continue as a PostDoc on this project. I sure hope to get to do at least a few years of postdoc somewhere, as I really enjoy working with students on this kind of project)
• New visualization engine. The current visualization engine, based on Apache Batik and SVG is quite okay. It does what I need, which is to get a quick glance at the results and the ability to export them for publications in print quality. (in particular, I can easily edit the SVG files with Inkscape) But it is not really something fancy (although we have a lot of cool visualizations). And it is slow. I havn't found a portable and fast graphics toolkit for Java yet that can produce SVG files. There is a lot of hype around processing, for example, but it seems to be too much about art for me. In fact, I'd love to use either something like Clutter or Cairo. But getting them to work for Windows and Mac OSX will likely be a pain.
• Human Computer Interaction (HCI). This is in my opinion the biggest challenge we are facing with all the "big data" stuff. If you really go into big data (and not just run Hadoop and Mahout on a single system; yes - a lot of people seem to do this), you will at some point need to go beyond just crunching the numbers. So far, the challenges that we are tackling are largely data summarization and selection. TeraSort is a cool project, and a challenge. Yet, what do you actually get from sorting this large amount of data? What do you get from running k-means on a terabyte? When doing data mining on a small data set, you quickly learn that the main challenge actually is preprocessing the data and choosing parameters the right way so that your result is not bogus. Unless you are doing simple prediction tasks, you often don't have a clearly defined objective. Sure, when predicting churn rates, you can hope to just throw all the data into a big cloud and hope you get some enlightement out. But when you are doing cluster analysis or outlier detection - unsupervised methods - the actual objective by definition cannot be hardcoded into a computer. The key objective then is learn something new on the data set. But if you want to have your user learn something on the data set, you will have to have the user guide the whole process, and you will have to present results to the user. Which gets immensely more difficult with larger data. Big data just does no longer look like this. And neither are the algorithms as simple as k-means or hierarchical clustering. Hierarchical clustering is good for teaching the basic ideas of clustering. But you will not be using a dendrogram for a huge data set. Plus, it has a naive complexity of O(n^3) and for some special cases O(n^2) - too slow for truly big data.
  For the "big data future" once we get over all the excitement of being able to just somehow crunch these numbers we will need to seriously look into what to do with the results (in particular, how to present them to the user), and how to make the algorithms accessible and usable for non-techies. Right now, you cannot expect a social sciences researcher to be able to use a Hadoop cluster. Yet to make sense of the results. But if you are a smart guy to actually solve this, and open up "big data processing" to the average non-IT user, this will be big.
• Oh, and of course there are just hundreds of algorithms not yet available (accessible) as open source. Not in ELKI, and usually not anywhere else either. Just to name a few from my wishlist (I could probably implement many of them in a few hours in ELKI, but I don't have the time to do so myself, plus they are good student or starter project to get used to ELKI): BIRCH, CLARA, CLARANS, CLINK, COBWEB, CURE, DOC, DUSC, EDSC, INSCY, MAFIA, P3C, SCHISM, STATPC, SURFING, ... just to name a few.

If you are a researcher in cluster analysis or outlier detection, consider contributing your algorithms to ELKI. Spend some time optimizing them, adding some documentation. Because, if ELKI keeps on growing and gaining popularity, it will be the future benchmark platform. And this can give you citations, which are somewhat the currency of science these days. Algorithms available in the major toolkits just do get cited more, because people compare to them. See this list for an overview of work cited by ELKI - scientific work that we reimplemented at least to some extend for ELKI. It is one of the services that we provide with ELKI for researchers: not only the algorithm, but also the appropriate citation.

This article documents how I'm currently building my firewalls. It builds on netfilter-based-port-knocking, and tries to integrate several components of a firewall as gracefully as possible. For some context: I'm getting involved with servers where the firewall policy goes beyond a handful of SSH from my home and my other servers rules. There are many different network streams that need to be policed, some of them are common across several (potentially many) servers, and so on. So I'm gradually giving up on my hand-made scripts, and trying out higher-level tools. I settled on FWbuilder, which seems nice. However, it only allows static policies, and I still want to keep dynamic policies such as what fail2ban provides, as well as my own port-knocking system. The problem I had was that fail2ban isn't really made to play nice as part of a complex firewalling setup, my port-knocking system was too tightly integrated within my firewall script, and FWbuilder wasn't too flexible when it came to delegating part of the firewall policy to something external. Fortunately, this was only a perceived problem (or a real problem in my understanding), because it is actually possible to have all three blocks playing nicely together. More context: as usual, I'm focusing on Debian-like systems. More precisely, on those with a Linux kernel; it may be that FreeBSD's firewalling subsystem has a feature comparable to Linux's recent module, but I don't know. Let's start with FWbuilder. This is not the place for a manual, the official documentation is rather complete. I'll assume you have defined most relevant blocks in there: firewall, hosts, IP addresses, services, and so on. You define your static policy with the standard rules. From then on, we want to integrate the external tools for dynamic rules. Step 1: Integrating fail2ban We want fail2ban to have its own playground, so that it doesn't overwrite anything in the standard policy. The trick is to define a new policy rule set named fail2ban. Leave it empty in FWbuilder. So far so good, but fail2ban (the daemon) still operates on the INPUT chain in the firewall, and could therefore still mangle the static rules. Fortunately, starting with fail2ban 0.8.5 (available from Debian Wheezy, or in the backports for Squeeze), you can define what chain to operate on: with a configuration item such as chain = fail2ban, fail2ban (the daemon) will now only add its rules to fail2ban (the firewall chain), and won't be able do damage the other chains. The missing part is to send some of the traffic to it using the standard policy: i defined a rule sending the incoming SSH connections to the fail2ban policy ( branching in FWbuilder jargon). Voil : the static policy delegates part of the decision-making to a sub-policy controlled by the fail2ban daemon. Step 2: Integrating port-knocking This is a bit trickier, but we'll use a similar method. First, the traffic used for port-knocking needs to be directed to the chain that does the listening. Define a policy rule set named portknocking, and leave it empty in FWbuilder. It'll be used by the dynamic rules to track progression of source IP addresses through the port-knocking sequence, so you'll need to send ( branch ) incoming traffic there, probably after the rules allowing incoming connections from known hosts. The dynamic part of this will only concern the refreshing of this listening chain , which we assume will do its work and mark IP addresses with PK_ESTABLISHED once the sequence is completed. What we do with these marked IP addresses will still be defined within the FWbuilder policy. We're going to need some complex rules since we want to filter according to this PK_ESTABLISHED bit and according to destination port, for instance; unfortunately FWbuilder doesn't allow combining filter criteria with and, so we define a new policy rule set called accept_if_pk_ok. This ruleset has two rules: the second is an ACCEPT and should be easy to understand. The first rule needs to ensure the ACCEPT is only reached for connections coming from PK_ESTABLISHED addresses, so it's going to be a bit tricky.

• The service needs to be a custom service (I called it PK not established ), since FWbuilder doesn't know about the marking feature in iptables. Use -m recent ! --rcheck --seconds 86400 --name PK_ESTABLISHED for the definition (change the duration to the number of seconds the door should stay open after the port-knocking sequence has been completed). Note the exclamation mark.
• The action is also going to be custom, defined as -j RETURN. Again, this feature is iptables-specific and FWbuilder doesn't provide any UI for it.

(Explanation: the first rule matches packets coming from IP addresses not marked as PK_ESTABLISHED, and returns them to the calling policy. Packets remaining after this rule are those coming from the appropriate addresses, and they go on to the ACCEPT. We could have had the first rule match on IP addresses that are marked, and branch to yet another ruleset with the ACCEPT part, but that would make it harder to read, I feel.) Now let's get back to the main policy and add rules concerning what kind of traffic we want to allow once the port-knocking sequence completed. For instance, we define a rule matching on the SSH service , where the action is to branch to accept_if_pk_ok. When an incoming packet tries to establish a connection to the SSH port, it's passed to the accept_if_pk_ok ruleset. If it comes from the same IP as a recent port-knocking sequence, it goes on to be ACCEPTed. If not, it returns to the main policy. Maybe static rules further on will allow it to go through. Step 3: tying it all together Now that we have all the pieces, the rest is plumbing.

• Get FWbuilder to compile a script from the data. I called mine $hostname.fw, and stored it into /usr/local/sbin.
• Write a /usr/local/sbin/port-knocking script that operates on the portknocking chain and manages the PK_ESTABLISHED bit. It need not do more than what's described in netfilter-based-port-knocking.
• Write an initialisation script that calls both /usr/local/sbin/$hostname.fw and /usr/local/sbin/port-knocking. I called mine /etc/init.d/firewall.
• Make sure that fail2ban's initialisation script is called after ours. Either with boot sequence numbers, or with the LSB dependency pseudo-headers: I made my firewall script to Provides: iptables; since fail2ban's script declares that it Should-Start: [ ] iptables, we're fine.
• Run /usr/local/sbin/port-knocking every hour, or as often as needed to recalculate the port numbers.

With this setup, at boot time, the $hostname.fw script creates the static policy and the extra playgrounds; then the port-knocking script implements the listening for the magic sequence; then fail2ban inserts its own rules. And there we are: three different parts for the firewall policy, all integrating nicely. Mission accomplished! Note: (Mostly copy-and-pasted from the previous article) This article is deliberately short on details and ready-to-run scripts. Firstly because firewall scripts vary wildly so any script would have to be adapted anyway, but mostly because security is best handled with one's brain switched on. Fiddling with a firewall can easily open gaping holes or lock everyone out. So please make sure you understand what goes on before blindly pasting stuff into your own setup. Some bits are left as an exercise to the reader.

---