Search Results: "ed"

14 September 2013

Lars Wirzenius: Ending squeeze support in my software

I have upgraded all of my own computers from Debian 6.0 (squeeze) now. I no longer use it myself at all. Some of my software already doesn't work on squeeze (particularly, yarn, which is part of cmdtest). I do not want to spend my time finding ways to support, say, the python-markdown library version that is in squeeze: I would rather make functional improvements than support ancient software versions. I am hereby ending my support of squeeze in my own software. This means I will be removing squeeze from my apt repository on code.liw.fi, and will no longer test that my software works in squeeze, and will not be making packages for squeeze. (I am removing squeeze from the apt repository so that people don't wrongly assume it is being kept up to date.) This change affects users of Obnam. Obnam was not part of the squeeze release of Debian, but I have maintained packages of it for squeeze on code.liw.fi. If any of my software that is packaged in squeeze has security bugs, or bugs that would be considered release-critical by Debian, I will fix those, but only by making the minimal necessary change to the version in squeeze rather than backporting the current version. Debian is committed to support squeeze for several more months, and I will help Debian do that for my packages.

Steve Kemp: CIDR-matching, in node.js

I recently mentioned that there wasn't any built-in node.js functionality to perform IP matching against CIDR ranges. This surprised me, given that lots of other functionality is available by default. As a learning experience I've hacked a simple cidr-matching module, and published it as an NPM module. I've written a few other javascript "libraries", but this is the first time I've published a module. Happy times. The NPM documentation was pretty easy to follow: Now I can take a rest, and stop talking about blog-spam.

Joachim Breitner: Adding safe coercions to Haskell

Yesterday, I pushed my first sizable contribution to GHC, the Haskell compiler. The feature solves the problem that newtypes are not always free: If we have newtype Age = MkAge Int, then we have all learned that the Age function has zero run-time cost. But if the Int that we want to convert is inside another type, the conversion is no longer free: Converting a Maybe Int to a Maybe Age using, for example, fmap Age, will cause time and space overhead at runtime, and there was no way around it. Well, there is unsafeCoerce, but really, that ought to be avoided. So after some discussion with and encouragement of Simon Peyton Jones at RDP in Eindhoven this year I worked on a design (which was developed, as far as I know, by Simon, Roman Cheplyaka, Stephanie Weirich, Richard Eisenberg and me). In GHC 7.8, there will be a function coerce :: Coercible a b => a -> b that works, from the user point of view, like unsafeCoerce (i.e. no run-time cost), but with the big difference that it will only typecheck if the compiler can infer that it indeed is safe to coerce between a and b. So it will derive Coercible Age Int, and Coercible Int Age, and Coercible (Maybe Age) (Maybe Int) and even stuff like Coercible (Int -> Age) (Age -> Int), but not Coercible Int Bool. It will also not coerce between Age and Int if the constructor MkAge is not exported, to respect module boundaries. Under the hood this relies on the also new feature of roles, which were solved to make the previously unsafe GeneralizedNewtypeDeriving feature safe again, and which also guarantee that coerce is indeed as safe as the name suggests. The feature will come with 7.8, but not fully advertised , so things might change again for 7.10, and bugs with the feature may not necessarily qualify to be fixed in further 7.8.x releases, so beware. It also does not automatically convert fmap Age into coerce, but it is a step in that direction.

Russell Coker: Is Portslave Still Useful?

Portslave is a project that was started in the 90 s to listen to a serial port and launch a PPP or SLIP session after a user has been authenticated, I describe it as a project not a program because a large part of it s operation is via a shared object that hooks into pppd, so if you connect to a Portslave terminal server and just start sending PPP data then the pppd will be launched and use the Portslave shared object for authentication. This dual mode of operation makes it a little tricky to develop and maintain, every significant update to pppd requires that Portslave be recompiled at the minimum, and sometimes code changes in Portslave have been required to match changes in pppd. CHAP authentication was broken in a pppd update in 2004 and I never fixed it, as an aside the last significant code change I made was to disable CHAP support, so I haven t been actively working on it for 9 years. I took over the Portslave project in 2000, at the time there were three separate forks of the project with different version numbering schemes. I used the release date as the only version number for my Portslave releases so that it would be easy for users to determine which version was the latest. Getting the latest version was very important given the ties to pppd. When I started maintaining Portslave I had a couple of clients that maintained banks of modems for ISP service and for their staff to connect to the Internet. Also multi-port serial devices were quite common and modems where the standard way of connecting to the Internet. Since that time all my clients have ceased running modems. Most people connect to the Internet via ADSL or Cable, and when people travel they use 3G net access via their phone which is usually cheaper, faster, and more convenient than using a modem. The last code changes I made to Portslave were in 2010, since then I ve made one upload to Debian for the sole purpose of compiling against a new version of pppd. I have no real interest in maintaining Portslave, it s no longer a fun project for me, I don t have enough spare time for such things, and no-one is paying me to work on it. Currently Portslave has two Debian bugs, one is from a CMU project to scan programs for crashes that might indicate security flaws, it seems that Portslave crashes if standard input isn t a terminal device [1]. That one shouldn t be difficult to solve. The other Debian bug is due to Portslave being compiled against an obsolete RADIUS client library [2]. It also shouldn t be that difficult to fix, when I made it use libradius1 that wasn t a difficult task and it should be even easier to convert from one RADIUS library to another. But the question is whether it s worth bothering. Is anyone using Portslave? Is anyone prepared to maintain it in Debian? Should I just file a bug report requesting that Portslave be removed from Debian?
Related posts:
  1. New Portslave release after 5 Years I ve just uploaded Portslave version 2010.03.30 to Debian, it replaces...
  2. Debian Work and Upstream Steve Kemp writes about security issues with C programs [1]....
  3. Ethernet Interface Naming As far as I recall the standard for naming Linux...

13 September 2013

Miriam Ruiz: Pink Pony: a 3D multiplayer Tron-like racing game, with ponies

The first time I found this cute game, was around 2009, but for a bunch of reasons I haven t been able to get it into the archive until now. Pink Pony is a cute game in which you have to control a pink pony who s main goal is to last in the game more time than the other ponies. You might remember the game Tron, or the film, in which there were light cycles who left a trace behind, with which the other cycles crashed. Well, this is the similar game concept, only with cute ponies instead of light cycles. You can see a video of the game in action, if you want to get a feeling of how it is. Thomas Weber (Ginko), has published a newer version of the game (1.3.1) a couple of days ago, and that is the one that has entered Debian repositories (sid) today. Thanks, Ginko, for this lovely game! Even though the game might not seem too appealing for some of the adult users of Debian or Ubuntu, maybe some of their kids will like it as much as I do. The game is quite quick, though, so very young kids might have trouble controlling the pony. I would suspect it might be all right for kids above 9 years old, but I haven t been able to test my hypothesis, so if you find out, please tell me :)

Justus Winter: cgroupfs is as cgroupy as it gets...

... at least until the cgroup interface is fixed. So, what can it do? So, what's missing? So, what's wrong with Linux cgroup API? Well for one thing the whole API is underspecified. Yes, there is Documentation/cgroups/cgroups.txt, but that is not a specification, that's a howto at best. Second, the notification API is not particularly nice: 
To register a new notification handler you need to:
 - create a file descriptor for event notification using eventfd(2);
 - open a control file to be monitored (e.g. memory.usage_in_bytes);
 - write "<event_fd> <control_fd> <args>" to cgroup.event_control.
   Interpretation of args is defined by control file implementation;
Seriously? There is a POSIXly way to pass file-descriptors around, but smashing the decimal representation of it into a string is not the way to do that. Linux gets away with this hack because the kernel knows the process who wrote(2) that string in the first place, parse the string into an integer and look it up in the table of file descriptors for that process. Now the trouble for cgroupfs is, that it is not the kernel and even if it were, it wouldn't solve the problem because on Hurd there are no file descriptors (well there are, but that's only to appease all the POSIX programs out there). Instead Hurd has ports, and you can send messages to ports, and this is pretty much everything that you can do on a Mach system. Reading a file works roughly like this:
  1. You open a file and get a port X.
  2. You send a message like "I'm like really interested in the first Y bytes of that file" to X.
  3. Whoever has the receiving end of X (probably the one who gave you X in the first place) answers your request.
Ports look pretty much like file descriptors, they are (usually small) integers, you can make them, destroy them, pass them around easily (yes, ports are first class objects in the Mach messaging system). Everything is implemented atop of this mechanism. It is transport-agnostic, the other end could be on another machine and you wouldn't even know. You can create proxies or filters (in fact, that is exactly how the firewall eth-filter is implemented). It's beautiful and extensible at it's heart, like Lego bricks. So if X were a port to e.g. memory.usage_in_bytes and the cgroups interface would be less braindead^W^Wmore carefully designed so that on Hurd it could be transported like ports usually are, then cgroupfs could in fact use port X' to look up which file the caller is interested in (this is possible because cgroupfs was the one handing out the port in the first place) and generate notifications for that file. This is not possible when X is "serialized for transport" using sprintf because port names are specific for each process, so X != X'. The kernel would do the translation while sending the message, but it obviously cannot do that if the number is carried in a character array. I'm not sure what I'm going to do next week. The gsoc timeline suggests a soft-pencils-down, time to scrub code and write documentation, not sure that this is applicable to me as I have pushed most of my work upstream as early as possible. I guess I will nag Samuel so that he merges the outstanding patches and continue working on my gnumach patch.

Daniel Pocock: Calendar and Contact data with free software in the Smartphone era

Since the rise of smartphones, first Blackberry, then iPhone and now Android, the world at large has had a convenient option to fall back on for management of contact and calendar data. The data itself seems somewhat trivial: many of us software developers have worked with a plethora of obscenely more obscure schemas. It is unlikely that most people would ever have to perform an operation on their address book that requires a statistical toolkit like the R project or any of the other advanced tools we now have for data processing. In terms of data, addresses and phone numbers can appear quite boring. So why hasn't the free software community mastered it and cloud providers are currently running rampant hoovering up the data of billions of users? Ruling out LDAP One problem that people have faced is the existence of different standards. LDAP has quite a high profile and appears to be supported by default in many desktop applications. For this reason, it tends to grab people's attention but after a close look, many people find it is not the right solution. LDAP tends to work well for organisations where the data is centrally managed and in situations where fine-grained access control is not required. LDAP servers often need a capable UNIX administer to manage effectively. To top it off, LDAP itself does not provide an effective solution for two-way synchronization of data with remote and mobile users. The emergence of SyncML, CalDAV, GroupDAV and CardDAV Addressing this final problem (two-way mobile/portable device sync) has brought on the development of further standards. Initially, SyncML (now known as Open Mobile Alliance Data Synchronization and Device Management) grabbed significant attention, partly because it had been included in some of the first generation of smartphones from once-prominent vendors like Nokia. Open source server solutions for SyncML did not gain traction though. The most prominent is Funambol, however, it is not packaged in any major Linux distribution and it requires some basic Java knowledge to maintain it effectively. Like LDAP it appears to be most comfortable in enterprise use cases and doesn't appeal to the people who run a private mail and web server. SyncML has further suffered with the lack of support in more recent generations of smartphones. WebDAV was formalised in 1999 and there are now various derivative standards such as CalDAV and CardDAV which use WebDAV as a foundation. However, standards didn't arrive at this place directly: CalDAV only appeared as RFC 4791 in 2007 and CardDAV as RFC 6352 in 2011. In the meantime, some vendors experiemented with an alternative called GroupDAV. GroupDAV is largely obsolete now and GroupDAV v2 is a subset of CalDAV and CardDAV. Finally gaining traction The good news is, CardDAV and CalDAV are gaining traction, in no small part due to support from Apple and the highly proprietary iPhone. Some free software enthusiasts may find that surprising. It appears to be a strategic move from Apple, using open standards to compete with the dominance of Microsoft Exchange in large corporate networks. Both protocols are supported natively in iPhones and Mac OS X. More exciting, Apple releases their Contacts Server and Calendar Server as open source software, making it easier to look under the hood and observe any deviations from the standards. Apple's adoption of CardDAV and CalDAV has been accompanied by a similar effort in free software, however, support is more patchy. The Mozilla Lightning calendar plug-in for Thunderbird (known as the iceowl-extension plugin for icedove on Debian) has supported CalDAV for some time. In contrast, Thunderbird's address book lacks native support for CardDAV and has other limitations (discussed in detail below). Limited CardDAV support in Thunderbird can be achieved using the third-party SOGo Connector, which is not just for ScalableOGo server. The Evolution PIM appears to support both CardDAV and CalDAV relatively smoothly and appears to be more functional and easier to get started with. Free software servers are also emerging. DAViCal, despite the name, supports both CardDAV and CalDAV. It is a good solution for people who have an existing mail server and don't want to transition to an all-in-one groupware suite. Packages available in Debian make it easy to deploy and get started. It is a PHP solution, using a PostgreSQL database underneath. For people who do want a groupware suite, CardDAV and CalDAV are supported in products like ScalableOGo and Zimbra. For people who don't want a server at all and just want to sync between a desktop and a mobile device, there are options like Calypso. Mobile support: the missing link A number of people have blogged about using the Android apps CardDAV-Sync and CalDAV-Sync to complete the picture. I've tested these myself and verified that they work. However, there is one vital thing missing: the source code. Nobody knows what is inside these apps except the developer. That makes it impossible for other developers to troubleshoot them and improve them and it also makes it impossible to be certain that they do not leak information. An alternative is the aCal app for Android. Despite the name, aCal can synchronize the address book as well as the calendar. However, in my tests against DAViCal, I found that the addressbook can only be synced once and that on subsequent syncs, it fails to discover any new server-side contacts. aCal source code is available on gitorious.
aCal appears to be a compelling solution for Android users, hopefully the contact sync will be fixed soon. It only syncs once after installation and then fails to detect any new contacts. Thunderbird's limitations It has already been mentioned that the Thunderbird address book can be synchronized to a CardDAV server by adding the SOGo Connector plugin. However, this is not a perfect solution. The plugin itself is fine: the SOGo web site describes the connector (as opposed to the SOGo Integrator plugin) as a portable solution: in other words, it is vendor neutral, compatible with any standards-compliant server. The first limitation is that it is necessary to duplicate the setup process for both the calendar sync and the address book sync. This is just a question of convenience.
The SOGo Connector plugin configuration window is very trivial. I recommend using read-only mode to avoid the problems with multiple email addresses or telephone numbers. More serious limitations exist in the address book itself: if the CardDAV server has more than 2 email addresses, they will not appear in the Thunderbird address book. Changing an email address on the client side will result in loss of those additional email addresses on the server-side. The Mozilla team initially committed to rewrite the address book from the ground up but it is not clear whether that work is still in progress. DAViCal confusion I had been using DAViCal for quite some time as a CalDAV server before I recently decided to try and use it for CardDAV as well. As a calendar server, it had been working fine. I have a small number of users with some private calendars and also some shared calendars. When I installed the SOGo Connector plugin and went to setup the client-side configuration, it asked me for the server URL. Having used CalDAV already, I simply tried copying the existing CalDAV URL, which is of the form https://myserver.example.org/davical/caldav.php/daniel/home The URL was accepted and I could create address book entries on the client side. No errors appeared. I then tried setting up the same URL as an address book in Evolution. This time, I found I was able to create contacts and sync them to the server and access them from another copy of Evolution. It appeared DAViCal was working fine and I felt that Thunderbird or the SOGo Connector was somehow broken, because the Evolution clients were syncing but Thunderbird was not. The lack of any errors from the Thunderbird GUI further undermined my confidence in the SOGo Connector. Nonetheless, I decided to dig a little deeper, looking through the DAViCal source code. This brought me to the realisation that each URL could be flagged as either a calendar resource or an address book resource. This had not been obvious to me before. Evolution had effectively been storing the contact entries into a calendar URL and DAViCal had been happily accepting them. SOGo Connector, querying the URL, appears to have received both calendar and contact records from DAViCal and given up.
Here is where I configured the extra resource for an address book store under my user account in DAViCal In the DAViCal web management console, I created an extra resource for my user ID and set the appropriate checkbox to mark the resource as an address book. Then I put this resource URL into all the clients and found that they were all able to sync with each other. DAViCal's maintainer has explained that this is a limitation in the CardDAV/CalDAV protocols: technically, it is valid for the client to store the wrong type of objects in each URL. Clients need to be more resilient when they stumble across objects they don't expect in a particular resource. Conclusion A complete free software based contact-management solution can be easily deployed using any of the servers mentioned above, particularly if Evolution is the preferred client software. Using Thunderbird as a client, it can also work, but attention is needed to avoid data loss on sync. Thunderbird could also be used in a read-only mode to access address books maintained with Evolution. Mobile support is still patchy on Android. The closed source solution (CardDAV-Sync and CalDAV-Sync) appears to work and a promising open source close, aCal, appears to be "nearly there". Some people would obviously prefer to use a standalone contact/calendar server, while others may prefer an all-in-one groupware suite. Both approaches appear to be feasible, but the standalone contact/calendar server appears to offer a more compelling way to start quickly and have minimal administrative overhead, particularly due to the fact that DAViCal is available in a supported package.
This is the contact server configuration window in Evolution. It offers slightly more options than SOGo Connector in Thunderbird and appears to be a more finished solution overall.

MJ Ray: Upgrading to Debian 7 wheezy

So I have upgraded my main workstation at last, but there are a few things I wanted to figure out. Some of these I found an answer to, but others I haven t and some answers open more questions: Where did debian-menu.menu go and why doesn t lxlauncher have the Debian menu by default anyway? Answer: It seems this is a combination of debian bug 333848 and bug 722563 which I reported yesterday in that lxlauncher doesn t use the debian menus. Workaround for now: install menu-xdg and hack the lxlauncher menu file. Why don t my xterm shells load .profile any more? Did I bodge that before without making a note? How do I stop kernel or initrd or whatever it is configuring the network with the dhcp settings at boot time instead of the better wicd ones that are available once the system is up? Continuing with networking, how did network-manager (which doesn t cope with my network configuration) get installed again? Answer: debian bug 645656 explains that debian is simply following a silly decision taken upstream and refers to attempts to make network-manager optional as the Crusade . Workaround: remove gnome and install its dependencies like gnome-core directly instead. Drawback: any new dependencies of gnome will have to be installed manually. Why is the audio capture volume at 10% by default and what s the best way to change that? I suspect this might be 682731 but that bug is untouched in over a year. I m tempted to remove pulseaudio, but gnome-core Depends pulseaudio so this is yet more hardware-breaking caused by gnome dependency changes and the only way out of this one would be to remove gnome-core and install its dependencies. Or maybe I should just give up and finally remove gnome because its packages have jumped the shark. I installed gnome to make the system easier to use and it s broken it in at least two ways now. One question that makes me hesitate is: why is gdm3 faster to start than lightdm?

12 September 2013

Russ Allbery: AFS::PAG 1.00

One of the very long-standing problems with AFS is that it doesn't have a stable library interface or API. The situation with shared libraries is even worse. Despite that, there is a native XS AFS Perl module that tries to provide a native Perl interface to the various AFS libraries, despite all of their quirks. We've used that off and on for various things over the years. Unfortunately, the historic AFS code used an LWP (lightweight threading) implementation that relied on hairy C internals, and when Perl switched over to being threaded with POSIX pthreads by default, the loadable module started having trouble. Various people managed to keep it limping along through Debian's squeeze release, but as of wheezy it just no longer worked; the thread mismatches and the mismatches between the various AFS libraries were too severe. There's work in progress to fix this by building yet another profile of the AFS libraries, with all their warts, but in the meantime we needed access to the AFS PAG functions for various backend scripts. Everything else the Perl module did could be replaced by running the regular commands and parsing their output, but setpag has to be called in the process that should be affected to work safely. Hence this module. AFS::PAG provides a native Perl interface to the API exposed by libkafs and libkopenafs for PAG manipulation: hasafs, haspag, setpag, and unlog. (Eventually it might provide access to pioctl, but I haven't done that work yet.) It supports any platform that has a libkafs or libkopenafs, or Linux without either (by implementing the pioctl interface itself). You can get the latest version from the AFS::PAG distribution page.

Gunnar Wolf: What is to be an author? How much code does grant you authorship? #encirc13

I signed up to take part of an online, massive course Arte y cultura en circulaci n: crear y compartir en tiempos digitales (The circulation of arts and culture: Creating and sharing in digital times). As the activities of us the attendees are to be published by each on their personal blog/space, I will be publishing them here. I hope they are interesting to some of my other regular readers. I am late in joining, and I should already be posting the second activity. Anyway, this text goes towards the first week's lesson: Qu es un autor: la (de)construcci n hist rica del concepto de autor a (What is an author? Historic (de)construction of the concept of authorship). Our homework is to find an example of a current discussion where the notion of authorship is discussed, and share it this way. And I'm somewhat in a hurry right now, so this will be hasty. But I didn't want to delay my (late) submission anymore! In the Debian Ruby mailing list, Hleb Valoshka asks basically how deep should copyright recognition be. Because, yes, while copyright attribution is simple (simpler, at least) in literary / artistic works, computer programs (and even more those developed following the Free Software distributed fashion) are harder to properly attribute. Is a two-line patch to a tens-of-thousands-of-lines project enough to warrant mention in the copyright file? Most of us would agree it is not. Few would contend this were the amount of changes to make up, say, even 1% of that scale of project. In the coding projects where I am the main author (which, yes, are usually very small I cannot recognize myself as a good programmer, and the size of my successful works proves me), I try to take care to mention explicitly each of the contributors, even if their contribution is quite small. But were I to lead a larger development work, with enough following to generate on the order of one submitted patch a day, would it make sense for me to follow up with such detail the authorship information? And as a minor side note: How much does the law require me? A very small patch can fix important functionality issues (think security or performance). If somebody sends a very small patch fixing an intellectually hard to grasp problem... I am sure it should be properly acknowledged! Anyway, I have to flee now. I am just dropping some ideas on the table. Of course, if anybody is willing to discuss them further, I'm interested in any debate that springs off it! (Meanwhile... I expect to spam you with this topic a couple of times. But it should not bug you much, as it's one of my usual blog topics!)

Steve Kemp: The blogspam code is live.

Living dangerously I switched DNS to point to the new codebase on my lunch hour. I found some problems immediately; but nothing terribly severe. Certainly nothing that didn't wait until I'd finished work to attend to. I've spent an hour or so documenting the new API this evening, and now I'm just going to keep an eye on things over the next few days. The code is faster, definitely. The load is significantly lower than it would have been under the old codebase - although it isn't a fair comparison: The old XML-RPC API is still present, but now it just proxies to the JSON-version, which is a cute hack. How long it stays alive is an open question, but at least a year I guess. God knows what my wordpress developer details are. I suspect its not worth my updating the wordpress plugin, since nobody ever seemed to love it. These days the consumers of the API seem to be, in rough order of popularity: There are few toy-users, like my own blog, and a few other similar small blogs. All told since lunchtime I've had hits from 189 distinct sources, the majority of which don't identify themselves. (Tempted to not process their requests in the future, but I don't think I can make such a change now without pissing off the world. Oops.) PS. Those ~200 users? rejected 12,000 spam comments since this afternoon. That's cool, huh?

11 September 2013

Brett Parker: Moving, Changing Jobs and the Bank Holiday Weekend...

So, in the last couple of weeks I've (mostly) moved out of Brighton, got a new job, and generally the world has gone crazy. The bank holiday weekend was spent at Debian UK BBQ, and was fantastic fun, including the reading corner on Sunday where we played through various arcs of To Be or Not To Be (small spoiler: PIRATES!). Since then I've been mostly back and forth from Brighton moving stuff, buying a car, and starting a new job... As mentioned in my new employers blog there was a small stumbling block with my new work laptop that stopped me from getting a sensible operating system on it from the get-go, but it's now running a shiny Debian Wheezy install and alllllllllll is well with the world... well, mostly. I have just removed gnome-keyring because I hate that I can't see a way of relocking it, or infact relocking any of the individual keys that it had. So, back to ssh-agent I went, and the world is good again. But that means that I haven't got a gpg agent running at the moment, though, unless I'm missing something, I can't see a way of forcing gpg-agent to forget a passphrase either (ARGH, why on earth wouldn't you have an option for that?!).

Steve Kemp: I've always relied upon the kindness of strangers

Many thanks to Vincent Meurisse who solved my node.js callback woe. Some history of the blogspam service:
Back in 2008 I was annoyed by the many spam-comments that were being submitted to my Debian Administration website. I added some simple anti-spam measures, which reduced the flow, but it was a losing battle. In the end I decided I should test comments, as the users submitted them, via some kind of external service. The intention being that any improvements to that central service would benefit all users. (So I could move to testing comments on my personal blog too, for example). Ultimately I registered the domain-name "blogspam.net", and set up a simple service on it which would test comments and judge them to be "SPAM" or "OK". The current statistics show that this service has stopped 20 million spam comments, since then. (We have to pretend I didn't wipe the counters once or twice.)
I've spent a while now re-implementing most of the old plugins in node.js, and I think I'll be ready to deploy the new service over the weekend. The new service will have to handle two different kinds of requests:
New Requests
These will be submitted via HTTP POSTed JSON data, and will be handled by node.js. These should be nice and fast.
Legacy Requests
These will come in via XML-RPC, and be proxied through the new node.js implementation. Hopefully this will mean existing clients won't even notice the transition.
I've not yet deployed the new code, but it is just a matter of time. Hopefully being node.js based and significantly easier to install, update, and tweak, I'll get more contributions too. The dependencies are happily very minimal: The only significant outstanding issue is that I need to pick a node.js library for performing CIDR lookups - "Does 10.11.12.23 lie within 10.11.12.0/24?" - I'm surprised that functionality isn't available out of the box, but it is the only omission I've missed. I've been keeping load & RAM graphs, so it will be interesting to see how the node.js service competes. I expect that if clients were using it, in preference to the XML-RPC version, then I'd get a hell of a lot more throughput, but with it hidden behind the XML-RPC proxy I'm less sure what will happen. I guess I also need to write documentation for the new/preferred JSON-based API... https://github.com/skx/blogspam.js

Olivier Berger: Publishing RDF views for tastypie/django resources

Here s some documentation about a hack I ve been working on to allow publication of RDF views for Tastypie resources (in Django applications). While working on implementing support for RDF meta-data for descriptions of Debian packages for the Debian Package Tracking System rewrite, I ve tried and idetify which libraries/frameworks would allow to create some RDF views for Django model objects with a minimal effort. Ideally, this would save the hassle of writing code, and could just be a matter of mapping some Django model fields to proper ontology attributes. I haven t found an existing tool to do so, but it seemed to me that Tastypie could offer a nice starting point. Tastypie (I focused on v. 0.9.15 which is currently in Debian testing) offers some REST content-negociation support, and other niceties. This post is an attempt at documenting an initial implementation for my problem. I ve implemented it as some code in the example blog application described in the Tastypie documentation. Unfortunately, it s not yet a patch that could be applied to Tastypie, to add this as a standard feature. The code is available in my git clone of Tastypie, based on 0.9.15, and lies in the commit(s) between the example_myapp and rdfviews branches. It needs an up to date RDFLib to work (Debian s is too old, btw). It basically relies on the addition of a _rdf_mapping dict in the Django model objects, and an RdfModelResource as a base class for Tastypie ModelResources. To provide an RDF view for a ModelResource, it also requires to declare a particular MetaClass, and adding a few bits to its Meta subclass. The principle is that the dehydratation steps of the Tastypie system will replace fields values by their RDF objects counterparts, as either RDFLib Literals or URIRefs. Then the resuting Bundle will be converted to some RDFLib Graph. It is then just a matter of serialization of that Graph to turtle (I ve not added another format, but it should be pretty straightforward). There s a bit of hackish drak magic in the code, in particular when introspecting the Tastypie / Django resource / model objects to be able to dynamically add some dehydrate_FOO methods to perform the conversion to Literals or URIRefs. Custom dehydrateFoo methods can still be written to process the fields in particular ways. I ve added some LDP style paging along the way, so I hope it will be quite usable by LDP compatible tools. Here s the result : First, listing all resources : $ curl -H 'Accept: text/turtle' "http://localhost:8000/api/v1/entry/"
@prefix dcterms: <http://purl.org/dc/terms/> .
@prefix ldp: <http://www.w3.org/ns/ldp#> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix sioc: <http://rdfs.org/sioc/ns#> .
@prefix xml: <http://www.w3.org/XML/1998/namespace> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> . </api/v1/entry/#list> dcterms:hasPart </api/v1/entry/1/#post>,
</api/v1/entry/2/#post>,
</api/v1/entry/3/#post> . Then, details of one particular post : $ curl -L -H 'Accept: text/turtle' "http://localhost:8000/api/v1/entry/1"
@prefix dcterms: <http://purl.org/dc/terms/> .
@prefix owl: <http://www.w3.org/2002/07/owl#> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix sioc: <http://rdfs.org/sioc/ns#> .
@prefix xml: <http://www.w3.org/XML/1998/namespace> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> . </api/v1/entry/1/#post> a sioc:Post ;
dcterms:created "2011-05-21T22:46:38+00:00"^^xsd:dateTime ;
dcterms:title "Another Post" ;
sioc:content "MESSAGE: This will prbbly be my lst post. /MESSAGE" ;
sioc:has_creator </api/v1/user/1/> ;
owl:sameAs </posts/another-post#post> .

I hope this is usable, and welcome any feedback.

Jan Wagner: ownCloud for Debian wheezy

Unfortunately ownCloud is not available in Debian wheezy. Even if it was promised to provide a package via backports, this seems never to happen. As I don't like the idea of adding source lists of testing/unstable/experimental to production systems, I walked the long and painfull path of backporting all (build-)dependencies and pushed them into our wheezy-backports repository. If you want to give it a try on Debian wheezy, you could install the packages from our wheezy-backports repository: 

    # wget "http://ftp.cyconet.org/debian/sources.list.d/wheezy-backports-cyconet.list" \
        -O /etc/apt/sources.list.d/wheezy-backports-cyconet.list
    # aptitude update
    # aptitude -t wheezy-backports install owncloud owncloud-mysql
For adding our archive key, you can just install the package "debian-cyconet-archive-keyring"

Bastian Venthur: What are the most popular .vimrc options?

Hi always wondered what the most popular options are, you usually find in .vimrc files. So I downloaded 155 .vimrc files from the net (mostly from dotfiles.org and github.com), and wrote a little script which counts the number of times an option has been set. Since most options come in normal- and shortcut form, I mapped the shortcuts to the long version whenever I recognized them. So without further ado, here are the most popular .vimrc options (without values!). The number specifies the number of times this option has been set. The most popular option is on the bottom:
10 tselect
10 dictionary
10 runtimepath
11 mousehide
11 t_vb
11 foldlevel
11 foldopen
12 suffixes
12 matchtime
12 fileencoding
13 modelines
13 wrap
14 sidescrolloff
14 clipboard
14 lines
14 novisualbell
15 linebreak
15 cursorline
15 fileformats
15 columns
15 cindent
16 undolevels
16 shiftround
16 lazyredraw
16 completeopt
18 modeline
18 whichwrap
18 comments
18 wildignore
19 list
19 autowrite
19 foldcolumn
19 grepprg
19 titlestring
20 autoread
20 title
21 foldenable
21 cmdheight
22 pastetoggle
23 formatoptions
23 fileencodings
24 tags
24 directory
25 ttyfast
26 termencoding
26 complete
27 nohlsearch
27 noerrorbells
27 visualbell
28 shortmess
30 showmode
31 wildmode
32 t_Co
32 listchars
32 backupdir
34 hidden
34 backup
35 smarttab
35 foldmethod
36 viminfo
36 textwidth
37 scrolloff
37 nobackup
41 nowrap
44 encoding
47 guifont
51 guioptions
53 smartcase
54 wildmenu
57 smartindent
60 mouse
63 background
64 softtabstop
66 history
70 showmatch
72 ignorecase
74 showcmd
74 laststatus
79 number
82 hlsearch
91 statusline
94 expandtab
94 ruler
96 autoindent
96 backspace
99 tabstop
109 incsearch
114 shiftwidth
124 nocompatible Out of 155 .vimrcs
Fun fact: nocompatible is the most popular, but also most useless one. The fact that you have an .vimrc automatically implies the nocompatible mode in vim.

10 September 2013

Steve Kemp: node.js is kicking me

Today I started hacking on a re-implementation of my BlogSpam service - which tests that incoming comments are SPAM/HAM - in node.js (blogspam.js) The current API uses XML::RPC and a perl server, along with a list of plugins, to do the work. Having had some fun and success with the HTTP+JSON mstore toy I figured I'd have a stab at making BlogSpam more modern: In theory this is easy, I've hacked up a couple of plugins, and a Perl client to make a submission. But sadly the async-stuff is causing me .. pain. This is my current status: 
shelob ~/git/blogspam.js $ node blogspam.js
Loaded plugin: ./plugins/10-example.js
Loaded plugin: ./plugins/20-ip.js
Loaded plugin: ./plugins/80-sfs.js
Loaded plugin: ./plugins/99-last.js
Received submission:  "body":"This is my body ..","ip":"109.194.111.184","name":"Steve Kemp" 
	plugin 10-example.js said next :next
	plugin 20-ip.js said next :next
	plugin 99-last.js said spam SPAM: Listed in StopForumSpam.com
So we've loaded plugins, and each has been called. But the end result was "SPAM: Listed .." and yet the caller didn't get that result. Instead the caller go this: 
shelob ~/git/blogspam.js $ ./client.pl
200 OK 99-last.js
The specific issue is that I iterate over every loaded-plugin, and wait for them to complete. Because they complete asynchronously the plugin which should be last, and just return "OK" , has executed befure the 80-sfs.js plugin. (Which makes an outgoing HTTP request). I've looked at async, I've looked at promises, but right now I can't get anything working. Meh. Surprise me with a pull request ;)

Gergely Nagy: How to monitor a directory of logfiles?

One feature of syslog-ng PE is support for wildcard file sources, meaning you can give it a wildcard, like /var/log/apache/*.log, and it will automatically notice new files being created or old ones disappearing. This is a very useful feature which has been missing from the open source edition, and one that is being worked on, but is far from complete, and will not make it into syslog-ng 3.5. However, at least on GNU/Linux, there is a way to implement something similar, although in a fairly crude manner. Yet, it works surprisingly well in most cases. You only need to abuse a few things...The trick itself is pretty easy, and is based on the same underlying kernel feature that syslog-ng PE uses under the hood: inotify(7). What we will do, is use incron to monitor a directory, and create syslog-ng config file snippets whenever a file creation event occurs. We also set up syslog-ng to include these snippets, and reload the configuration after each event received. This way, we immediately notice newly created logfiles without the need to poll the directory, and with config snippets created on the fly, we do not modify the main syslog-ng configuration file, either.To do this, we first need to create an appropriate syslog-ng configuration: 
@version: 3.5
destination d_all_logs  
  file("/var/log/all.log");
 ;
include "/etc/syslog-ng/conf.d/notify-*.conf";
The config itself is a dummy: it has no sources, no logpaths, just a destination, which we will use in the snippets. Next, we create the script that will create these for us! It takes a single command-line argument: the filename of the newly created logfile.
#! /bin/sh set -e CONFDIR="/etc/syslog-ng/conf.d" SNGCTL=/usr/sbin/syslog-ng-ctl id="$(uuidgen tr '-' '_')" conffn="$ CONFDIR /notify-$ id .conf" fn="$1" cat >$ conffn <<EOF source s_$ id file("$ fn "); ; log source(s_$ id ); destination(d_all_logs); ; EOF $ SNGCTL reload
This creates the config snippet with a unique filename and source name, adds a new logpath that binds it to the d_all_logs destination defined in the main configuration file, and reloads the configuration afterwards.Now, all we need is to tell incron to call this script. Assuming we saved it as /usr/local/sbin/syslog-ng-wildcard-notify, this is how the incrontab entry would look like: 
/var/log/remote.d/*.log  IN_CREATE,IN_MOVED_TO /usr/local/sbin/syslog-ng-wildcard-notify $@/$#
And that's about it! Now whenever a new file gets dropped into the directory, syslog-ng will get notified. There is a downside, however: when a file gets deleted, the configuration is not removed. That part is not so simple, though: we want syslog-ng to finish reading the file before reloading the config, so that makes it a bit trickier. But if we don't care about that, we could just add an IN_DELETE event, and write another script that looks up which snippet is responsible for the file, delete it and reload. Doing that is left as an exercise for the reader!

Craig Small: Odd WordPress pingbacks

I m getting some odd log messages for the apache module modsecurity. Essentially its xml parser is breaking when random places are sending pingbacks. The requests go to xmlrpc.php and the response headers are ok, but the body is binary. The message in the modsecurity log looks like: 
Message: XML parser error: XML: Failed parsing document.
After a bit of guessing and sending messages to and fro, I can now see that it is a gziped response. So I m not sure if it is modsecurity not realising that the response is gziped or wordpress not marking it correctly. In any case I can regularly get very similar binary strings using gzip and the usual xml response. So that s half the mystery solved. Most of the requests are spammers so I m not too worried. I think it also impacts legitimate pingbacks because I ve not had any, even from the usual automatic places.

Julien Danjou: OpenStack Ceilometer Havana-3 milestone released

Last week, the third and last milestone of the Havana development branch of Ceilometer has been released and is now available for testing and download. This means the end of the OpenStack Havana development time is coming, and that the features are now frozen. New features
Eleven blueprints have been implemented as you can see on the release page. That's one more than during Havana-2, but it's less than was planned initially, though we had a pretty high score considering the size of our contributors team. I'm going to talk through some of them here, that are the most interesting for users. Bug fixes Fifty-six bugs were fixed, though most of them might not interest you so I won't elaborate too much on that. Go read the list if you are curious. Toward our final Havana release With the feature freeze in place, we're now focusing on fixing bugs and improving documentation. I'll try to make sure we'll get there without too much trouble for the 17th October 2013. Stay tuned!

Next.

Previous.