Shortly before 9AM this morning I discovered that the IP address for my mail server was not being routed, according to my logs the problem started shortly after midnight. It s on a TPG ADSL connection, there is one IP address for the PPPOE link and 6 addresses in a /29 routed to it one of the addresses in the /29 is for my mail server. It wasn t until 3PM that I was able to visit the server to sort the problem out. It turned out that the main IP address was working but the /29 wasn t being routed to it. So TPG had somehow dropped the route from my routing table. I pinged all the addresses from a 3G broadband connection on my EeePC while running tcpdump on the server, and no packets for the /29 came through but the IP address for the PPP link worked fine. I was even able to ssh in to the server once I knew the IP address of the ppp0 device for future use I need to keep ALL IP addresses of all network gear on my EeePC not just the ones used for providing services. So I phoned the helpdesk, and naturally they asked me inane questions. My patience extended to telling them the broadcast address etc that was being used on the Ethernet device (actually a bridge for Xen but I wasn t going to confuse them). The system been power-cycled before I got there in the hope that it might fix the problem so I could honestly answer the question have you rebooted it (usually I lie rebooting systems to fix network problems is a Windows thing). But my patience started to run out when they asked me to check my DNS settings, I explained very clearly that my problem was that IP packets couldn t get through and that I wasn t using DNS and demanded that they fix it. I didn t get anyone technical to look at the problem until I firmly demanded that the help-desk operator test the routing by pinging my systems. The help-desk people don t have Internet access so that actually testing the connection required escalating the issue. It seems that the algorithm used for help-desk people is to just repeatedly tell people to check various things on their own system, and that continues until the customer s patience runs out. Either the customer goes away or makes requests firmly enough to get something done about it. So their technician did some tests and proclaimed that there was no problem. While said tests were being done things started working, so obviously their procedure is to fix problems and then blame it on the customer. It is not plausible to believe that a problem in their network which had persisted for more than 15 hours would accidentally disappear during the 5 minute window that the technician was investigating the problem. In the discussion that followed the help-desk operator tried to trick me into admitting that it was my fault. They claimed that because I had used multiple IP addresses I must have reconfigured my system and had therefore fixed a problem on my end, my response was I HAVE A HEAP OF MACHINES HERE RUNNING ALL THE TIME, I USE WHICHEVER ONE I FEEL LIKE, I CHANGED NOTHING . I didn t mention that the machines in question are DomUs on the same Xen server, someone who doesn t understand how ping works or what routing is wouldn t have been able to cope with that. I stated clearly several times that I don t like being lied to. Either the help-desk operator was lying to me or their technician was lying to them. In either case they were not going to trick me I know more about how the Internet works than they do. TPG was unable to give me any assurance that such problems won t happen again. The only thing I can be sure of is that when they lie they will stick to their story regardless of whether it works.

This is the fifth year we hold an EDUSOL, and we are closing in on it. EDUSOL is an online encounter whose topics are Education and Free Software Actually, this year we are widening our scope, and we will include Free Culture as well as a base area. Now, besides those three general areas, each year we have had a base topic around which we invite the speakers to talk about (although it is a lax requirement). This year, the base topic is social networks No, not in the Twitter/Facebook/blah sense, but as a wider phenomenon, studying interaction between people, the forming of communities. And for our particular areas, the forming of knowledge-based communities. Anyway I agreed with the organizers to provide the English translation of the participation invitation. I will skip the call for papers, as we are basically at the proposal deadline (October 17), but if you have anything you want to propose, please tell us so! Leaving that aside... Please excuse the quality of my translation, it is late and I'm tired. We will work on it :) Videoconference reception EDUSOL spans several participation categories. The closest category to a traditional, face to face conference. Each year, we invite a group of speakers to talk about a topic related to our main discoursive line. Among the speakers that have confirmed so far, we have:

• Sof a Liberman Shklonikoff, Mexico. Social networks.
• Dolors Reig. Barcelona, Spain. Open social education.
• Marko Txopitea. Bilbao, Spain. Politics 2.0
• Carolina Flores. Costa Rica. Building methodologies for infoinclusion
• Luis Rodrigo Gallardo Cruz. United States. OpenSocial.
• Margarita Manterola. Argentina. Debian-Women

We invite you to be active participants in the videoconference cycle. You can invite your social or user group to be part of the Encounter. There are three ways to do it:

1. Using the videoconference facilities in your University network
2. Connecting from a personal computer with broadband access, by using Ekiga or any other H.323 client
3. Following the .ogg stream and participating back via IRC

In any case (specially in the first two, which require more coordination), please contact us. For further information, visit http://edusol.info/es/e2009/convocatorias/videoconfrencias November 6: Free Education and Free Culture day We invite social and user groups to host talks regarding Free Education and Free Culture. This is not a call to promote Free Software, as there are many other spaces devoted to it. We should start with the idea that freedom resides in us, not in the software. Some of the topics our community recommends are:

• Common goods
• Free culture: Success stories
• Be legal: Sharing digital media
• Free culture at school
• Collaboration and collectivity
• Freeing your computer

Further details at http://edusol.info/es/e2009/convocatorias/dialternativo We need some help in this area to generate contents with slides, making it easier for proposed scripts for the talks. If you want to collaborate, please write to our academic support list, http://lists.edusol.info/listinfo.cgi/soporte-academico-edusol.info Want to collaborate? Further questions or comments?

• Questions to contacto@edusol.info
• identi.ca group: http://identi.ca/group/edusol
• Academic support group (EDUSOL Encounter and Free Culture day): http://lists.edusol.info/listinfo.cgi/soporte-academico-edusol.info
• And of course, the EDUSOL Community Planet: http://edusol.info/planeta/

We are holding on-line meetings for this group of topics on Thursday 22:00 GMT-5, in the #edusol channel in OFTC (irc.debian.org); you can enter the IRC channel using the Web client at http://irc.bine.org.mx or http://edusol.info/irc About EDUSOL EDUSOL started as a proposal seeking to provide a space so that people interested in education could discuss and analize the good and bad points about Free Culture and Free Software, with no geographic restrictions. Year after year, freedom-loving educators of all Latin America and Spain gather for our annual party. EDUSOL's core language is Spanish, although participation in English is allowed and encouraged (although understanding Spanish will be a strong aid). We invite you to participate and contribute in this collective effort using and commenting on our blogs, or adding your personal blog to our planet: http://edusol.info/es/node/add/blog
http://edusol.info/es/node/add/feedapi-node You can follow us by identi.ca or RSS: http://identi.ca/group/edusol
http://edusol.info/rss.xml

Building a web-based API with Turbogears2 I am using TurboGears2 to export a python API over the web. Every API method is wrapper by a controller method that validates the parameters and returns the results encoded in JSON. The basic idea is this:

@expose("json")
def list_colours(self, filter=None, productID=None, maxResults=100, **kw):
    # Call API
    res = self.engine.list_colours(filter, productID, maxResults)
    # Return result
    return res

To validate the parameters we can use forms, it's their job after all:

class ListColoursForm(TableForm):
    fields = [
            # One field per parameter
            twf.TextField("filter", help_text="Please enter the string to use as a filter"),
            twf.TextField("productID", help_text="Please enter the product ID"),
            twf.TextField("maxResults", validator=twfv.Int(min=0), default=200, size=5, help_text="Please enter the maximum number of results"),
    ]
list_colours_form=ListColoursForm()
#...
    @expose("json")
    @validate(list_colours_form, error_handler=list_colours_validation_error)
    def list_colours(self, filter=None, productID=None, maxResults=100, **kw):
        # Parameter validation is done by the form
        # Call API
        res = self.engine.list_colours(filter, productID, maxResults)
        # Return result
        return res

All straightforward so far. However, this means that we need two exposed methods for every API call: one for the API call and one error handler. For every API call, we have to type the name several times, which is error prone and risks to get things mixed up. We can however have a single error handler for all methonds:

def get_method():
    '''
    The method name is the first url component after the controller name that
    does not start with 'test'
    '''
    found_controller = False
    for name in pylons.c.url.split("/"):
        if not found_controller and name == "controllername":
            found_controller = True
            continue
        if name.startswith("test"):
            continue
        if found_controller:
            return name
    return None
class ValidatorDispatcher:
    '''
    Validate using the right form according to the value of the "method" field
    '''
    def validate(self, args, state):
        method = args.get("method", None)
    # Extract the method from the URL if it is missing
        if method is None:
            method = get_method()
            args["method"] = method
        return forms[method].validate(args, state)
validator_dispatcher = ValidatorDispatcher()

This validator will try to find the method name, either as a form field or by parsing the URL. It will then use the method name to find the form to use for validation, and pass control to the validate method of that form. We then need to add an extra "method" field to our forms, and arrange the forms inside a dictionary:

class ListColoursForm(TableForm):
    fields = [
            # One hidden field to have a place for the method name
            twf.HiddenField("method")
            # One field per parameter
            twf.TextField("filter", help_text="Please enter the string to use as a filter"),
    #...
forms["list_colours"] = ListColoursForm()

And now our methods become much nicer to write:

    @expose("json")
    def api_validation_error(self, **kw):
        pylons.response.status = "400 Error"
        return dict(form_errors=pylons.c.form_errors)
    @expose("json")
    @validate(validator_dispatcher, error_handler=api_validation_error)
    def list_colours(self, filter=None, productID=None, maxResults=100, **kw):
        # Parameter validation is done by the form
        # Call API
        res = self.engine.list_colours(filter, productID, maxResults)
        # Return result
        return res

api_validation_error is interesting: it returns a proper HTTP error status, and a JSON body with the details of the error, taken straight from the form validators. It took me a while to find out that the form errors are in pylons.c.form_errors (and for reference, the form values are in pylons.c.form_values). pylons.response is a WebOb Response that we can play with. So now our client side is able to call the API methods, and get a proper error if it calls them wrong. But now that we have the forms ready, it doesn't take much to display them in web pages as well:

def _describe(self, method):
    "Return a dict describing an API method"
    ldesc = getattr(self.engine, method).__doc__.strip()
    sdesc = ldesc.split("\n")[0]
    return dict(name=method, sdesc = sdesc, ldesc = ldesc)
@expose("myappserver.templates.myappapi")
def index(self):
    '''
    Show an index of exported API methods
    '''
    methods = dict()
    for m in forms.keys():
        methods[m] = self._describe(m)
    return dict(methods=methods)
@expose('myappserver.templates.testform')
def testform(self, method, **kw):
    '''
    Show a form with the parameters of an API method
    '''
    kw["method"] = method
    return dict(method=method, action="/myapp/test/"+method, value=kw, info=self._describe(method), form=forms[method])
@expose(content_type="text/plain")
@validate(validator_dispatcher, error_handler=testform)
def test(self, method, **kw):
    '''
    Run an API method and show its prettyprinted result
    '''
    res = getattr(self, str(method))(**kw)
    return pprint.pformat(res)

In a few lines, we have all we need: an index of the API methods (including their documentation taken from the docstrings!), and for each method a form to invoke it and a page to see the results. Make the forms children of AjaxForm, and you can even see the results together with the form.

The first Telepathy session session on Saturday evening at the Boston GNOME Summit was very much of a Q&A where myself and Will answered various technical and roadmap issues from a handful of developers and downstream distributors. It showed me that there s a fair amount of roadmap information we should do better at communicating outside of the Telepathy project, so in the hope its useful to others, read on MC5 Ted Gould was wondering why Mission Control 4 had an API for getting/setting presence for all of your accounts whereas MC5 does not. MC5 has a per-account management of desired/current presence which is more powerful, but loses the convenience of setting presence in one place. Strictly speaking, doing things like deciding which presence to fall back on (a common example being if you have asked for invisible but the connection doesn t support it) is a UI-level policy which MC should not take care of, but in practice there aren t many different policies which make sense, and the key thing is that MC should tell the presence UI when the desired presence isn t available so it could make a per-account choice if that was preferable. As a related side point, Telepathy should implement more of the invisibility mechanisms in XMPP so it s more reliably available, and then we could more meaningfully tell users which presence values were available before connecting, allowing signing on as invisible. Since MC5 gained support for gnome-keyring, its not possible to initialise Empathy s account manager object without MC5 prompting the user for their keyring password un-necessarily (especially if the client is Ubuntu s session presence applet and the user isn t using Empathy in the current session but has some accounts configured). Currently the accounts D-Bus API requires that all properties including the password are presented to the client to edit. A short-term fix might be to tweak the spec so that accounts don t have to provide their password property unless it s explicitly queried, but this might break the ABI of tp-glib. Ultimately, passwords being stored and passed around in this way should go away when we write an authentication interface which will pass authentication challenges up to the Telepathy client to deal with, enabling a unified interface for OAuth/Google/etc web token, Kerberos or SIP intermediate proxy authentication, and answering password requests from the keyring lazily or user on demand. Stability & Security Jonathan Blandford was concerned about the churn level of Telepathy, from the perspective of distributions with long-term support commitments, and how well compatibility will be maintained. Generally the D-Bus API and the tp-glib library APIs are maintained to the GNOME standards of making only additive changes and leaving all existing methods/signals working even if they are deprecated and superseded by newer interfaces. A lot of new interfaces have been added over the past year or so, many of which replace existing functionality with a more flexible or more efficient interface. However, over the next 4-6 months we hope to finalise the new world interfaces (such as multi-person media calls, roster, authentication, certificate verification, more accurate offline protocol information, chat room property/role management), and make a D-Bus API break to remove the duplicated cruft. Telepathy-glib would undergo an ABI revision in this case to also remove those symbols, possibly synchronised with a move from dbus-glib to GVariant/etc, but in many cases clients which only use modern interfaces and client helper classes should not need much more than a rebuild. Relatedly there was a query about the security of Telepathy, and how much it had been through the mill on security issues compared to Pidgin. In the case of closed IM protocols (except MSN where we have our own implementation) then we re-use the code from libpurple, so the same risks apply, although the architecture of Telepathy means its possible to subject the backend processes to more stringent lockdowns using SElinux or other security isolation such as UIDs, limiting the impact of compromises. Other network code in Telepathy is based on more widely-used libraries with a less chequered security history thus far. OTR The next topic was about support for OTR in Telepathy. Architecturally, it s hard for us to support the same kind of message-mangling plugins as Pidgin allows because there is no one point in Telepathy that messages go through. There are multiple backends depending on the protocol, multiple UIs can be involved in saving (eg a headless logger) or displaying messages (consider GNOME Shell integration which previews messages before passing conversations on to Empathy), and the only other centralised component (Mission Control 5) does not act as an intermediary for messages. Historically, we ve always claimed OTR to be less appealing than native protocol-level end-to-end encryption support, such as the proposals for Jingle + peer to peer XMPP + TLS which are favoured by the XMPP community, mostly because if people can switch to an unofficial 3rd party client to get encryption, they could switch to a decent protocol too, and because protocol-level support can encrypt other traffic like SRTP call set-up, presence, etc. However, there is an existing deployed OTR user base, including the likes of Adium users on the Mac, who might often end up using end to end encryption without being aware of it, who we would be doing a disservice by Telepathy not supporting OTR conversations with these people. This is a compelling argument which was also made to me by representatives from the EFF, and the only one to date which actually held some merit with me compared to just implementing XMPP E2E encryption. Later in the summit we went on to discuss how we might achieve it in Telepathy, and how our planned work towards XMPP encryption could also help. Tubes We also had a bit of discussion about Tubes, such as how the handlers are invoked. Since the introduction of MC5, clients can register interest in certain channel types (tubes or any other) by implementing the client interface and making filters for the channels they are interested in. MC5 will first invoke all matching observers for all channels (incoming and outgoing) until all of them have responded or timed out (eg to let a logger daemon hook up signal callbacks before channel handling proceeds), all matching approvers for incoming channels until one of them replies (eg to notify the user of the new channel before launching the full UI), and then sending it to the handler with the most specific filter (eg Tomboy could register for file transfers with the right MIME type and receive those in favour to Empathy whose filter has no type on it). Tubes can be shared with chat rooms, either as a stream tube where one member shares a socket for others to connect to (allowing re-sharing an existing service implementation), or a D-Bus tube where every member s application is one endpoint on a simulated D-Bus bus, and Telepathy provides a mapping between the D-Bus names and the members of the room. In terms of Tube applications, now we ve got working A/V calling in Empathy, as well as desktop sharing, and an R&D project on multi-user calls, our next priority is on performance and Tube-enabling some more apps such as collaborative editing (Gobby, AbiWord, GEdit, Tomboy ?). There was a question about whether Tube handlers can be installed on demand when one of your contacts initiates that application with you. It d be possible to simulate this by finding out (eg from the archive) which handlers are available, and dynamically registering a handler for all of those channel types, so that MC5 will advertise those capabilities, but also register as an approver. When an incoming channel actually arrives at the approval stage, prompt the user to install the required application and then tell MC5 to invoke it as the handler. Colin Walters asked about how Telepathy did NAT traversal. Currently, Telepathy makes use of libnice to do ICE (like STUN between every possible pair of addresses both parties have, works in over 90% of cases) for the UDP packets involved in calls signalled over XMPP, either the Google Talk variant which can benefit from Google s relay servers if one or other party has a Google account, so is more reliable, or the latest IETF draft which can theoretically use TURN relays but its not really hooked up in Telepathy and few people have access to them. XMPP file transfers and one-to-one tube connections use TCP which is great if you have IPv6, but otherwise impossible to NAT traverse reliably, so often ends up using strictly rate-limited SOCKS5 -ish XMPP proxies, or worse, in-band base64 in the XML stream. We hope to incorporate (and standardise in XMPP) a reliability layer which will allow us to use Jingle and ICE-UDP for file transfers and tubes too, allowing peer to peer connections and higher-bandwidth relays to enhance throughput significantly. Future Ted Gould had some good questions about the future of Telepathy

Should Empathy just disappear on the desktop as things like presence applets or GNOME Shell take over parts of its function? Maybe, yes. In some ways its goal is just to bring Telepathy to end users and the desktop so that its worth other things integrating into Telepathy, but Telepathy allows us to do a lot better than a conventional IM client. Maemo and Sugar on the OLPC use Telepathy but totally integrates it into the device experience rather than having any single distinct IM client, and although Moblin uses Empathy currently it has its own presence chooser and people panel, and may go on to replace other parts of the user experience too. GNOME Shell looks set to move in this direction too and use Telepathy to integrate communications with the desktop workflow. Should Telepathy take care of talking to social networking sites such as Facebook, Twitter, etc? There s no hard and fast rule Telepathy only makes sense for real-time communications, so it s good for exposing and integrating the Facebook chat, but pretty lame for dealing with wall posts, event invitations and the like. Similarly on the N900, Telepathy is used for the parts of the cellular stack that overlap with real-time communications like calling and SMS, but there is no sense pushing unrelated stuff like configuration messages through it. For Twitter, the main question is whether you actually want tweets to appear in the same UI, logging and notification framework as other messages. Probably not anything but the 1-to-1 tweets, meaning something like Moblin s Mojito probably makes more sense for that. Later in the summit I took a look at Google Latitude APIs, which seem like something which Telepathy can expose via its contact location interface, but probably not usefully until we have support for metacontacts in the desktop. Can/will Telepathy support IAX2? It can, although we d have to do a local demultiplexer for the RTP streams involved in separate calls. It s not been a priority of ours so far, but we can help people get started (or Collabora can chat to people who have a commercial need for it). Similarly nobody has looked at implementing iChat-compatible calling because our primary interest lies with open protocols, but if people were interested we could give pointers its probably just SIP and RTP after you dig through a layer of obfuscation or two.

If you want to know more about Telepathy feel free to comment with some follow-up questions, talk to us in #telepathy on Freenode, or post to the mailing list.

I spent this weekend in Boston for the annual GNOME summit. I really enjoyed it this year, although there were fewer attendees than previously it felt very focussed and productive. There s some cool stuff going on, and it s always great to catch up with all of the usual free software suspects in Boston. Some highlights from the weekend:

• Corridor session with David Zeuthen, Ryan Lortie, Matthias Clasen and later joined by me and Will Thompson from Collabora, discussed a lot of the issues of integrating D-Bus into Glib and I think achieved a pretty good consensus about how GVariant and GDBus should fit together and start getting the pieces merged. Really looking forwards to it, GVariant looks mad ninja.
• Some nice discussions about Gtk+ 3.0 roadmap, although I m still worried that the sealing/accessor work will take so much developer time there won t be that much time to make the improvements its supposed to enable. Theming was mentioned but what else should there be? Also spotted Kristian H gsberg, Cody Russell and Matthias again talking about client-side decorations, presumably Wayland scheming :)
• Nice session about messaging and other notifications in GNOME Shell. I really like the way the project is going, and hopefully we can join in and spend some time hooking Telepathy up here, although as a fallback to make the existing stuff (which doesn t have specific code to hook in and make a UI), then it d be nice if the shell could also be a frontend for the notify/indicate stuff which Ubuntu have been pushing in GNOME 2.x already.
• Had some good brainstorming about Telepathy integration in games and Tomboy sharing too, as well as some slightly less conclusive pondering about how to deliver metacontacts (ie, merging multiple sources such as IM, social networking and other address books) in GNOME. We need to sync up with what Moblin is doing here as it looks quite promising and should give us some components to re-use.
• Will, Sjoerd and I spent an hour or two around a blackboard working out how XMPP end-to-end TLS encryption might be exposed to clients in Telepathy, in order to work out how best we d expose OTR too. It looks like we have a fairly workable proposal now which we ll be explaining in due course, but it means at least we can give more useful advice to people who are interested in implementing it, or move forwards on implementing it ourselves.

I was really impressed by Jason Clinton and others summaries of the sessions, which I think are really valuable for the people who couldn t make it to the summit. He asked me to take some notes about the first Telepathy session on Saturday evening while he was taking notes about the Outreach session. Rather than lumber him with my deranged scratchings from Tomboy, I ll blog them separately.

Once upon a time I wrote a blog compiler, a simple tool that would read in a bunch of text files and output a blog. This blog would contain little hierarchies for tags, historical archives, etc. It would also have a number of RSS feeds too. Every now and again somebody will compare it to ikiwiki and I'll ignore that comparison entirely, because the two tools do different things in completely different fashions. But I was interested to see Joey talk about performance tweaks recently as I have a blog which has about 900 pages, and which takes just over 2 minutes to build from start to finish. (Not this one!) I've been pondering performance for a while as I know my current approach is not suited to high speed. Currently the compiler reads in every entry and builds a giant data structure in memory which is walked in different fashions to generate and output pages. The speed issue comes about because storing the data structure entirely in memory is insane, and because sometimes a single entry will be read from disk multiple times. I've made some changes over the past few evenings such that a single blog entry will be read no more than once from disk (and perhaps zero times if Memcached is in use :) but that doesn't solve the problem of the memory usage. So last night I made a quick hack - using my introduction to SQLite as inspiration I wrote a minimal reimplementation of chronicle which does things differently:

• Creates a temporary SQLite database with tables: posts, tags, comments.
• Reads every blog entry and inserts it into the database.
• Uses the database to output pages.
• Deletes the database.

This is a significantly faster approach than the previous one - with a "make steve" job taking only 18 seconds, down from just over 2 minutes 5 seconds. ("make steve" uses rsync to pull in comments on entries, rebuilds the blog, then uses rsync to push the generated output into its live location.) ObFilm: If...

NewServers.com [1] provides an interesting service. They have a cloud computing system that is roughly comparable to Amazon EC2, but for which all servers are physical machines (blade servers with real disks). This means that you get the option of changing between servers and starting more servers at will, but they are all physical systems so you know that your system is not going to go slow because someone else is running a batch job. New Servers also has a bandwidth limit of 3GB per hour with $0.10 per GB if you transfer more than that. Most people should find that 3GB/hour is enough for a single server. This compares to EC2 where you pay $0.10 per GB to receive data and $0.17 to transmit it. If you actually need to transmit 2100GB per month then the data transfer fees from EC2 would be greater than the costs of renting a server from New Servers. When running Linux the EC2 hourly charges are (where 1ECU is provides the equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor):

NAME	Cost	Desc
Small	$0.10	1.7G, 160G, 32bit, 1ECU, 1core
Large	$0.20	7.5G, 850G, 64bit, 4ECU, 2core
Extra Large	$0.40	15G, 1690G, 64bit, 8ECU, 4core
High CPU Medium	$0.20	1.7G, 350G, 32bit, 5ECU, 2core
High CPU Extra Large	$0.80	7G, 1690G, 64bit, 20ECU, 5core

The New Servers charges are:

NAME	Cost	Desc
Small	$0.11	1G, 36G, 32bit, Xeon 2.8GHz
Medium	$0.17	2G, 2*73G, 32bit, 2*Xeon 3.2GHz
Large	$0.25	4G, 250G, 64bit, E5405 Quad Core 2Ghz
Jumbo	$0.38	8G, 2*500G, 64bit, 2 x E5405 Quad Core 2Ghz
Fast	$0.53	4G, 2*300G, 64bit, E5450 Quad Core 3Ghz

The New Servers prices seem quite competitive with the Amazon prices. One down-side to New Servers is that you have to manage your own RAID, the cheaper servers have only a single disk (bad luck if it fails). The better ones have two disks and you could setup your own RAID. Of course the upside of this is that if you want a fast server from New Servers and you don t need redundancy then you have the option of RAID-0 for better performance. Also I don t think that there is anything stopping you from running Xen on a New Servers system. So you could have a bunch of Xen images and a varying pool of Dom0s to run them on. If you were to choose the Jumbo option with 8G of RAM and share it among some friends with everyone getting a 512M or 1G DomU then the cost per user would be a little better than Slicehost or Linode while giving better management options. One problem I sometimes have with virtual servers for my clients is that the disk IO performance is poorer than I expect. When running the server that hosts my blog (which is shared with some friends) I know the performance requirements of all DomUs and can diagnose problems quickly. I can deal with a limit on the hardware capacity, I can deal with trading off my needs with the needs of my friends. But having a server just go slow, not knowing why, and having the hosting company say I can move you to a different physical server (which may be better or worse) doesn t make me happy. I first heard about New Servers from Tom Fifield s LUV talk about using EC2 as a cluster for high energy physics [2]. According to the detailed analysis Tom presented using EC2 systems on demand can compete well with the costs of buying Dell servers and managing them yourself, EC2 wins if you have to pay Japanese prices for electricity but if you get cheap electricity then Dell may win. Of course a major factor is the amount of time that the servers are used, a cluster that is used for short periods of time with long breaks in between will have a higher cost per used CPU hour and thus make EC2 a better option.

• [1] http://www.newservers.com/
• [2] http://luv.asn.au/2009/10/06

This entry was originally posted in slightly different form to Server Fault If you're coming from a Windows world, you're used to using tools like zip or rar, which compress collections of files. In the typical Unix tradition of doing one thing and doing one thing well, you tend to have two different utilities; a compression tool and a archive format. People then use these two tools together to give the same functionality that zip or rar provide. There are numerous different compression formats; the common ones used on Linux these days are gzip (sometimes known as zlib) and the newer, higher performing bzip2. Unfortunately bzip2 uses more CPU and memory to provide the higher rates of compression. You can use these tools to compress any file and by convention files compressed by either of these formats is .gz and .bz2. You can use gzip and bzip2 to compress and gunzip and bunzip2 to decompress these formats. There are also several different types of archive formats available, including cpio, ar and tar, but people tend to only use tar. These allow you to take a number of files and pack them into a single file. They can also include path and permission information. You can create and unpack a tar file using the tar command. You might hear these operations referred to as "tarring" and "untarring". (The name of the command comes from a shortening of Tape ARchive. Tar was an improvement on the ar format in that you could use it to span multiple physical tapes for backups).

# tar -cf archive.tar list of files to include

This will create (-c) and archive into a file -f called archive.tar. (.tar is the convention extention for tar archives). You should now have a single file that contains five files ("list", "of", "files", "to" and "include"). If you give tar a directory, it will recurse into that directory and store everything inside it.

# tar -xf archive.tar
# tar -xf archive.tar list of files

This will extract (-x) the previously created archive.tar. You can extract just the files you want from the archive by listing them on the end of the command line. In our example, the second line would extract "list", "of", "file", but not "to" and "include". You can also use

# tar -tf archive.tar

to get a list of the contents before you extract them. So now you can combine these two tools to replication the functionality of zip:

# tar -cf archive.tar directory
# gzip archive.tar

You'll now have an archive.tar.gz file. You can extract it using:

# gunzip archive.tar.gz
# tar -xf archive.tar

We can use pipes to save us having an intermediate archive.tar:

# tar -cf - directory   gzip > archive.tar.gz
# gunzip < archive.tar.gz   tar -xf -

You can use - with the -f option to specify stdin or stdout (tar knows which one based on context). We can do slightly better, because, in a slight apparent breaking of the "one job well" idea, tar has the ability to compress its output and decompress its input by using the -z argument (I say apparent, because it still uses the gzip and gunzip commandline behind the scenes)

# tar -czf archive.tar.gz directory
# tar -xzf archive.tar.gz

To use bzip2 instead of gzip, use bzip2, bunzip2 and -j instead of gzip, gunzip and -z respectively (tar -cjf archive.tar.bz2). Some versions of tar can detect a bzip2 file archive with you use -z and do the right thing, but it is probably worth getting in the habit of being explicit. More info:

• http://en.wikipedia.org/wiki/Tar_(file_format)
• http://en.wikipedia.org/wiki/Bzip2
• http://en.wikipedia.org/wiki/Gzip

Sean Kenny emailed us to tell us about an event in Bristol: This week s Freeskilling is called Who needs Microsoft? and we are meeting at the Better Food Co. Cafe, St Werburghs @ 7pm Tues 13th Oct. Ben Smith will be giving a talk about how you can replace Microsoft Windows and all the very expensive and security flawed software that goes with it, with Free and Open Source Software (FOSS). This will save you lots of money and is more ethical into the bargain! For more info, go to http://thegurinet.doesntexist.com/ Only thing I m disappointed about is that it s centred on replacing Microsoft, but that s still the starting point for most people.

So, now that I completed my 2009 program by running the Cologne marathon 6 months after my first one in Paris, I now plan the 2010 "season"..:-) (one point for you guys who told me that once you start you don't stop) The next step is definitely Paris. The race is close to my home, this is a great race in a great city at a great time. I really can't miss it. So, that one is registered. The point is "what about 'the other one'". I plan to run another marathon during that year, preferably during autumn. Actually, I would probably be able to run at least one more but, well, many people urge me to avoid exxagerating and that includes my beloved wife..:-) So, one more. Which one? At this very moment, I have about 4 options:

• Cologne again. Pros: this is a nice race, human-size, in a welcoming place, also an oppotunity to run again with local friends. Cons: what the hell would we do in Cologne during the few days we would stay there, now that we've seen most of the city (one cathedral, 12 roman churches)...and that would mean the same program than 2009
• Berlin. Pros: the top german marathon, one of the "Big Ones", in a city that deserves a few days sightseeing and which Elizabeth really wants us to visit. Cons: a little bit early in September...and Germany again..:-)
• Donostia (San Sebastian). Pros: this is an open invitation by a relative of my brother in law, this is in Basque country which I enjoy, probably with a quite warm crowd. Cons: late in November and I fear the hills to be quite hard over there
• Chicago. Pros: would be a great opportunity to met with Dirk and even though we would not run together, I would enjoy this. Also a nice place to visit and p ssible to arrange a visit to my wife's family in Kansas City neighbourhood. Cons: arrangign a 10-days trip tp USA in early October is very very hard to fit with our professionnal schedule...and fairly expensive...:)

So, at this moment, Berlin and Donostia have the lead. A good compromise could be running a marathon in Berlin and the 20km Behobia-San Sebastian run which is very popular in Basque country. We'll see..

new OCaml Autoconf release: 1.1 We have just released version 1.1 of OCaml Autoconf: a set of M4 macros for integrating your OCaml project with autoconf. It is mostly a regular maintenance release with some bug fixes, some new features, and most notably a new mini-tutorial and example by Richard Jones (including how to deal with automake). Check the news for more info.

I have not been a frequent fixture on my own blog as I am writing what is called an Annual Review of Anthropology on digital media and ethnography. Truth be told it is killing me as there is a 6000 word limit and 100-150 works one must mention and entertain (usually by throwing in some categorical statement that makes sense for 10+ works). One thing is clear: the literature on digital media by anthropologists is switching from trickle to steady and very interesting stream. Even if I Epically Fail, I have already learned a lot, which is what I keep telling myself as I struggle through the writing stage of the article. But if you want a taste of some recent work, there are some blog entries you can check out: Daniel Miller who was one of the first anthropologists to venture in this area (and kick-started the first program in digital anthro) at UCL has written a nice review of various books recently published. And for the same blog, I wrote an overview
of my work on hacking, liberalism, and pleasure. So if you want a short introduction to the books being published by anthropologists on digital media, I highly recommend checking Daniel Miller s post.

I have not been a frequent fixture on my own blog as I am writing what is called an Annual Review of Anthropology on digital media and ethnography. Truth be told it is killing me as there is a 6000 word limit and 100-150 works one must mention and entertain (usually by throwing in some categorical statement that makes sense for 10+ works). One thing is clear: the literature on digital media by anthropologists is switching from trickle to steady and very interesting stream. Even if I Epically Fail, I have already learned a lot, which is what I keep telling myself as I struggle through the writing stage of the article. But if you want a taste of some recent work, there are some blog entries you can check out: Daniel Miller who was one of the first anthropologists to venture in this area (and kick-started the first program in digital anthro) at UCL has written a nice review of various books recently published. And for the same blog, I wrote an overview
of my work on hacking, liberalism, and pleasure. So if you want a short introduction to the books being published by anthropologists on digital media, I highly recommend checking Daniel Miller s post.

Earlier today the 33rd Chicago Marathon was held. I will organise my comments by pointing out three epic fails. First, of course, was the Chicago weather. After two successive marathon in excessive heat --- the 2007 race I completed, poorly, with its thousands of runners forced to abandon when the race was cancelled due to excessive heat, and the 2008 version which I skipped as I ran Berlin that year just weeks before Chicago, this year had forecasts of temperatures in the thirties and possibly snow the night before. Well, the weather cleared up -- but with those clear skies we still got a severe weather alert for the area due to frost! So at the (now earlier by 30 minutes) race start, it was very nippy and in the higher 30s, improving steadily under sunny skies. Overall, a little chilly and hence not exactly ideal, but not too bad in the grand scheme of things. A little windy coming up Michigan Avenue. But hey, Wanjiru still finished with a course record though still well off the world record. So maybe not a fail after all. The second fail, though, was my GPS which I had just blogged about yesterday (albeit indirectly). I had had my issues with the previous (much simpler and older) Forerunner 205 which lost satellite tracking when running downtown amidst the skyscrapers. I had high hopes that this newer model would do better. But no, not only did it loose track within the first few miles, it even managed to outdo the older model by turning its GPS tracking off in the later stages. Now that's a fail! The third and final fail, unfortunately, concerns my run. Training had gone well enough to let me hope for another decent race. However, a latent cold during the last two weeks had left me somewhat afraid I might not do well. Things felt better yesterday, and I ended up running a decent first eighteen miles to the waterstop the rest of my household was working. And thereafter: well let's just say that the wheels came off. I ended up with a 3:25:40 (or around 7:51 min/mile and 4:53 min/km) which is not terrible but also not one of my better races. I updated the two race data geekery charts shown here before to illustrate the fail. As I am running out of shades of blue for Chicago, I show the 2009 race in purple. A decent, flat chat indicating a reasonably steady pace throughout most of the race ... followed by one of the very worst finished. And as discussed above, I cannot really blame the weather either. I also updated my 'performance by race type through time' chart which suggests that I may be getting slower for marathons (if we allow for a non-linear effect). Eek.

Angie has my old laptop that some might remember from a couple Ubuntu developer summits. Adam Conrad used to refer to it as the surfboard: a lovely 15" widescreen laptop. It's my test machine for Ubuntu updates and so far I'm loving Karmic. This machine has been continually upgraded since either Hoary or Breezy without a reinstall.

Jaunty was miserable. Bugs in the Intel drivers meant that DVD playing stopped working, sounds would occasionally lock up, and things were just painfully slow. Karmic to the rescue. I did the update at beta time like I usually do, and without any further effort:

* Flash just worked.
* DVD playing in totem just worked.
* Boot time was slightly quicker
* nethack-gnome continues to work (despite being removed from the archive a few releases ago)
* Suspend/resume continues to just work.
* Atheros wireless continues to just work with the free kernel drivers.

I then added the dev channel of Google Chrome, which imported all the Firefox bookmarks and started to remove some optional things that Angie doesn't use anymore and don't run well on the now 5 year-old laptop. I took out openoffice, a pile of -dev packages that were leftover from when I used to build glibc on this thing (This laptop had the Hurd running on it at one point... *g*), and then discovered the System->Admin->Package Cleanup (Or at least, it's called something like that, the laptop is in French) where I could start to remove unused packages. I did a few rounds of that, some apt-get autoremove. I took universe and multiverse out of the packages list and did some more removals, making sure to leave in things like deborphan and the gstreamer packages that are in universe and multiverse. Lastly, I did a round of deborphan to take out some pieces that were just hanging out.

As a final step I've updated to grub2. I don't have a separate /boot partition on this machine, so I haven't been brave enough to convert it to ext4 yet.

The boottime at this point still feels a little long for what this laptop does. While at Canonical, I played with some appliance configs and had from kernel load to running firefox in a dozen seconds or some. This laptop doesn't have anything I'd see as unusual for daemons - there's no mysqld, sshd, or anything unneeded like that. But to have everything Just Work(tm) at this stage is amazing.

Nice job. =)

update: Angie's asked me to figure out why the screen saver kicks in after 30 seconds instead of 10 minutes like it's set to. But still, if this is the worst that goes wrong with this release, then it's still a wild success already at Beta.

Hey all! I m sorry I haven t been very active with my packages recently. I all-of-a-sudden started grad school and have been swamped with studying. I also started a contract and have been busy trying to learn a new codebase while contributing something other than snark. I promise I ll get back to packaging IronRuby and IronPython on Mono for Debian as soon as things start settling down. Getting an A in the class is higher priority, though, sorry Don t worry. I haven t forgotten about you ;) Cheers, C.J. PS, I am implementing a Perl library to exercise my understanding of the class. You can follow along at the search.cpan.org page for Lingua::HPSG or by cloning the git repo: $ git clone git://karma.colliertech.org/colliertech/langparser

Marvell made a new version of U-Boot for the SheevaPlug available last month. This version, 3.4.19, integrates SD card support and improves support for FAT32 USB partitions (although USB support in general is still a bit flaky). This page describes how to install this new version of U-Boot on your SheevaPlug. First of all, download the SheevaPlug U-Boot 3.4.19. Unpack the zip file and copy the file u-boot-3.4.19/u-boot-rd88f6281Sheevaplug_400db_nand.bin to your TFTP server. I suggest you call the file u-boot.bin. Then start your SheevaPlug and type the following: The upgrade process will ask whether you want to change the environment: Make sure to answer "n" here. We also need to make a small change to the u-boot configuration because this version of u-boot no longer accepts the console parameter we have been using. Thanks to Tom Rinehart for pointing this out. Instead of console, we'll therefore add a new bootargs_console and update bootcmd to make use of it: You'll see that bootcmd contains $(console). You have to set a new bootcmd and change the console parameter to $(bootargs_console). In my case, I'd type: Make sure you don't forget the quotes! Finally, save the environment and restart your machine:

Debian maintainer: Some days ago I had the great announcement that my Debian Maintainer s application was accepted and thanks to Jonathan McDowell my key is finally into the debian-maintainers keyring. (which is now part of the debian-keyring itself thanks to the ftp-masters / keyring maintainer work that made the changes on both DAK and keyring) I applied on the 22th of August, and thanks to Reinhard s advocation, everything went on the right side. Many thanks to all the people who made this possible. Pkg-mozext: Around 3 weeks ago I started being involved with the pkg-mozext team working on the new policy and to mozilla-devscripts transition for all iceweasel / icedove extensions. The work is going great so far and we managed to update several extensions that are now part of the team itself. For a fast look at our package list, feel free to browse pkg-mozext s QA page. A lot of work needs to be done, so if anyone wanna jump in, I would suggest to read our guidelines / policy into the team s Wiki Page. Pkg-gnome: GNOME 2.28 is now released, and the pkg-gnome team is now working hard to provide all the needed updates into unstable and experimental. Everything is going great and we are cleaning up everything listed on our schedule. Ephipany is under transition (gecko > webkit) so there are still some open issues / bugs, but now it looks fine so far and it can be used without any particular problem.