Search Results: "xam"

17 October 2009

Eugene V. Lyubimkin: Debian packages with big number of bugs

While working on Debian, I often don't realize how some important packages lack manpower to handle issues. Examples that touched me are 'apt' and 'perl'. As someone told many times, it would be good if someone with free time, especially newcomers to Debian, help triaging/reproducing bugs in the BTS, writing patches etc. This activity usually doesn't require helper to be a DD or DM, and will usually be more appreciated than preparing 11th web server or 30th window manager to Debian archive. Just ask maintainer(s). Also, you may want to read http://wiki.debian.org/HelpDebian to get the wider picture.

I tried to quickly find a table with Debian packages with highest bug rates over the Debian BTS, but failed, so I constructed my own one, with the help of UDD. This list contains BTS packages with number of bugs > 100 and is available here. The top ten:


          source            rc_bugs   all_bugs 
--------------------------+---------+----------
 wnpp                             0       3024
 installation-reports            10       2119
 apt                              4        682
 linux-2.6                        0        617
 aptitude                         0        576
 iceweasel                        3        490
 openssh                          1        417
 evolution                        0        362
 util-linux                       1        344
 dpkg                             1        343

14 October 2009

Robert McQueen: Telepathy Q&A from the Boston GNOME Summit

The first Telepathy session session on Saturday evening at the Boston GNOME Summit was very much of a Q&A where myself and Will answered various technical and roadmap issues from a handful of developers and downstream distributors. It showed me that there s a fair amount of roadmap information we should do better at communicating outside of the Telepathy project, so in the hope its useful to others, read on MC5 Ted Gould was wondering why Mission Control 4 had an API for getting/setting presence for all of your accounts whereas MC5 does not. MC5 has a per-account management of desired/current presence which is more powerful, but loses the convenience of setting presence in one place. Strictly speaking, doing things like deciding which presence to fall back on (a common example being if you have asked for invisible but the connection doesn t support it) is a UI-level policy which MC should not take care of, but in practice there aren t many different policies which make sense, and the key thing is that MC should tell the presence UI when the desired presence isn t available so it could make a per-account choice if that was preferable. As a related side point, Telepathy should implement more of the invisibility mechanisms in XMPP so it s more reliably available, and then we could more meaningfully tell users which presence values were available before connecting, allowing signing on as invisible. Since MC5 gained support for gnome-keyring, its not possible to initialise Empathy s account manager object without MC5 prompting the user for their keyring password un-necessarily (especially if the client is Ubuntu s session presence applet and the user isn t using Empathy in the current session but has some accounts configured). Currently the accounts D-Bus API requires that all properties including the password are presented to the client to edit. A short-term fix might be to tweak the spec so that accounts don t have to provide their password property unless it s explicitly queried, but this might break the ABI of tp-glib. Ultimately, passwords being stored and passed around in this way should go away when we write an authentication interface which will pass authentication challenges up to the Telepathy client to deal with, enabling a unified interface for OAuth/Google/etc web token, Kerberos or SIP intermediate proxy authentication, and answering password requests from the keyring lazily or user on demand. Stability & Security Jonathan Blandford was concerned about the churn level of Telepathy, from the perspective of distributions with long-term support commitments, and how well compatibility will be maintained. Generally the D-Bus API and the tp-glib library APIs are maintained to the GNOME standards of making only additive changes and leaving all existing methods/signals working even if they are deprecated and superseded by newer interfaces. A lot of new interfaces have been added over the past year or so, many of which replace existing functionality with a more flexible or more efficient interface. However, over the next 4-6 months we hope to finalise the new world interfaces (such as multi-person media calls, roster, authentication, certificate verification, more accurate offline protocol information, chat room property/role management), and make a D-Bus API break to remove the duplicated cruft. Telepathy-glib would undergo an ABI revision in this case to also remove those symbols, possibly synchronised with a move from dbus-glib to GVariant/etc, but in many cases clients which only use modern interfaces and client helper classes should not need much more than a rebuild. Relatedly there was a query about the security of Telepathy, and how much it had been through the mill on security issues compared to Pidgin. In the case of closed IM protocols (except MSN where we have our own implementation) then we re-use the code from libpurple, so the same risks apply, although the architecture of Telepathy means its possible to subject the backend processes to more stringent lockdowns using SElinux or other security isolation such as UIDs, limiting the impact of compromises. Other network code in Telepathy is based on more widely-used libraries with a less chequered security history thus far. OTR The next topic was about support for OTR in Telepathy. Architecturally, it s hard for us to support the same kind of message-mangling plugins as Pidgin allows because there is no one point in Telepathy that messages go through. There are multiple backends depending on the protocol, multiple UIs can be involved in saving (eg a headless logger) or displaying messages (consider GNOME Shell integration which previews messages before passing conversations on to Empathy), and the only other centralised component (Mission Control 5) does not act as an intermediary for messages. Historically, we ve always claimed OTR to be less appealing than native protocol-level end-to-end encryption support, such as the proposals for Jingle + peer to peer XMPP + TLS which are favoured by the XMPP community, mostly because if people can switch to an unofficial 3rd party client to get encryption, they could switch to a decent protocol too, and because protocol-level support can encrypt other traffic like SRTP call set-up, presence, etc. However, there is an existing deployed OTR user base, including the likes of Adium users on the Mac, who might often end up using end to end encryption without being aware of it, who we would be doing a disservice by Telepathy not supporting OTR conversations with these people. This is a compelling argument which was also made to me by representatives from the EFF, and the only one to date which actually held some merit with me compared to just implementing XMPP E2E encryption. Later in the summit we went on to discuss how we might achieve it in Telepathy, and how our planned work towards XMPP encryption could also help. Tubes We also had a bit of discussion about Tubes, such as how the handlers are invoked. Since the introduction of MC5, clients can register interest in certain channel types (tubes or any other) by implementing the client interface and making filters for the channels they are interested in. MC5 will first invoke all matching observers for all channels (incoming and outgoing) until all of them have responded or timed out (eg to let a logger daemon hook up signal callbacks before channel handling proceeds), all matching approvers for incoming channels until one of them replies (eg to notify the user of the new channel before launching the full UI), and then sending it to the handler with the most specific filter (eg Tomboy could register for file transfers with the right MIME type and receive those in favour to Empathy whose filter has no type on it). Tubes can be shared with chat rooms, either as a stream tube where one member shares a socket for others to connect to (allowing re-sharing an existing service implementation), or a D-Bus tube where every member s application is one endpoint on a simulated D-Bus bus, and Telepathy provides a mapping between the D-Bus names and the members of the room. In terms of Tube applications, now we ve got working A/V calling in Empathy, as well as desktop sharing, and an R&D project on multi-user calls, our next priority is on performance and Tube-enabling some more apps such as collaborative editing (Gobby, AbiWord, GEdit, Tomboy ?). There was a question about whether Tube handlers can be installed on demand when one of your contacts initiates that application with you. It d be possible to simulate this by finding out (eg from the archive) which handlers are available, and dynamically registering a handler for all of those channel types, so that MC5 will advertise those capabilities, but also register as an approver. When an incoming channel actually arrives at the approval stage, prompt the user to install the required application and then tell MC5 to invoke it as the handler. Colin Walters asked about how Telepathy did NAT traversal. Currently, Telepathy makes use of libnice to do ICE (like STUN between every possible pair of addresses both parties have, works in over 90% of cases) for the UDP packets involved in calls signalled over XMPP, either the Google Talk variant which can benefit from Google s relay servers if one or other party has a Google account, so is more reliable, or the latest IETF draft which can theoretically use TURN relays but its not really hooked up in Telepathy and few people have access to them. XMPP file transfers and one-to-one tube connections use TCP which is great if you have IPv6, but otherwise impossible to NAT traverse reliably, so often ends up using strictly rate-limited SOCKS5 -ish XMPP proxies, or worse, in-band base64 in the XML stream. We hope to incorporate (and standardise in XMPP) a reliability layer which will allow us to use Jingle and ICE-UDP for file transfers and tubes too, allowing peer to peer connections and higher-bandwidth relays to enhance throughput significantly. Future Ted Gould had some good questions about the future of Telepathy
Should Empathy just disappear on the desktop as things like presence applets or GNOME Shell take over parts of its function? Maybe, yes. In some ways its goal is just to bring Telepathy to end users and the desktop so that its worth other things integrating into Telepathy, but Telepathy allows us to do a lot better than a conventional IM client. Maemo and Sugar on the OLPC use Telepathy but totally integrates it into the device experience rather than having any single distinct IM client, and although Moblin uses Empathy currently it has its own presence chooser and people panel, and may go on to replace other parts of the user experience too. GNOME Shell looks set to move in this direction too and use Telepathy to integrate communications with the desktop workflow. Should Telepathy take care of talking to social networking sites such as Facebook, Twitter, etc? There s no hard and fast rule Telepathy only makes sense for real-time communications, so it s good for exposing and integrating the Facebook chat, but pretty lame for dealing with wall posts, event invitations and the like. Similarly on the N900, Telepathy is used for the parts of the cellular stack that overlap with real-time communications like calling and SMS, but there is no sense pushing unrelated stuff like configuration messages through it. For Twitter, the main question is whether you actually want tweets to appear in the same UI, logging and notification framework as other messages. Probably not anything but the 1-to-1 tweets, meaning something like Moblin s Mojito probably makes more sense for that. Later in the summit I took a look at Google Latitude APIs, which seem like something which Telepathy can expose via its contact location interface, but probably not usefully until we have support for metacontacts in the desktop. Can/will Telepathy support IAX2? It can, although we d have to do a local demultiplexer for the RTP streams involved in separate calls. It s not been a priority of ours so far, but we can help people get started (or Collabora can chat to people who have a commercial need for it). Similarly nobody has looked at implementing iChat-compatible calling because our primary interest lies with open protocols, but if people were interested we could give pointers its probably just SIP and RTP after you dig through a layer of obfuscation or two.
If you want to know more about Telepathy feel free to comment with some follow-up questions, talk to us in #telepathy on Freenode, or post to the mailing list.

13 October 2009

David Pashley: Tarballs explained

This entry was originally posted in slightly different form to Server Fault If you're coming from a Windows world, you're used to using tools like zip or rar, which compress collections of files. In the typical Unix tradition of doing one thing and doing one thing well, you tend to have two different utilities; a compression tool and a archive format. People then use these two tools together to give the same functionality that zip or rar provide. There are numerous different compression formats; the common ones used on Linux these days are gzip (sometimes known as zlib) and the newer, higher performing bzip2. Unfortunately bzip2 uses more CPU and memory to provide the higher rates of compression. You can use these tools to compress any file and by convention files compressed by either of these formats is .gz and .bz2. You can use gzip and bzip2 to compress and gunzip and bunzip2 to decompress these formats. There are also several different types of archive formats available, including cpio, ar and tar, but people tend to only use tar. These allow you to take a number of files and pack them into a single file. They can also include path and permission information. You can create and unpack a tar file using the tar command. You might hear these operations referred to as "tarring" and "untarring". (The name of the command comes from a shortening of Tape ARchive. Tar was an improvement on the ar format in that you could use it to span multiple physical tapes for backups).
# tar -cf archive.tar list of files to include
This will create (-c) and archive into a file -f called archive.tar. (.tar is the convention extention for tar archives). You should now have a single file that contains five files ("list", "of", "files", "to" and "include"). If you give tar a directory, it will recurse into that directory and store everything inside it.
# tar -xf archive.tar
# tar -xf archive.tar list of files
This will extract (-x) the previously created archive.tar. You can extract just the files you want from the archive by listing them on the end of the command line. In our example, the second line would extract "list", "of", "file", but not "to" and "include". You can also use
# tar -tf archive.tar
to get a list of the contents before you extract them. So now you can combine these two tools to replication the functionality of zip:
# tar -cf archive.tar directory
# gzip archive.tar
You'll now have an archive.tar.gz file. You can extract it using:
# gunzip archive.tar.gz
# tar -xf archive.tar
We can use pipes to save us having an intermediate archive.tar:
# tar -cf - directory   gzip > archive.tar.gz
# gunzip < archive.tar.gz   tar -xf -
You can use - with the -f option to specify stdin or stdout (tar knows which one based on context). We can do slightly better, because, in a slight apparent breaking of the "one job well" idea, tar has the ability to compress its output and decompress its input by using the -z argument (I say apparent, because it still uses the gzip and gunzip commandline behind the scenes)
# tar -czf archive.tar.gz directory
# tar -xzf archive.tar.gz
To use bzip2 instead of gzip, use bzip2, bunzip2 and -j instead of gzip, gunzip and -z respectively (tar -cjf archive.tar.bz2). Some versions of tar can detect a bzip2 file archive with you use -z and do the right thing, but it is probably worth getting in the habit of being explicit. More info:
Read Comments (8)

Jonathan McDowell: What to wear?

I tend to dress quite casually - unless there's a good reason to do otherwise I'll be in jeans and a t-shirt, or something similar. I'm comfortable in it and I take the view point that in general people shouldn't be making assumptions based on what I'm wearing. Of course they do, and while this can be infuriating at times it can also be amusing. The man at the RBoS who witnessed me signing the personal DD guarantee form for Black Cat obviously couldn't reconcile my appearance with what I was there to do, for example.

I'd never thought about this from the other side until a few weeks ago. I spoke to someone who explained how difficult it had become to conduct business meetings with other companies with the increased level of business casual. In fact on occasion different branches of the same company that he'd be meeting with would have different dress codes, so he'd go to something involving casual dress in the morning and something with full business suit attire in the afternoon and end up feeling over or under dressed.

This isn't likely to make me change my own behaviour in the immediate future (I don't have to interact with external parties as part of my current job). It did help me realise that there was another explanation for awkward behaviour when I'm my usual scruffy self that wasn't just about judging on appearances though.

(All of the above blatantly obvious once you think about it, but it took that conversation for me to do so.)

12 October 2009

Stefano Zacchiroli: ocaml-autoconf 1.1

new OCaml Autoconf release: 1.1 We have just released version 1.1 of OCaml Autoconf: a set of M4 macros for integrating your OCaml project with autoconf. It is mostly a regular maintenance release with some bug fixes, some new features, and most notably a new mini-tutorial and example by Richard Jones (including how to deal with automake). Check the news for more info.

11 October 2009

David Pashley: mod_proxy or mod_jk

This entry was originally posted in slightly different form to Server Fault There are several ways to run Tomcat applications. You can either run tomcat direcly on port 80, or you can put a webserver in front of tomcat and proxy connections to it. I would highly recommend using Apache as a front end. The main reason for this suggestion is that Apache is more flexible than tomcat. Apache has many modules that would require you to code support yourself in Tomcat. For example, while Tomcat can do gzip compression, it's a single switch; enabled or disabled. Sadly you can not compress CSS or javascript for Internet Explorer 6. This is easy to support in Apache, but impossible to do in Tomcat. Things like caching are also easier to do in Apache. Having decided to use Apache to front Tomcat, you need to decide how to connect them. There are several choices: mod_proxy ( more accurately, mod_proxy_http in Apache 2.2, but I'll refer to this as mod_proxy), mod_jk and mod_jk2. Mod_jk2 is not under active development and should not be used. This leaves us with mod_proxy or mod_jk. Both methods forward requests from apache to tomcat. mod_proxy uses the HTTP that we all know an love. mod_jk uses a binary protocol AJP. The main advantages of mod_jk are: A slight disadvantage is that AJP is based on fixed sized chunks, and can break with long headers, particularly request URLs with long list of parameters, but you should rarely be in a position of having 8K of URL parameters. (It would suggest you were doing it wrong. :) ) It used to be the case that mod_jk provided basic load balancing between two tomcats, which mod_proxy couldn't do, but with the new mod_proxy_balancer in Apache 2.2, this is no longer a reason to choose between them. The position is slightly complicated by the existence of mod_proxy_ajp. Between them, mod_jk is the more mature of the two, but mod_proxy_ajp works in the same framework as the other mod_proxy modules. I have not yet used mod_proxy_ajp, but would consider doing so in the future, as mod_proxy_ajp is part of Apche and mod_jk involves additional configuration outside of Apache. Given a choice, I would prefer a AJP based connector, mostly due to my second stated advantage, more than the performance aspect. Of course, if your application vendor doesn't support anything other than mod_proxy_http, that does tie your hands somewhat. You could use an alternative webserver like lighttpd, which does have an AJP module. Sadly, my prefered lightweight HTTP server, nginx, does not support AJP and is unlike ever to do so, due to the design of its proxying system.
Read Comments (3)

10 October 2009

David Pashley: Name-Based HTTPS

This entry was originally posted in slightly different form to Server Fault There are two methods of using virtual hosting with HTTP: Name based and IP based. IP based is the simplest as each virtual host is served from a different IP address configured on the server, but this requires an IP address for every host, and we're meant to be running out. The better solution is to use the Host: header introduced in HTTP 1.1, which allows the server to serve the right host to the client from a single IP address. HTTPS throws a spanner in the works, as the server does not know which certificate to present to the client during the SSL connection set up, because the client can't send the Host: header until the connection is set up. As a result, if you want to host more than one HTTPS site, you need to use IP-based virtual hosting. However, you can run multiple SSL sites from a single IP address using a couple of methods, each with their own drawbacks. The first method is to have a SSL certificate that covers both sites. The idea here is to have a single SSL certificate that covers all the domains you want to host from a single IP address. You can either do this using a wildcard certificate that covers both domains or use Subject Alternative Name. Wildcard certificates would be something *.example.com, which would cover www.example.com, mail.example.com and support.example.com. There are a number of problems with wildcard certificates. Firstly, every hostname needs to have a common domain, e.g. with *.example.com you can have www.example.com, but not www.example.org. Secondly, you can't reliably have more than one subdomain, i.e. you can have www.example.com, but not www.eu.example.com. This might work in earlier versions of Firefox (<= 3.0), but it doesn't work in 3.5 or any version of Internet Explorer. Thirdly, wildcard certificates are significantly more expensive than normal certificates if you want it signed by a root CA. Subject Alternative Name is a method of using an extension to X509 certificates that lists alternative hostnames that are valid for that certificate. It involves adding a "subjectAltName" field to the certificate that lists each additional host you want covered by the certificate. This should work in most browsers; certainly every modern mainstream browser. The downside of this method is that you have to list every domain on the server that will use SSL. You may not want this information publicly available. You probably don't want unrelated domains to be listed on the same certificate. It may also be difficult to add additional domains at a later date to your certificate. The second approach is to use something called SNI (Server Name Indication) which is an extension in TLS that solves the chicken and egg problem of not knowing which certificate to send to the client because the client hasn't sent the Host: header yet. As part of the TLS negotiation, the client sends the required hostname as one of the options. The only downside to this is client and server support. The support in browsers tends to be better than in servers. Firefox has supported it since 2.0. Internet Explorer supports it from 7 onwards, but only on Vista or later. Chrome only supports it on Vista or later too. Opera 8 and Safari 8.2.1 have support. Other browsers may not support it. The biggest problem preventing adoption is the server support. Until very recently neither of the two main webservers supported it. Apache gained SNI support as of 2.2.12, which was released July 2009. As of writing, IIS does not support SNI in any version. nginx, lighttpd and Cherokee all support SNI. Going forward, SNI is the best method for solving the name-based virtual hosting of HTTPS, but support might be patchy for a year or two yet. If you must do HTTPS virtual hosting without problems in the near future, IP based virtual hosting is the only option.
Read Comments (14)

8 October 2009

Rapha&#235;l Hertzog: 3-way merge of Debian changelog files

I ve been considering for some time now to create a merge tool specifically suited for debian/changelog files. My goal was to let Git use it automatically thanks to gitattributes. I ve just gone ahead, so let me introduce you git-merge-dch. Grab it with git clone git://git.debian.org/~hertzog/git-merge-dch.git, you can build a package if you wish. Beware, you need to have a dpkg-dev 1.15.5 that is not yet published (so you need to build dpkg from its git repository, git clone git://git.debian.org/dpkg/dpkg.git) as I rely on features that I introduced recently you will also need the libalgorithm-merge-perl package. Using it in a git repository requires two changes: Now you can safely maintain two branches of a package with changelog files evolving separately and merge one into the other without creating undue conflicts. Suppose you created an experimental branch for version 2.28 (you use a version 2.28-1~exp1) when 2.26.2 was current stable in the master branch. In the mean time, 2.26.3 got out and was packaged in master. Next time you merge stable into experimental, the changelog entries for 2.28 and 2.26.3 won t collide despite being at the same place in the changelog file compared to the common ancestor. Let s continue with this example, 2.28 is out. Instead of adding a new changelog entry with New upstream release without further changes, you keep the current changelog entry and simply change the version into 2.28-1. While preparing this you discover a branch with fixes that was based on 2.28-1~exp1, if you merge it it will reintroduce a 2.28-1~exp1 entry that you don t want. Fortunately you can use the --merge-prereleases (-m) option of git-merge-dch so that it strip the prerelease part of the version string and considers 2.28-1~exp1 and 2.28-1 to be the same entry really. The only limitation is that this merge tool will remove any lines which are not parsed by Dpkg::Changelog (and which in theory are not supposed to be there). Feel free to test, share your comments, report bugs and send patches! Partagez cet article / Share This

Barry Hawkins: More Expensive and Complicated Equals Better: Carseats and Software

So I finally got around to checking out the TED site; I ve quickly become a fan. One of the first talks I watched was Steven Levitt s child carseats talk. Both the talk and the feedback comments on the TED site reminded me of things I see in software development. Here s the video if you haven t seen it.
<object height="326" width="334"><param name="movie" value="http://video.ted.com/assets/player/swf/EmbedPlayer.swf"><param name="allowFullScreen" value="true"><param name="wmode" value="transparent"><param name="bgColor" value="#ffffff"><param name="flashvars" value="vu=http://video.ted.com/talks/dynamic/StevenLevitt_2005G-medium.flv&amp;su=http://images.ted.com/images/ted/tedindex/embed-posters/StevenLevitt-2005G.embed_thumbnail.jpg&amp;vw=320&amp;vh=240&amp;ap=0&amp;ti=30&amp;introDuration=16500&amp;adDuration=4000&amp;postAdDuration=2000&amp;adKeys=talk=steven_levitt_on_child_carseats;year=2005;theme=unconventional_explanations;theme=presentation_innovation;theme=how_the_mind_works;theme=not_business_as_usual;event=TEDGlobal+2005;&amp;preAdTag=tconf.ted/embed;tile=1;sz=512x288;"><embed allowfullscreen="true" bgcolor="#ffffff" flashvars="vu=http://video.ted.com/talks/dynamic/StevenLevitt_2005G-medium.flv&amp;su=http://images.ted.com/images/ted/tedindex/embed-posters/StevenLevitt-2005G.embed_thumbnail.jpg&amp;vw=320&amp;vh=240&amp;ap=0&amp;ti=30&amp;introDuration=16500&amp;adDuration=4000&amp;postAdDuration=2000&amp;adKeys=talk=steven_levitt_on_child_carseats;year=2005;theme=unconventional_explanations;theme=presentation_innovation;theme=how_the_mind_works;theme=not_business_as_usual;event=TEDGlobal+2005;" height="326" pluginspace="http://www.macromedia.com/go/getflashplayer" src="http://video.ted.com/assets/player/swf/EmbedPlayer.swf" type="application/x-shockwave-flash" width="334" wmode="transparent"></embed></object>
Steven Levitt shows in his talk how 30 years of data on car crash fatalities imply that carseats do not outperform regular seatbelts for children ages 2 and up. Anyone who has a child or grandchildren will probably bristle at hearing that; as the father of a pre-schooler, it certainly gave me pause. We spent no small amount of time and decent chunk of money in selecting carseats for our child, thinking we had done our best to ensure our child s safety. For that matter, it would be illegal for us not to have done so. To see a decent-sized data sample suggesting my child would be better off in a seatbelt at her age is rather unsettling. By the end of the talk, I took away these observations: So, software professionals, does any of that sound familiar? It reminds me of numerous initiatives over the years that have led us inexorably to the software productivity morass we have slogged through for years now. I suppose the most heinous case study in my own experience would be J2EE and the insistence that it was the solution that any reasonable business application would choose for a platform. (Before you .Net folks jump on that one, DCOM and later the .Net enterprise stack was much the same.) And who hasn t read a benchmark or white paper with seemingly incontrovertible data depicted in highly-polished graphics insisting that Product Y is the one solution to address them all? I noticed that there were comments below the video on its page at the TED site, largely because the negative verbiage of the topmost comment jumped out at me. I took the time to read through them all (there were 36 at the time of this writing), and to my amusement, I saw parallels between them and how people react to questioning and examining our existing practices and means of developing software. See if any of these strike a chord: In recent years I ve been greatly encouraged by the willingness of companies to question whether or not the heavyweight frameworks and technology stacks are what they should be using. Helping companies slough off high-ceremony processes in favor of right-sized process that focuses on delivering the right software in a timely manner has been some of the more rewarding work I ve done the past several years. I think we have an encouraging number of people in the industry who are challenging the more expensive and complicated always equals better mantra. For the brave individuals willing to put these questions to the community at large, I hope you find some comfort in knowing that the resistance and rejection you will encounter is a thing to be expected; you are not alone in that respect. Here s hoping we continue to be open to self-examination, no matter what emotional responses it might provoke or what fear of the future it may stir up within us.

Joey Hess: ikiwiki dependency types

Like make(1), ikiwiki knows about dependencies between the files that it is building, and will rebuild a file if its dependencies change. I've always known that it was possible to greatly reduce the amount of work ikiwiki does, if it gets smarter handling of different types of dependencies. It would be as if make knew that changes to comments inside header files did not change the resulting binary, and avoided rebuilding it for such a change -- but did rebuild the doxygen documentation. Except that, in ikiwiki, this can be done sanely, since it figures out the dependencies by examing the wiki's files, not by using a Makefile. Also, there were some nasty bugs with certian complex dependencies not being handled right. Fixing one of them, the transitive dependency problem, would require ikiwiki doing significantly more work. So fixing all this has been on my todo list for a while. Though when I mentioned "transitive dependencies" at the ikiwiki BOF @ DebConf, eyes glazed over. :) And I expect I've lost most of my readers already here too.. Which is a pity, because I've just gotten to the fun stuff. So, a few (how many?!) days ago I started my current insane hackfest, and began adding dependency types into ikiwiki. The first special type was a presence dependency -- a dependency that only cares if a page is created, or deleted, but does not care about changes to the page. Next type was a links dependency -- a dependency that only cares if the page changes what pages it links to. That was great, except it often didn't work. For features like gathering "posts/*" together into a blog, or displaying a tag cloud of "tags/*", ikiwiki uses a PageSpec. This is a small, domain specific language, that can specify a set of pages with lots of control. For example, "(index or backlink(index)) and created_after(oldpage)". The problem is that when using a PageSpec like the one above in a presence dependency, it doesn't only matter if pages are added/removed. Certian changes to "oldpage" or "index" can also change what it matches. So now, I had this PageSpec DSL, that I needed to somehow analyise to find these influences. While thankfully not turing-complete, parsing it to extract the influences would still be annoying. Except: Operator overloading to the rescue! The solution was this perl:
package IkiWiki::FailReason;
use overload (
        '""'    => sub   $_[0][0]  ,
        '0+'    => sub   0  ,
        '!'     => sub   bless $_[0], 'IkiWiki::SuccessReason' ,
        '&'     => sub   $_[0]->merge_influences($_[1]); $_[0]  ,
        ' '     => sub   $_[1]->merge_influences($_[0]); $_[1]  ,
        fallback => 1,
);
package IkiWiki::SuccessReason;
use overload (
        '""'    => sub   $_[0][0]  ,
        '0+'    => sub   1  ,
        '!'     => sub   bless $_[0], 'IkiWiki::FailReason' ,
        '&'     => sub   $_[1]->merge_influences($_[0]); $_[1]  ,
        ' '     => sub   $_[0]->merge_influences($_[1]); $_[0]  ,
        fallback => 1,
);
Ikiwiki already dealt with pagespecs by translating them to perl (which is then compiled (yes, perl can), and happily memoized). A pagespec is translated to something like this:
(match_glob("index")   match_backlink("index")) & match_created_after("oldpage")
The match_* functions return IkiWiki::SuccessReason or IkiWiki::FailReason objects. These objects are special in a lot of ways; they act like boolean values most of the time, but if viewed as a string, you get a user-understandable message saying why the pagespec didn't match. And when ANDed or ORed together, they build up a list of influences, that can be accessed at the end to get all the influences for the whole pagespec. (Why am I using what's supposed to be bitwise operations & and , rather than "and" and "or"? Because perl doesn't seem to consider the latter to be operators. Odd.) Anyway, this stuff should be available in an ikiwiki release just as soon as I've finished making it smoking fast.

3 October 2009

Bernhard R. Link: An argument for symbol versioning

A little example for why it is nice to have symbol versioning in libraries. Safe the following as test.sh. Call without arguments: segfault; call with argument "half": segfault; call with argument "both": works.
#!/bin/sh
cat >s1.h <<EOF
extern void test(int *);
#define DO(x) test(x)
EOF
cat >libs1.c <<EOF
#include <stdio.h>
#include "s1.h"
void test(int *a)  
	printf("%d\n", *a);
 
EOF
cat >libs1.map <<EOF
S_1  
 global:
  test;
 ;
EOF
cat >s2.h <<EOF
extern void test(int);
#define DO(x) test(*x)
EOF
cat >libs2.c <<EOF
#include <stdio.h>
#include "s2.h"
void test(int a)  
	printf("%d\n", a);
 
EOF
cat >libs2.map <<EOF
S_2  
 global:
  test;
 ;
EOF
cat >a.h <<EOF
void a(void);
EOF
cat >liba.c <<EOF
#include "s.h"
#include "a.h"
void a(void)  
	int b = 4;
	DO(&b);
 
EOF
cat > test.c <<EOF
#include "a.h"
#include "s.h"
int main()  
	int b = 3;
	DO(&b);
	a();
	return 0;
 
EOF
rm -f liba.so libs.so* test s.h
if test $# -le 0   test "x$1" != "xboth" ; then
gcc -Wall -O2 -shared -o libs.so.1 -Wl,-soname,libs.so.1 libs1.c
else
gcc -Wall -O2 -shared -o libs.so.1 -Wl,-soname,libs.so.1 -Wl,-version-script libs1.map libs1.c
fi
if test $# -le 0 ; then
gcc -Wall -O2 -shared -o libs.so.2 -Wl,-soname,libs.so.2 libs2.c
else
gcc -Wall -O2 -shared -o libs.so.2 -Wl,-soname,libs.so.2 -Wl,-version-script libs2.map libs2.c
fi
ln -s libs.so.1 libs.so
ln -s s1.h s.h
gcc -Wall -O2 -shared -o liba.so -Wl,-soname,liba.so liba.c -L. -ls
gcc -Wall -O2 test.c -L. -ls -la -o test
rm libs.so s.h
ln -s libs.so.2 libs.so
ln -s s2.h s.h
gcc -Wall -O2 -shared -o liba.so -Wl,-soname,liba.so liba.c -L. -ls
LD_LIBRARY_PATH=. ./test

Andrew McMillan: I still know where it came from...

I publish my photos on the internet. Well, I do when I get around to updating it, anyway. When I do so, I include some information about licensing them. I say to people "I'm very open to people wanting to use these pictures for something else, and will be happy to release photos into the public domain on request." so when I discover one of my photos abused by a travel website I do wonder why I didn't hear about it. Well it seems I also upload a lot of photos to the Burning Well public domain photo repository or to Wikimedia Commons where I also mostly mark them as PD-self. Still, you don't expect a photo of Cuilapan Monastery near Zaachila, Oaxtepec, Mexico to appear on a page about Edinburgh Castle, but really. Am I just being picky? Is this the norm? Are there really people all over the intawebs taking this little care about what photos they use for this sort of thing? I realise there isn't much sunshine in my photos of Scotland, and my photos of Edinburgh Castle, but really. Scotland is renowned for it's preponderance of rain over sunshine and it is considerably glossing the reality of a visit to the real Edinburgh Castle to even suggest the sun might appear. On closer examination I see that the site is actually talking about some weird nutter's home in Jamaica. Nothing to do with anywhere I've ever been, though if I ever visit Jamaica I probably would actually visit the place they're talking about. Now that I know about it, anyway, and especially if I was with Max who claims his favourite castles are Craigmillar & Tolquhon because most of the others aren't at quite the right stage of ruination. Oh well. I guess by putting the photo into the public domain I gave up responsibility for them, and it does seem that it might be hard to find real photos of the ruin in question (let alone free ones). On the other hand I do feel obliged to point and laugh, because that photo isn't Jamaica by a considerable distance, and just because I PD a photo I should still be well within my rights to have a good old giggle when I see a Mexican monastery linked to a Scottish castle. Even one in Jamaica.

1 October 2009

Eddy Petri&#537;or: Some Debian work in a long time

I finally managed to do some Debian work in a long time, I managed to package Wormux 0.8.5 for Debian Sid (aka Unstable).

This new version delivers one new binary package, wormux-servers, which contains the stand alone (aka headless) game server and the index server. Unfortunately, the information on setting these up resumes to have a couple of configuration files which are placed in /usr/share/wormux/examples/ .

Also, there's a bug that affects network games when using the construction tool, but fortunately, upstream already has some fixes, including one for this bug, and as soon as 1:0.8.5-1 enters squeeze, I'll prepare a patched version (including some other fixes present in upstream).

Until then, please enjoy Wormux.

Martin F. Krafft: Thesis submitted to examiners

At 14:45 CEST, today, 1 October 2009, I submitted my thesis to the print shop. Sarah has kindly agreed to pick it up tomorrow and drop it off at Brian s office, from where it will go to the three examiners. On 30 October at 9:30 IST, I will stand to publicly defend my work at the University of Limerick. I expect to receive a list of changes to be made from the examiners, hopefully minor, such that there won t be much in the way before I submit the final version to the University Board by 3 December. Once they give their green light, I will make it available on my research website, along with the data. I expect this to happen in January 2010. I have many many people to thank and will do so in due time. Right now I just want to thank Brian Fitzgerald and Kieran Conboy for their flexibility. I was supposed to have the thesis done by the end of August to given them a month to read it over, but by then, I only had the first six (of eight) chapters done. We hence spent all of September sending back and forth PDFs and paper comments, working iteratively so that today, I could put the finishing touch on the work and keep the schedule. The next thing on my agenda is the second edition of my book, which we are targetting for Debian squeeze . If you would like to be the first to receive more details, sign up to the announcement mailing list.

30 September 2009

Dirk Eddelbuettel: RInside release 0.1.1, and a fresh example

Last week's 0.1.0 release of RInside, and the first to have been published on CRAN, still had some issues with builds and use on OS X. Thanks to testing and fixes by Jan de Leeuw, Jeff Horner and particularly Simon Urbanked, things are said to be better now with the new release 0.1.1 which went onto CRAN yesterday. So no new features, but fixes to the main Makefile as well as the Makefile for the examples directory, some minor fixes and editing for the examples. I also added a file THANKS to show some appreciation for the various patches and fixes I have been receiving -- they are appreciated! However, today I committed a new example to SVN archive at R-Forge. It is based on this thread on r-devel. Abhijit Bera tries to do this in C, but to me his questions provide rather clear motivation for showing how much simpler things can be via C++ and the Rcpp classes along with RInside. Using a small example, the task was to pass a weight vector to a portfolio solver from the Rmetrics package fPortfolio and to then access the computed solution. The original poster struggled with access from C to the S4 classes used by fPortfolio and could not set the weights. But when using RInside, we simply pass a C++ vector of weights down to R, solve the problem and pass a solution vector back using the handy evaluation of R expressions:
// -*- mode: C++; c-indent-level: 4; c-basic-offset: 4;  tab-width: 8; -*-
//
// Another simple example inspired by an r-devel mail by Abhijit Bera
//
// Copyright (C) 2009 Dirk Eddelbuettel and GPL'ed
#include "RInside.h"                    // for the embedded R via RInside
#include "Rcpp.h"                       // for the R / Cpp interface used for transfer
#include <iomanip>
int main(int argc, char *argv[])  
    try  
        RInside R(argc, argv);          // create an embedded R instance
        SEXP ans;
        std::string txt = "suppressMessages(library(fPortfolio))";
        if (R.parseEvalQ(txt))          // load library, no return value
            throw std::runtime_error("R cannot evaluate '" + txt + "'");
        txt = "lppData <- 100 * LPP2005.RET[, 1:6]; "
	  "ewSpec <- portfolioSpec(); "
	  "nAssets <- ncol(lppData); ";
        if (R.parseEval(txt, ans))      // prepare problem
            throw std::runtime_error("R cannot evaluate '" + txt + "'");
	const double dvec[6] =   0.1, 0.1, 0.1, 0.1, 0.3, 0.3  ; // choose any weights you want
	const std::vector<double> w(dvec, &dvec[6]);
	R.assign( w, "weightsvec");	// assign STL vector to R's 'weightsvec' variable
	txt = "setWeights(ewSpec) <- weightsvec";
        if (R.parseEvalQ(txt))		// evaluate assignment
            throw std::runtime_error("R cannot evaluate '" + txt + "'");
	txt = "ewPortfolio <- feasiblePortfolio(data = lppData, spec = ewSpec, constraints = \"LongOnly\"); "
	  "print(ewPortfolio); "
	  "vec <- getCovRiskBudgets(ewPortfolio@portfolio)";
        if (R.parseEval(txt, ans))      // assign covRiskBudget weights to ans
            throw std::runtime_error("R cannot evaluate '" + txt + "'");
	RcppVector<double> V(ans);      // convert SEXP variable to an RcppMatrix
	R.parseEval("names(vec)", ans);	// assign columns names to ans
	RcppStringVector names(ans);
	for (int i=0; i<names.size(); i++)  
	  std::cout << std::setw(16) << names(i) << "\t"
		    << std::setw(11) << V(i) << "\n";
         
      catch(std::exception& ex)  
        std::cerr << "Exception caught: " << ex.what() << std::endl;
      catch(...)  
        std::cerr << "Unknown exception caught" << std::endl;
     
    exit(0);
 

Chris Lamb: GeoDjango and the UK postcode database

This post describes how to play with the UK postcode database that recently turned up on WikiLeaks using GeoDjango. You will need the following pieces of free software:
  1. PostgreSQL (MySQL does not provide sufficient geospacial features) I will be using Postgres 8.4.1; please adjust your paths for your version and/or distributor.
  2. PostGIS spacial database extensions I will be using version 1.4.0.
  3. Django I will be using version 1.1. This version includes the GeoDjango framework.
  4. GEOS geometry engine I will be using version 3.0.0 from the libgeos-c1 package in Debian unstable.
You will also need a copy of the leaked postcode data. Usage of this data is almost certainly a copyright violation.
Getting started First, create a Django project and application for this example:
$ django-admin startproject ukpostcodes
$ cd ukpostcodes
$ ./manage.py startapp postcodes
Next, edit settings.py:
  • Configure DATABASE_*. You must use the postgresql_psycopg2 PostgreSQL database driver.
  • Add django.contrib.gis & ukpostcodes.postcodes to INSTALLED_APPS.
If you have not already done so, you must install PostGIS in your database:
$ sudo -u postgres createlang plpgsql -d <databasename>
$ sudo -u postgres psql <databasename> -f /usr/share/postgresql/8.4/contrib/postgis.sql
$ sudo -u postgres psql <databasename> -f /usr/share/postgresql/8.4/contrib/spatial_ref_sys.sql
Modelling a postcode Now define a simple model to store imported postcodes:
from django.contrib.gis.db import models
class Postcode(models.Model):
    name = models.CharField(max_length=8, db_index=True)
    location = models.PointField()
    objects = models.GeoManager()
  • We import models from django.contrib.gis.db instead of django.db.
  • We override the default objects manager with GeoDjango's GeoManager() so we can perform spacial queries.
Save this model in postcodes/models.py. We can now create this model in our database using:
$ ./manage syncdb
Importing postcode data We will use a management command to import the data. First, create the necessary modules with:
$ mkdir -p postcodes/management/commands
$ touch postcodes/management/commands/__init__.py
$ touch postcodes/management/__init__.py
Next, save the following code in postcodes/management/commands/import_postcodes.py:
import sys
import csv
from django.contrib.gis.geos import Point
from django.core.management.base import BaseCommand
from postcodes.models import Postcode
class Command(BaseCommand):
    def handle(self, \*args, \*\*options):
        Postcode.objects.all().delete()
        count = 0
        for row in csv.reader(sys.stdin):
            try:
                name = row[0].upper().strip().replace('  ', ' ')
                location = Point(map(float, row[13:15]))
            except ValueError:
                print "I: skipping %r" % row
                continue
            Postcode.objects.create(name=name, location=location)
            count += 1
            if count % 10000 == 0:
                print "Imported %d" % count
Finally, we can import the data using:
$ bunzip2 /path/to/uk-post-codes-2009.bz2   ./manage.py import_postcodes
This takes approximately one hour on my computer. If you plan to import the data multiple times, consult the Django documentation for faster methods of importing initial data.
Quering the postcode database We can now query our postcode database. First, let's find the postcode for Buckingham Palace:
>>> from postcodes.models import Postcode
>>> palace = Postcode.objects.get(name='SW1A 1AA')
>>> palace
<Postcode: SW1A 1AA>
Next, let's find all postcodes within 1 mile. For this, we will need to construct a tuple representing the area we want to search in.
>>> from django.contrib.gis.measure import Distance
>>> area = (palace.location, Distance(mi=1))
>>> Postcode.objects.filter(location__distance_lte=area)
[<Postcode: SE1 7BF>, <Postcode: SE1 7SG>, <Postcode: SE1 7JA>, <Postcode: SE1 7PB>,
 <Postcode: SE1 7PD>, <Postcode: SW1A 1DW>, <Postcode: SW1A 1EA>, <Postcode: SW1A 1EE>,
 <Postcode: SW1A 1EF>, <Postcode: SW1A 1EG>, '...(remaining elements truncated)...']
>>> Postcode.objects.filter(location__distance_lte=area)
7024
To find "neighbour" postcodes, we can pass a Point object to the .distance(..) method on our queryset this adds a field we can then sort on:
>>> Postcode.objects.distance(palace.location).order_by('distance')[:4]
[<Postcode: SW1A 1AA>, <Postcode: SW1E 6JP>, <Postcode: SW1E 6LA>, <Postcode: SW1E 6WG>]
Putting it all together Let's pretend we are making an app to find pizza from an arbitrary user-submitted postcode. (We will assume we have parsed the user's postcode and have found its Postcode object, but we will continue to use Buckingham Palace here.) First, define the following model in postcodes/models.py and re-run ./manage.py syncdb:
class PizzaJoint(models.Model):
    name = models.CharField(max_length=200)
    postcode = models.ForeignKey(Postcode)
    objects = models.GeoManager()
    def __unicode__(self):
         return self.name
Now, add some local pizza places:
>>> def add_pizza_joint(name, loc):
...     code = Postcode.objects.get(name=loc)
...     PizzaJoint.objects.create(name=name, postcode=code)
...
>>> add_pizza_joint('Pizza Hut', 'SW1E 6SQ')
>>> add_pizza_joint('Pizza Express', 'SW1E 5NA')
>>> add_pizza_joint('Pizza On The Park', 'SW1X 7LY')
If we were not using GeoDjango, we would have to be content with simply matching the postcode:
>>> PizzaJoint.objects.filter(postcode=palace)
[]
However, we can now find the cloest pizza places:
>>> PizzaJoint.objects.distance(palace.location,
            field_name='postcode__location').order_by('distance')
[<PizzaJoint: Pizza Hut>, <PizzaJoint: Pizza Express>, <PizzaJoint: Pizza On The Park>]
The .distance(..) method also sets a distance attribute on all the objects returned; this makes it easy to show the actual distance to the user:
>>> for x in PizzaJoint.objects.distance(palace.location,
                     field_name='postcode__location').order_by('distance'):
...    print "%s (%d metres away)" % (x, x.distance.m)
...
Pizza Hut (499 metres away)
Pizza Express (509 metres away)
Pizza On The Park (902 metres away)
For more details, see the GeoDjango documentation.

27 September 2009

Steve Kemp: Looks like me an Vincent caught you boys at breakfast

It is interesting that Fran ois Marier recently posted a brief "howto" document on debugging problems caused by overly-agressive filtering with privoxy, as I've recently been having problems with that tool. My home network frequently changes configuration depending on what I'm concentrating upon, but every few months I'll start/cease using the following tools: Recently I was experimenting with XSS attacks against various browsers, which meant using them for real. As not all browsers have the same anti-advert setups I was running privoxy to filter out web-annoyances, and I spotted a major flaw with it. Unfortunately I can only describe the problem, not reproduce it, or track it down. I'm 80% certain the bug is in privoxy, but the stack is suitably high that determining that for sure is problematic. In short the issue is that HTTP requests would end up being sent to the wrong host: After that clicking around consistently sends requests to the first HTTP host which was accessed succesfully. So, for example, attempting to visit http://foo.com/bar/ will send the request to http://steve.org.uk/bar - which then gives a 404. In terms of setup I use a dnsmasq DNS cache, privoxy and iceweasel from Debian unstable. From the symptoms I'm not sure if iceweasel's "KeepAlive" system is to blame, or if privoxy has a bad cache of hosts. Perhaps it is dnsmasq returning bogus DNS data, or my cable connection itself having DNS issues. Anyway once the symptoms present themselves closing the browser and restarting the cache fixes it. Until the next time which might be hours or days later. I'd report it as a bug - but I don't know where it should be. Privoxy caching things it shouldn't? iceweasel having keepalive issues? dnsmasq returning wrong DNS entries? I'd ask "Have you seen this before, internet world?" but I guess if you have tracked it down it'd be fixed by now, and it clearly isn't! Anyway for the moment I've uninstalled privoxy. ObFilm: Pulp Fiction

26 September 2009

Pablo Lorenzzoni: Ruby versus Python

This is not another rant to praise one in spite of the other (an everybody knows I love Ruby, so it would not be impartial), but sometimes people seems to live in another world and do things for the wrong reasons. I just read this blog post by Kanwei Li in which he gives 2 or 3 reasons he ditched Ruby in favor of Python. First of all, both are great languages and, although I favor Ruby, I use Python for some projects and they are not all that different. Of course, everyone is free to choose which language one favors, but Kanwei seems to be ditching Ruby out of not knowing much about it, or out of preferring one style over the other His first reason is that in Python white spaces matter. I used to think this is just a matter of style, but every now and then mandatory alignment hurts me (just try to put together a code generator and you ll notice it). Although my code is always correctly aligned, I like that it s done so because I want it that way, and not because some language demands it. Rants and more rants have been written about Python s mandatory alignment (or other languages lack of it), and I am not going through all of it Just I don t think it s a good reason to ditch Ruby After, he makes a big deal out of Ruby s ternary if. As written by him, he prefers

#python
if len(a) > 0:
        v = a[0]
        a = a[1:]
else:
        v = None
over Ruby s ternary if

#ruby
v = a.empty? ? a.shift : nil
Hey! Come on Ruby s ternary if is not mandatory It was copied from C just as a syntax sugar. You can do without it, just as in Python:

#ruby
if ! a.empty?
    v = a.shift
else
    v = nil
end
Better yet! you can use if s return as v value:

#ruby
v = if ! a.empty?
    a.shift
else
    nil
end
How beautiful is that! Python lacked ternary if for a long time, and when it finally acquired one via PEP 308 its syntax was made different from every other language! Although I don t think that is a problem, some people might think it would be better not reinventing the wheel. Next, Kanwei goes over a famous problem of Ruby: the lack of a sum method for Array. I admit it s strange, but that is completely coherent: Ruby s Arrays are ordered collection of objects and not mathematical arrays. How do you sum objects that are not numbers? Many different people will have many different answers to that, so Ruby leaves this decision for the programmer and provides basic methods to deal with collections of anything (that can be used to apply sum to numbers, if wished). So, in Ruby you have to use Array#inject to perform a sum:

[1,2,3].inject(0)    sum, value  sum + value  
Array#inject (actually Enumerable#inject) was borrowed from Smalltalk and allows you to loop through an array, building up an accumulator value as you go. When it s done, the final value of this accumulator is returned. Very useful for combining array elements, whether by summing them, building up a pretty display string, whatever. In the example above, I am initializing the accumulator with 0. If you use Array to mathematical operations and you want your arrays to work that way, you can always add a sum method to Array class:

class Array
    def sum
        self.inject(0)  sum, value  sum + value 
    end
end
Maybe it would be better if you just use Arrays as containers (as it was intended to) and implement that sum inside your own class I completely agree with Reg Braithwaite here. Kanwei also mention Python is faster than Ruby. That is true, but was more true some time ago. First of all, Python is older and has had more time to improve its speed. Ruby, ITOH, just now acquired a good VM and improvements to it finally can run parallel to improvements in the language itself, so I am expecting this to be less true every release. Python is already not getting much faster between releases, unlike Ruby (the differences between 1.8.7 and 1.9.1 are really impressive!). IMHO this is not a good reason to choose one instead of the other: if you really need speed, go for C :-) Now this is something I find interesting Kanwei has mentioned: Python is more production ready . He argues that Google is using it, so it must be good. Well I cannot argue against that: Google is really using Python. But IBM, Oracle, EA, Cisco, Siemens, etc are using Ruby so that is just a matter of preferring one or another company. Both are production ready I agree, though, that Ruby 1.9.1 has many differences from 1.8.7, and that that may be seen as some inconsistency, but Python also has changed a lot since its 2.0 version, for that matter. And the changes to Ruby brought many benefits I think they worth it. At last, Kanwei compares Python and Ruby docstrings. Here I also have to agree with him: Ruby docstrings sucks. Actually that s why everybody uses rdoc instead (and that is much more powerful than Python s docstrings). Again, I don t think that is reason enough to ditch Ruby (actually, the existence of rdoc, rubygems & friends should bring people to Ruby instead), but that is a matter of personal taste. Surely, Kanwei s reasons were easy to argue against. There are areas were Python shines much more than Ruby (and vice-versa), but those Kanwei mentioned are not among them. I think both languages are powerful enough, and both are way better than Perl or PHP, so either one you choose would be fine. Better if you don t have to choose and use both ;). If you have to, ITOH, pay more attention on how you feel while coding in each one, and not to some cheap reasons such as above. If you are a programmer, what matters most is that you ll spend a lot of time coding with any given language let that be something pleasant then.

25 September 2009

Julian Andres Klode: cupt and how to write package managers


cupt is a new package manager written in Perl by Eugene V. Lyubimkin, who previously contributed to APT. And more than all, the project makes no sense at all. First of all, there is a language issue. Implementing a package manager in Perl has some major drawbacks. One of the features of APT was it being written in a lower-level language (i.e. C++ which really is below Perl), making it possible to write applications like synaptic and python bindings which in turn lead to applications like gnome-app-install or Ubuntu s new Software Store. Furthermore, writing a package manager in Perl means that Distributions such as Emdebian might not be able to use it since they have excluded Perl due to its space requirements. This becomes even more important considering that cupt depends on even more perl libraries. This means that cupt will never be able to replace APT. Secondly, a package manager should not be designed specifically for one distribution. This is another major drawback of cupt and other package managers such as yum or zypper. The smart package manager, written in Python and funded by Canonical Ltd. is an example for a distribution-neutral package manager. Now let s take a look at package management in modern distributions. Usually we have two levels of package managers, the first being tools like dpkg and rpm which take care of installing and removing the packages and level-2 package managers implementing dependency resolvers and package retrieval from remote locations. Recently, distributions started to add a third layer named PackageKit, which shall provide distribution-independent package management user interfaces. The project was well-received by RPM-based distributions, but failed in Debian-based distributions due to not supporting debconf. Furthermore, adding a third layer just increases the possibility of problems. The right way to do package management is a distribution-independent level-2 package manager written in C. The smart project shows us that this is possible although itself fails to meet the lower-level language (C) requirement. That s why I decided to write a package manager in Vala, a GObject-based language which gets converted to C and then compiled. If successful, this project will be able to replace most of the current level-2 package managers and will also provide the same distribution-independence as provided by a level-3 package manager such as PackageKit. It is also easy to create binding for other programming language such as Python or Perl thus enabling application developers to choose the language they like most. The core of this project is a vendor-neutral library, temporarily called libapt (as the project is called APT2 for now). This library contains all the code which is not specific to a vendor i.e. file retrieval, dependency resolver, caches, etc and is then enhanced by several vendor-specific plugins, each implementing a PackageManager (interface to the distribution s level-1 package manager) and a Repository (well, repositories from which you can download packages) interface. We could even enhance the vendor-independent interface to include more details of a repository. Most repositories nowadays consist of 4 components: A meta index (Release files for Debian, repomd.xml for Fedora/openSUSE/etc.), a package index (e.g. Packages files for Debian, *primary.xml.gz on Fedora, etc. ), a source index (e.g Sources files in Debian) and a files index (e.g. Contents-*.gz for Debian). I took a look at the repository formats of Slackware, openSUSE and Fedora and it seems that this concept can be applied to all of them. So maybe all we need are distribution dependent parsers for those files. One of the most important issues with APT is its use of mmap() for the cache. Using mmap() makes it hard to grow the cache, which is sometimes needed. We see a lot of bug reports from people with too small cache sizes. We can circumvent this problem by utilizing an embedded database like SQLite for this, but we would probably loose some speed and it may be harder to maintain a flexible API. We should see what the best option is here, both ways are possible since Vala 0.7.6 includes my patches for adding mmap(), ftruncate(), mremap() and some other functions. An idea to circumvent the mmap() issue is gathering statistics about the relation between the number of repositories and the size of the cache and then using a value which is slightly above the average statistical value. The project is not very mature yet, it only includes basic library functions for downloading files and parsing configuration files, etc. You can find the (MIT licensed) code at http://git.debian.org/?p=users/jak/apt2.git. I also have some local code for repository management and multi-threaded file fetching, but it s just not ready to be merged yet. Posted in APT2, Debian

Kumar Appaiah: Community at work - a success story

This is a positive event which happened to me recently in a Debian bug report, with regard to audio troubles. There are several positives, and I owe some gratitude to several people in the Debian (and ALSA) community, so I thought a blog post would be a good idea. It also shows the advantage in filing a well-written installation report to the Debian project. Here is the description of events.This was an extremely positive experience. While I agree that the number of people who would encounter this quirk may not be large, it makes me happy that at least some users who use Linux on this hardware will not have to face this issue anymore, irrespective of which distribution they use, as long as it carries an up-to-date ALSA.I wanted to share this experience with you, since this is a glowing example of how the "community" approach works perfectly for FOSS. Thanks to Elimar, for being a concerned maintainer, and to the ALSA upstream developers who were proactive in producing a fix to this problem. And, most of all, thanks to me, for filing an installation report; these things really work! (OK, tongue-in-cheek. But I did take the effort of filing it, so I deserve the credit, don't I?) :-)So, have you filed an installation report for your Debian installation? If not, I urge you to do so; I sincerely believe that filing good bugs and being proactive in helping developers zero in on the problem and fix is an important contribution to free software which any user can do. And the best way to start this process in Debian, is with an installation-report. :-)Tags: debian

Next.

Previous.