Search Results: "xam"

23 April 2008

Joey Hess: cdbs killer (design phase)

simple case Imagine a very simple debian/rules, that should work for most packages w/o modification:
#!/usr/bin/make -f
build:
    dh build
clean:
    dh clean
binary-arch:
    dh binary-arch
binary-indep:
    dh binary-indep
binary: build binary-arch binary-indep
absurdly simple case
AKA debian/rules zen guru level.
#!/usr/bin/make -f
%:
    dh %@
the complex cases Often you'll want to handle configuring by hand. (And maybe, run automake too.)
build:
    dh build --before configure
    #aclocal
    #autoheader
    #automake --add-missing
    #autoconf
    ./configure --kitchen-sink=yes
    dh build --after configure
Here's how to skip two steps in a row (configure and build):
build:
    dh build --before configure
    ./mondoconfig
    make universe-explode-in-delight GROOVINESS=420
    dh build --after build
Another common case is wanting to run some code manually after a particular debhelper command is run.
binary: build
    dh binary --until dh_fixperms
    # dh_fixperms has run, now override it for one program
    chmod 4755 debian/foo/usr/bin/foo
    # and continue
    dh binary
Suppose you hate dh_strip, and just don't want it to run. (This might be a contrived example; if people frequently don't want to run a few commands in a set, it may be better to have -X exclude them.)
binary: build
    dh binary --before dh_strip
    dh binary --after dh_strip
Maybe you want to run a command out of order. This should be no problem:
binary: build
    dh binary --until dh_fixperms
    dh_install foo usr/bin
    dh binary --after dh_fixperms
A separate install target can be added if you like. And maybe you want to run make install by hand in it. (This is tricky because dh binary needs to avoid running dh_auto_install which was skipped over in the install target.)
install: build
    dh install --before install
    $(MAKE) install PREFIX=debian/tmp
    dh install --after install
binary: install
    dh binary
Once you have an install target, you can do this to run some commands manually after the install target runs, and before the remaining steps are run to build a binary.
binary: install
    dh_strip -X foo
    dh_fixperms -X bar
    dh binary --remaining
If the --remaining was left off, dh binary would only run commands in its list after dh_fixperms. With --remaining, it runs commands that haven't run before, avoiding running dh_strip and dh_fixperms again. A more elaborate way to do the same thing, without needing a separate install target is this.
binary: build
    dh binary --before dh_strip
    dh_strip -X foo
    dh binary --after dh_strip --before dh_fixperms
    dh_fixperms -X bar
    dh binary --after dh_fixperms
the dh_auto_* commands dh_auto_* are thin wrappers, with enough intelligence to handle 90% of cases (autofoo, perl/python packages, etc) automatically. These will rarely be run by hand, instead if they don't do what is needed, they can just be skipped. the dh command dh runs a set of other debhelper commands in order (and prints their names as it does so). The set that it runs is determined by its parameter. The sets of commands are: Possibly dh does smart optimisations to avoid running lesser-used commands if they will do nothing. (Or not.) Each debhelper command will record when it's successfully run in debian/package.log.debhelper. (Which dh_clean deletes.) So dh can tell which commands have already been run, for which packages, and skip those. Options passed to dh are passed on to each command it runs, which is useful for common options like -v, -X, and the like. For that matter, you can use it for uncommon options as well, since commands will ignore options that don't apply to them. dh has a few options of its own: cmd can be a full name of a debhelper command, or a substring. It'll first search for a command exactly matching the name, to avoid any ambiguity. For example, "--before dh_installdeb" does not refer to dh_installdebconf. Otoh, "--before build" is much easier to type, and clearer, than "--before dhautobuild". If there are multiple substring matches, the last in the command list is taken. So "--before install" stops before dh_auto_install, and not before the earlier dh_install. Each time dh is run, it examines the log, and finds the last logged command that is in its list of commands to run. It then continues with the following command in its list. (Or the one after --after, if that was specified. Or if --remaining was specified, start at the first command in the list that is not in the log.) So... If you run debian/rules build twice, and it just has "dh build" in it, the second run will be a no-op. The same as if the build target touched a build-stamp. This is generally considered a feature, so I'll consider it a feature in dh too. (To force a rebuild, run dh_clean to remove the log.) If you run debian/rules binary, and a build hasn't happened yet, it will run the build steps first. Even if the rules file is a bit lame (or over-golfed), and doesn't have the binary target explicitly depend on the build target. Ditto for the install target. If a rules file runs a command out of order, it'll need to use --after to get dh back on track. If you manually run a debhelper command at the command line, running debian/rules binary will skip forward until after that command. Similarly, if debian/rules binary is interrupted (or fails), running it again will resume after the last debhelper command run. This is a bit of a behavior change, and could be considered a bug, but is easy enough to work around by running debian/rules clean binary. (Or by manually deleting the log.)

22 April 2008

Evan Prodromou: 3 Flor al CCXVI

I'm happy to point to the announcement that Wikitravel Press now has a guide to Paris. Paris has long been a targeted city for Wikitravel Press, and I'm really happy we were able to get this book out. Bonus: the Wikitravel Guide to Paris includes maps from OpenStreetMap, the Open Content geo database and mapping system. We've got some new software to overlay Wikitravel listings onto the maps, and it's working really well. Thanks to OSM, editor Mark Jaroski, and managing editor Jani Patokallio for getting this great book out... just in time for Paris spring! tags:

Podcasts I asked a couple of days ago (Journal/30 Germinal CCXVI) about podcasts on the subject of Open Source and Open Content. I wanted to summarize a few that came in through comments on my blog: This has turned into a pretty decent list, but I'd love to see some more diverse discussion of Open Content and Open Source in general. Please, feel free to send more tips. tags:

Vinismo For some reason Vinismo went over some tipping point with StumbleUpon this weekend, and we had a real flood of users from that bookmarking site. I can't say why that happens; I find SU to be a totally opaque Web site and service. In other news, Stevey likes our business cards. Me too! I think they look great. M-C Doyon, the Montreal graphic designer who laid out our Web site, also did our paper branding, and I think she did a great job. tags:

Yay Firefox 3.0 I don't know when it happened, but Firefox 3.0 beta does non-ASCII characters in the address bar correctly. So, if you're reading this entry on my site with FF3, you'll see the in "Flor al" rather than the URL-encoded "Flor%c3%a9al". I also like seeing Japanese Wikipedia pages in the address bar showing up correctly. Nice job, FF3 team. tags:

Planet software should preserve categories So, apparently there's once again rising problems with content drift on Planet Debian. Personally, I think this is a problem with a technical solution. All three main flavours of RSS support post categorization. Many, many kinds of feed software provide categories (see, for example, this feed). If the Planet software would preserve these categories and pass them through to the output RSS feeds, then people who really only want to read about Debian could filter the output feeds for category "debian". People who wanted to know about other parts of their fellow Debianistas' lives would just leave the feeds unfiltered. I'm not sure if the Planet software doesn't support categories in output at all, or if it's just an option that's turned off on Planet Debian. If nobody else wants to take a look, I can look into making this work. It seems like a pretty simple fix. tags:

21 April 2008

Russell Coker: The Purpose of Planet Debian

An issue that causes ongoing discussion is what is the purpose of a Planet installation such as Planet Debian [1]. The discussion usually seems to take the less effective form of what is “appropriate” content for the Planet or what is considered to be “abuse” of the Planet. Of course it’s impossible to get anything other than a rough idea of what is appropriate is the purpose is not defined, and abuse can only be measured on the most basic technical criteria. My personal use of Planet Debian and Planet Linux Australia [2] is to learn technical things related to Linux (how to use new programs, tricks and techniques, etc), to learn news related to Linux, and to read personal news about friends and colleagues. I think that most people have some desire to read posts of a similar nature (I have received a complaint that my blog has too many technical posts and not enough personal posts), but some people want to have a Planet with only technical articles. In a quick search of some planets the nearest I found to a stated purpose of a Planet installation was from the Wiki to document Planet Ubuntu [3] which says ‘Subscribed feeds ought to be at least occasionally relevant to Ubuntu, although the only hard and fast rule is “don’t annoy people”‘. Planet Perl [4] has an interesting approach, they claim to filter on Perl related keywords, I initially interpreted this to mean that if you are on their list of blogs and you write a post which seems to refer to Perl then it will appear - but a quick browse of the Planet shows some posts which don’t appear to match any Perl keywords. Gentoo has implemented a reasonable system, they have a Universe [5] configuration which has all blog posts by all Gentoo bloggers as well as a Planet installation which only has Gentoo related posts. It seems to me that the a reasonable purpose for Planet Debian would be to have blog feeds which are occasionally specific to Debian and often relevant to Debian. Personal blog posts would be encouraged (but not required). Posts which are incomprehensible or have nothing to say (EG posts which link to another post for the sole purpose of agreeing or disagreeing) would be strongly discouraged and it would be encouraged to make links-posts rare. Having two installations of the Planet software, one for posts which are specific to Debian (or maybe to Debian or Linux) and one for all posts by people who are involved with Debian would be the best option. Then people who only want to read the technical posts can do so, but other people can read the full list. Most blog servers support feeds based on tag or category (my blog already provides a feed of Debian-specific posts). If we were going to have a separate Planet installation for only technical posts then I expect that many bloggers would have to create a new tag for such posts (for example my posts related to Debian are in the categories Benchmark, Linux, MTA, Security, Unix-tips, and Xen) and the tag Debian is applied to only a small portion of such posts. But it would be easy to create a new tag for technical posts. Ubuntu is also the only organisation I’ve found to specify conditions upon which blogs might be removed from the feed, they say: We reserve the right to remove any feed that is inaccessible, flooding the page, or otherwise interfering with the operation of the Planet. We also have the right to move clearly offensive content or content that could trigger legal action. That is reasonable, although it would be good to have a definition for “flooding the page” (I suggest “having an average of more than two posts per day appear over the period of a week or having posts reappear due to changing timestamps”). Also the “could trigger legal action” part is a minor concern - product reviews are often really useful content on a Planet… Some time ago my blog was removed from Planet Fedora for some reason. I was disappointed that the person who made that change didn’t have the courtesy to inform me of the reason for their action and by the fact that there is no apparent way of contacting the person who runs the Planet to ask them about it. Needless to say this did not encourage me to write further posts about Fedora. If a blog has to be removed from a feed due to technical reasons then the correct thing to do is to inform the blogger of why it’s removed and what needs to be fixed before it can be added again. If a blog is not meeting the content criteria then I expect that in most cases the blogger could be convinced to write more content that matches the criteria and tag it appropriately. Having criteria for some aspects of blog quality and encouraging the bloggers to meet the criteria can only improve the overall quality. Currently there is a Planet installation on debian.net being recommended which is based on Planet Debian, but with some blogs removed (with no information available publicly or on debian-private as to what the criteria are for removing the blogs in question). It seems to me that if it’s worth using Debian resources to duplicate the Planet Debian then it should be done in a way that benefits readers (EG by going to the Planet vs Universe model that Ubuntu follows), and that if blogs are going to be removed from the feed then there should be criteria for the removal so that anyone who wants their blog to be syndicated can make whatever changes might be necessary.

Ingo Juergensmann: SNMP == Snmpd Not Memory Proof

Something I already noticed in my previous company is the fact that snmpd is suffering badly from memory leaks.
For example:


PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
13904 snmp 15 0 856m 837m 2268 S 0 64.2 0:13.02 snmpd


This is from an amd64 xen system, whereas on an i386 xen host it looks like:


PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3279 snmp 16 0 8064 1668 912 S 0.0 0.3 33:11.93 snmpd


Both are Etch systems, so same versions.

I've found Bug #421906 and my amd64 host is indeed using 2.6.18-6-xen-vserver-amd64 whereas my i386 is using 2.6.18-5-xen-686.
Does this mean that using 2.6.18-6-xen-amd64 would solve my snmpd problems, dear lazyweb?

20 April 2008

Julian Andres Klode: Fedora 9 & Kernel Mode Setting


Today, I decided to try out the new kernel mode setting feature in Fedora 9, which moves some stuff about video from userspace into the kernel. I tested this on my notebook, a HP Compaq 6720s with Intel X3100 (GM965) graphics controller. I downloaded the preview live image for x86_64 and booted with the i915.modeset=1 option. The boot was almost normal, except that it was flicker-free. After the system booted I switched the virtual terminal from Xorg to tty1 and back, and it was extremely fast. The terminals all had the same resolution. This does not mean that everything is perfect. There are a lot of things which do not work. For example, running glxgears and moving the window does not “look good”. Also, speed was a bit low. Normally, in Ubuntu Hardy, I get about 1000 FPS, in Fedora I only got 230 FPS. But the difference was not visible. Another thing which did not work was suspend & resume, because the graphics card seemed to be not correctly re-initialized. This is worse than Ubuntu & Debian. In Ubuntu, suspend & resume works almost every time. In Debian, it works except that the backlight is disabled in X. If these problems can be fixed, it would be very interesting to get this feature into Ubuntu Intrepid, maybe not as the default, but as an option for people who want to test it. Fedora 9 seems to be a fast and stable distribution, with experimental features for experienced users and developers. It also provides a nice application for configuring PulseAudio, like per-application volume settings, a functionality I miss in Ubuntu.

Steve Kemp: On the other side of the screen, it all looks so easy

I've updated the IMAP utility that I mentioned previously, which has now been given the name sift. It will accept, and process, a much simpler configuration file format keeping state as it goes. Here's my updated sample file:
username: blah.bah
password: pas.word
#
#  Comments are fine.
#
folder:livejournal status:new subject:temp mark:read exec:~/bin/notify
folder:foo status:new mark:read
folder:bar status:old exec:/usr/local/bin/record delete
Each line consists of a set of tokens, split by whitespace, which is "executed" in order. So the first line selects the folder "livejournal", finds messages which are "new", then each message containing "temp" in the subject is marked as read, and the program "notify" is executed once for each match. Essentially we keep a list of messages as "current" as we process each line, that list of messages is then refined as we move through the line. (When a folder is opened all messages are selected by default.) As a simple example to delete all the messages contained in a folder we'd use this:
folder:foo delete
To refine that to only delete messages from "fred" we'd say:
folder:foo from:fred delete
(If there were no matches the "delete" action wouldn't occur.) Consider each line of input a collection of filters each operating on the previous result. Simple to understand, simple to extend with more operations, and simple for me to code! TODO: Add a "move:xxx" to move a message to folder "xxx", and a bit more polish, then release. ObQuote: Tron.

Russell Coker: Planets and Resignations

Recently a Debian Developer resigned from a position of responsibility in the project by writing a blog post. I won’t name the DD or the position he resigned as I think that there are general issues which need discussion and specific examples will get in the way (everyone who is seriously involved will know who it is anyway - for those who don’t know, it’s not really exciting). Also I think that the issue of the scope of a Planet installation is of wider importance than the Debian project, so it would be of benefit for outsiders who stumble upon this to see a discussion of general issues rather than some disagreements within the Debian project. There has been some mild criticism of the DD in question for announcing his resignation via a blog post. I don’t think that this is appropriate. In the absence of evidence to the contrary I’ll assume that the DD in question announced his resignation to the relevant people (probably the team he worked with and the Debian Project Leader) via private email which was GPG signed (if he indeed intended to formally resign). The resignation of one DD from one of the many positions of authority and responsibility in the project is not going to have a great affect on the work of most DDs. Therefore I don’t think that it was necessarily a requirement to post to the debian-private mailing list (the main list for communication between all developers regarding issues within the project) about this. It was however an issue that was bound to get discussed on debian-private (given that the circumstances of the resignation might be considered to be controversial) so it seems to me that sending an email of the form “here is a blog post I’ve written about my resignation” would have saved some pointless discussion (allowing us to skip the “why didn’t you send email” and get right on to the main discussion). A resignation letter from a public position of responsibility is a significant document. Having such documents stored on publicly accessible places is good for the community. Having a record of all such documents that you have written stored on your own server for reference (by yourself and by other people you work with) is a good thing. Therefore it seems to me that a blog is an ideal place for a resignation letter. It used to be regarded that there was a certain formality in such things, and that a letter of resignation was required to be delivered in the most direct way possible (by hand if convenient) to the person who receives it. If such conventions were followed then a blog post would occur after the receipt of the letter of resignation had been confirmed (possibly in this case a confirmation email from the DPL). But in recent times things have become less formal and the free software community is particularly informal. So it seems quite appropriate to me to have the blog post come first and the email notification merely contain the URL. Now a letter of resignation is expected to contain certain specific details. It should say specifically what duties are being resigned (particularly important when a person performs many tasks), it should have a date from which it will take effect, and it might be appropriate to mention issues related to the hand-over of tasks (whether the person resigning is willing to work with their replacements). The “resignation” (if we should call it that) in question did not contain any of the specific details that I would expect to see in a formal resignation. This indicates to me that it could be interpreted as not being a formal and official resignation, but instead being a post (possibly written in haste while angry) about a situation which may not end up being an official resignation. Until we get some more information we won’t know for sure either way. This demonstrates one problem with blogs, people usually have a mixture of serious documents and trivial things on the one blog. It can be difficult to determine how seriously to take blog posts. I’m not sure that there can be a good solution to this. At the moment some people are suggesting that every DD should read Planet Debian [1]. I disagree with that. If there is an issue which is significant and affects the entire project then it should be announced on one of the mailing lists such as debian-private, debian-announce, or debian-devel-announce (and will be announced on one of them eventually even if not by the person closest to the events). Forcing every DD to read a lot of blog posts is not in the best interests of the project. Now we could create a separate Planet installation for such things, there is already Debian Times [2] and Debian Administration [3] which serve as a proof of concept. If there was a Planet installation for important stuff related to Debian which had it’s content syndicated in various ways (including an email gateway - Feedburner.com provides a quite useful one) then requesting that everyone read it’s content in some way (either by web browsing, an RSS feed reader, syndication in another Planet, email, or something else) would not be unreasonable. The volume of posts on such a Planet would be quite small (similar to the current announcement mailing lists) so if received by email it wouldn’t fill anyone’s mailbox and if people visited the web site they would only need to do so every second month if that suited them. The issue of what types of posts are suitable for Planet Debian is probably going to get raised again soon as a result of this.

19 April 2008

Matthew Garrett: Why Linux claims to be Windows

Section 5.7.2 of the ACPI specification provides for a method called _OSI. Platform firmware can query for various interfaces, which allow it to tweak its behaviour depending on the interfaces that the OS supports. Linux claims to support the following interfaces:which all correspond (unsurprisingly) to different versions of Windows. Linux used to be in there, but was removed for reasons I'll get into later.

The typical way in which this feature is used by a platform is by querying each of the interfaces in turn, starting with the most recent a firmware is aware of. After the first match, a value will be stashed somewhere and used elsewhere in the firmware when making certain decisions. An example of this may be only enabling certain devices (such as the HPET) if the OS claims to support Windows Vista.

As I mentioned, recent versions of Linux no longer claim to support the Linux interface. There's a couple of reasons for this. Firstly, some platforms break if they're told that they're running on Linux. It's usually an entirely untested codepath. Secondly, the whole point of the _OSI method is to provide the platform with information about the interface supported by an OS. "Linux" is too coarse grained - we make no guarantees that the interface between the OS and the platform will remain constant. "Linux 2.6.25" isn't all that much better, since distribution patches may alter the behaviour and BIOSes would become specific to a given kernel version. We could have something like "Linux ABI 1" and then bump that every time we altered the behaviour of the kernel in this respect, but this would become unwieldy fairly quickly (what would count as an ABI breaking change?)

So what kind of interface can we claim to provide? The sad truth of the matter is that most vendors will only test their code against Windows, and that as a result Windows provides the de-facto specification for OS/platform interaction. We could skip claiming to be Windows, but (in the best case) that means that the system will assume it's working with an OS that supports no advanced features. Rather than getting support for brightness changes being handled by the OS, we get stuck with SMM traps and unexpectedly missing time. If we want hardware to work to its full potential, we need to claim to be a recent version of Windows.

Does this mean the entire _OSI mechanism is useless? Not really. It's still useful for providing information about supported functionality. One example used in the spec is whether or not the OS implements the ACPI 3.0 thermal model. Another obvious (but unspecced) example might be whether or not the video opregion spec is implemented in the OS. _OSI strings which explicitly advertise the existence of advanced functionality are something that should be specified and encouraged. What they shouldn't be used for is as a way of advertising lacking functionality, which is effectively the only way that Linux can be usefully interpreted today. We don't want machines to automatically repost their video if they detect they're running Linux. It makes it harder to implement proper kernel-level graphics resume code. We don't want them to suddenly alter their keyboard behaviour, because that probably makes it harder to support older machines that don't have these workarounds.

Message to vendors: If we're not behaving in the same way as Windows does, let us know. We'll fix it. This has the benefit of fixing not only your latest and greatest platform, but also all the older ones that expected the same behaviour. Please don't try to work around our bugs in your BIOS - it just means that we need to special case you forever, and in the long run your hardware won't work as well as it would if we just fixed things properly.

Lior Kaplan: How to describe free software involvement in a CV?


I’ve started to write an updated version of my CV. This is the first version which should mention my involvement in free software, and I’m not sure what exactly to write. As I don’t know who would be the next employer, I don’t know how many term I can I is actually understandable. I could write I’m a Debian Developer. But how do people understand that? I added I do package building and maintenance. But I’m not sure a lot of people also understand that. I’m also not sure how to describe involvement in project, like openoffice.org, which I’m not officially a member. Although I have a lot of interest in this project, and I invest time on Hebrew related bugs, I don’t think I can write much more that “a community member”. Remember this is a CV and I don’t think I can put a few passages to describe my work in each project. The last thing I’m not sure about is how to mention integration work done based on experience. Most of them are too small to be mentioned by themselves, but I have a lot of them and that would be a waste not to mention them at all. I’ll be happy to get examples from other people who had to deal with the same problem. Thanks.

Enrico Zini: Laptop stolen

Laptop stolen Last thursday I flew from Italy to Manchester as usual, and while walking home from Levenshulme railway station my laptop has been snatched off me by a gang of thieves. I've managed to give the police all the details of the laptop including the serial number. I could also precisely describe to them the dynamics of the incident pointing at places over satellite maps in google maps. And give them the time of the theft with 10 seconds accuracy. And show them pictures of all stolen goods with a few simple internet searches. They were impressed. All sensitive data in the laptop are protected with one or two layers of strong encryption, and I have fresh backups, so the only work that I've lost was the work I did on the train and airplane on my way to the UK. If anyone around Manchester or Stockport happens to see, in a Cash Generator or second hand shop, a suspicious looking white ASUS laptop with a Taiwanese keyboard (US-style keyboard with extra Traditional Chinese and Bopomofo glyphs on it), please quietly walk out of the shop, alert the police and send me an e-mail. The day after the theft I managed to talk with a pub owner in the area, and there I learnt that pubs are networked and alert each other when suspicious people are roaming around. Lesson learned: if I see suspicious people around the street whem I'm walking home with my laptop, it's a good idea to go inside a pub and ask how's the situation. That night, for example, they've been alerting their customers of the danger. I wish the railway station had done the same.

18 April 2008

MJ Ray: Explaining web site improvements: what's important to you?

[Photo of Some Traffic]
Is this traffic or congestion? This is one of those Friday Afternoon Projects - it's been put off all through a busy week because it's unpaid, I'm not completely sure how to approach it and now my arms hurt like hell from travel jabs which are making it hard to concentrate! So I'm going to float it on here... I've been asked to brief a meeting next week about that group's current web site and its problems. I'm not linking it yet to avoid insulting/embarrassing them. The site looks OK, but doesn't rank well on search engines and doesn't allow much member participation. I need to explain why that's a bad thing and how the site's technical choices have led to that. I'm not directly pitching for TTLLP to get any work (because I'm a member of that group, it might be a conflict of interest and we're pretty busy anyway - even our own site needs work on some of the points I'm going to mention), but I don't want to be unhappy if we're asked to implement my recommendations. I've got a usual outline that I follow, but my presentation's time is limited, so I'd like to ask you: what about this is important and what isn't? If you give me useful feedback, I'll put you in the Acknowledgements with a backlink and I hope the briefing will be shared pretty widely over the next few months. The current plan is to start with a basic explanation of how search engines rank pages, as far as we can tell, referring to PageRank Explained Correctly with Examples, by Ian Rogers as well as the shorter official summaries from the dominant search sites at Yahoo, Microsoft and Google. Then I go through a quick evaluation of the site against the basics of validation, accessibility and robot-friendliness, followed by a couple of SEO-style checks of its current rankings and inbound links. Next is a bit different because I have access to some of their web access stats: I summarise what we know and suggest some other stats they've probably not considered and why they're useful, along the lines of Dave Briggs's measures of blog success. Finally, I suggest ways to improve the site. The top tip will be to take control of the site hosting and stop using the cheap and cheerful donated server that makes all links except the front page point to another domain. I'll probably suggest a mix of free and open source software tools to power it. If they don't want to move it all yet, I'll suggest running a second site for member participation, using tools like Wordpress, NoseRub and so on. What do you think? Plan for success, am I missing some tricks, or am I setting myself up for a lynching? Let me know with a comment or email, please.

17 April 2008

Neil Williams: Data Centric programming - a new start at Alioth.

It's time for me to stop complaining about the commercialisation of SourceForge and vote with my code. I'm leaving for brighter, bluer pastures at Alioth.

Query Object Framework is first (libqof1). The others will follow. I'm leaving because of unusable mailing list archives, impossibly slow web pages and wiki and the general amount of work involved in the SF interface. I'm gaining sane mailing lists, a friendly and usable ikiwiki and a simpler (although very similar) interface.

I'm also migrating most of those packages to subversion. With so many other changes, it's not exactly a problem to add cvs2svn into the mix.

Now, of course, this does not mean that these upstream packages suddenly become Debian-only or Debian-native. It does mean that I have a chance of encouraging new developers to join the team by providing the kinds of services and support that people need when starting off with a new project. (I suppose the one thing I don't yet have is an IRC channel. Not sure if it is worthwhile but I'll see what happens.)

QOF will form the basis of the support available for each of the other projects - all will share the same low-volume QOF-devel mailing list and QOF Wiki as provided by Alioth. I'll also migrate QuickList because I'm looking at restarting development on a stalled project allied to QOF and which could provide missing functionality in QuickList. I did look at Gaby in this mix but a botched Gtk1->Gtk2 transition that ended up with two incompatible forks running different parts of the original codebase put me off completely (and neither of the forks actually worked). (Some packages deserve to be dead upstream.)

I'm looking for new blood, fresh faces and a new desire to take these projects forwards. The driving force is data freedom - creating data-driven applications that are completely agnostic to where and how the data happens to be stored. It is user data, it should be as free as the source code that wrote it. Who cares what format it is in? Let's get it into a format that some other program can use and make use of the existing data instead of duplicating it all the time. (The observant will note that my old haunt of GnuCash is stubbornly ignorant of the merits of a decent import/export implementation that actually covers more than 50% of the data in the application. No wonder I took the chance to sponsor homebank.)

DWI Data With Interaction (aka DUI -- Data Under the Interface) is what can provide the missing interfaces. It'll take a while to get it back up to date and available in Debian but it can provide the link between data and interfaces. GNU/Linux users should be able to design an interface purely on the basis of the data that it needs to handle and not care one jot about how that data comes from one free software app and goes on to the next free software app. QOF is gaining a connection to libgda which means that data can go to and fro between a whole range of databases and file formats. Homebank and pilot-qof can gain new data inputs (and possibly outputs) by utilising the existing import/export support in new ways via DWI/QOF (as libqof2). That can give a complete interface for all PIM and financial data for a typical SOHO user and all based on SQL-type queries so that every stage of the process can be tweaked and optimised according to user requirements at runtime.

(Where does Emdebian come into this? Well, many people like me need to record and process PIM/financial data on the move so Emdebian means that those users can have the same data processing on their handhelds as on the desktop. Smaller apps to capture the data quickly and efficiently whilst on the move; larger apps to refine, optimise and filter the data on the desktop entirely under user control but with a simple, scriptable, interface based on runtime queries so that the user can do exactly the same conversions tomorrow with new data but none of the configuration. Easy!)

Is this groupware ? No - it is personal, ultimately flexible and entirely customised for particular users by the users themselves. It's about small niche markets, about independent business people with bespoke and highly specialised requirements. No one is going to spend time writing their reports in Scheme or Haskell, because they are the only people who need that particular report. Instead, that individual user needs to be empowered through a simple and intuitive interface to take a piece of data from one source, correlate it with a static piece of data from individual configuration sources, calculate a result according to a third piece of data from a completely separate source and combine them all into a new set of objects that can be used in other queries. It's about preparing invoices from Palm data, it's about handling customer accounts for a one-man company or handling a stocks and shares portfolio for a partnership. Ultimately customised, ultimately flexible and completely format-agnostic.

These are the kind of applications that hold people back from GNU/Linux migrations - bespoke applications that are developed for just a few small companies or just a few situations. Free software finds it hard to provide those apps because free software depends (currently) on someone else being motivated to write a bespoke program and that person rarely understands the niche. This is a chance to put the development into the hands of those with the motivation to do the work, by making the development process more WYSIWYG.

The question is: Can we come up with a better name for DWI, dear lazyweb?

"DWI is a fairly simple environment for quickly creating data-driven applications, that is, graphical applications that manipulate and show info from a database. This environment differs from others in that it is focused on native GTK/Gnome support through the Glade GUI designer, and thus allows you to build user interfaces as elegant as you can make them in Glade."
http://dwi.sourceforge.net/

The big appeal is that DWI is database-agnostic - it only cares about the data structure, not the file format. Ally that with QOF which provides data-centric SQL-based queries and conversions between objects (including combining elements from one object into one or more other objects without having to hardcode the connection between the objects at compile time) and we have an interesting and powerful combination of engines, frontends and backends that are all data-centric.

Please, if this sparks your interest, join the QOF-devel mailing list, have a play with the ikiwiki at http://qof.alioth.debian.org/ and have a go with the code. This whole area is begging for development but it is just so far from being usable by the target audience at this stage. There is a lot of programming to be done by current developers before the principles can be let loose on the masses.
:-)

This is about working between existing projects and between distributions - no longer dedicated to a single application, platform or file format, no more tunnel vision. Work on the edges of existing applications (and indeed projects), remove the sharp edges of applications that should work together but can't. The majority of the code in these libraries and applications is already stable and mature. Like so many parts of free software, it is time, IMHO, to put aside our egotistical devotion to the one-true-executable and come up with an actual solution.

Bah! I'm beginning to sound like a PR exec instead of a dedicated geek. What does it take to motivate people and get things done around here?
:-)

Joerg Jaspert: debadmin delegation - should I accept or deny it?

As you might have read on debian-devel-announce lately, Sam delegated the ftpmaster rights to me. Steve, our new DPL, did state his full support in an IRC query minutes later. Now I got a query from someone who is, pretty strongly, argumenting that I should decline this delegation. Mostly on the grounds that it is not nice to force new team members into an existing team, and that it is not a good environment to work in, if a DPL can do that if he feels like proving a point. Leaving alone the fact that delegations are one (central) part of the DPL job, and that the constitution leaves a way if you do not like those delegations (GR), I’m now unsure what to do. (Just for the record - i once was bitten by a DPL using his powers in a way I absolutely did hate (and still do). Some of you might remember the fun around the DebConf7 venue decision). I could do
  1. decline the delegation, stay ftp assistant only. Which I am for a long time now. Possibly see some flame on the lists when I do this.
  2. do nothing, ie. accept the delegation. That will get at least one of the long time ftpmasters upset, possibly more of them. Maybe with a resign from that job. Or Debian. But it would enable me to do the work only ftpmasters can do, like changes to the code, changes to the archive directly, etc. Without delays while waiting for a master to comment on something I prepared. Good example here is the current situation with the release teams transitions: They are 90% done and merged, just one little more merge is waiting to happen before they are ready to use… And whenever I tried to get a master (mostly Aj, as he was working with me on this) to do the final steps I got nothing but silence back, even when he was otherwise active…
Bah. I hate this. The thing is - I really do want to have debadmin rights, and am working in that direction for a long time now. Not publically, but by talking with James in the past months, … As Sams mail showed - that did go nowhere, unfortunately. Now, could readers of my blog (and I know there are people reading this, my traffic stats tell me so :) ) please comment on this and give me advice? I just want some opinions about this. (Yes, my blog does understand comments since some time :) ) UPDATE: Thanks to everyone who commented on this, either by using the comment form here, by mail or via IRC. I really appreciate your input. I’ve decided to not decline the delegation and one DSA member already added my account to the debadmin group. (Thanks).

16 April 2008

Simon McVittie: Implementing D-Bus services with telepathy-glib

16:46 < epmf> tp-glib is made of magic

—#telepathy, 2008-04-15

Rob pointed out today that I'd been promising to blog about telepathy-glib for several months and still haven't done so. I think there's too much to cover in one blog post, so I'll start with the oldest part - implementing a service (typically a Telepathy connection manager).

While telepathy-glib is (obviously) intended for Telepathy implementors, I think the ideas we've been implementing are likely to be useful for any D-Bus API, particularly flexible/complex APIs where an object implements many interfaces.

Introduction to telepathy-glib telepathy-glib started out as a collection of code extracted from our XMPP connection manager, telepathy-gabble, but at some point it grew beyond that into a wrapper around/partial replacement for dbus-glib. We've been quite pleased with it, really :-) My first major round of work on telepathy-glib, during the first half of 2007, was to split it out from telepathy-gabble (versions up to 0.5.9 were released as part of Gabble 0.5.x tarballs, in fact), leading to the release of the 0.6.x stable branch in late September. This was very helpful for implementing connection managers - Salut, Idle and telepathy-sofiasip (our connection managers for link-local XMPP, IRC and SIP) all started off by forking/cargo-culting from Gabble, and achieved huge code reductions by porting to telepathy-glib. Also, Will Thompson used it in his Summer of Code project, telepathy-haze, to go from nothing to a working and useful Telepathy implementation based on libpurple in a matter of a few months. The second major round of work, which I'll discuss in a later article, added client code to telepathy-glib, obsoleting the older/worse libtelepathy and allowing client programs like Empathy and telepathy-stream-engine to be written using telepathy-glib. In the process, I accidentally reinvented DBusGProxy :-) More details to come later!
Some background: service-side code generation
The secret Collabora code-generation process

(credit: daf)

Supporting D-Bus in the GObject world has always involved quite a lot of code generation. The core API of dbus-glib is heavily reliant on varargs functions, which aren't type-safe and are easy to get wrong. dbus-glib contains a program called dbus-binding-tool, which is meant to generate reasonably sane GObject APIs for D-Bus objects. Unfortunately, it seems to be intended to generate a whole GObject at a time. Early versions of telepathy-gabble used dbus-binding-tool plus a script called gengobject.py to generate API stubs for the exported objects; developers then filled in the blanks, and hey presto, we had a GObject on D-Bus. This was fine up to a point, but had a couple of major drawbacks. Whenever we changed the D-Bus API (quite common during the early development of Telepathy), there was a very laborious and error-prone merging process. We ended up with the following process:
  • copy the D-Bus introspection XML from the telepathy-spec repository into a directory xml/pristine (actually, the canonical form for the spec was Python for a while, and we had a script that exported a contrived object onto D-Bus and introspected itself to get the XML - but that's another story!)
  • preprocess the introspection XML to add the dbus-glib CFunctionName annotation, resulting in a directory xml/modified
  • run dbus-binding-tool and a Python script gengobject.py to generate C source in xml/src
  • three-way-diff the old version of xml/src, the new version of xml/src, and the real C code in src to create a new version of src
  • pray that the diff process hadn't randomly exchanged functions' implementations
  • update the resulting code in src so it worked
  • check the whole mess back into darcs
  • hope that nobody else had made changes that would result in darcs conflicts
If you haven't yet gathered, this was a bit of a nightmare. Also, it was very easy to return the wrong thing from a method without noticing - because all the APIs were varargs, the compiler didn't notice, and the first we'd know about it was a mysterious segfault under certain circumstances.
How telepathy-glib fixes code generation telepathy-glib improves on this by taking advantage of this innocuous-looking feature in dbus-glib:
commit 355ef78d98d6fc65715845d56232199162cab12a
Author: Steve Frécinaux <steve istique net>
Date:   Thu Aug 17 11:48:20 2006 +0200
    Interface support for bindings.
Instead of running dbus-binding-tool or creating GObjects with D-Bus interfaces, we put the entire Telepathy spec through glib-ginterface-gen.py, a distant descendant of Gabble's gengobject.py. For each interface in the spec, we generate a GInterface that mirrors the D-Bus interface. Any GObject that implements the GInterface automatically gets the D-Bus interface if it's exported onto D-Bus. There are a few non-obvious refinements, though:
  • The signature of the method implementation is always in the mode that dbus-glib calls "asynchronous", where the method implementation can either send a reply message before or after it returns. For "slow" methods, this is the only thing you can do. For instance, many Telepathy D-Bus methods can't return anything until some TCP round-trips to the server have happened. This is also the only thing you can do if you want to extract extra information, like the sender's unique name, from the method-call message (dbus-glib's "synchronous" API doesn't allow this to be done). For "fast" methods, sending a reply message before returning is just as easy as using the "synchronous" API (it's just a difference of syntax), and has an API consistent with that of the "slow" methods, making it easier for service authors.

  • The layout of the GInterface vtable is private, and (auto-generated) accessor functions are used to fill in implementations. This means our ABI doesn't change just because we re-ordered functions in the spec.

  • If no implementation is provided for a method, we just raise an appropriate error (org.freedesktop.Telepathy.Errors.NotImplemented), rather than suffering an assertion failure. This means we can safely add methods to an interface.

  • While we're generating code anyway, we generate some static inline wrapper functions which wrap dbus_g_method_return(), to have type-safe method returns. You can easily check that a method replies with correct types, by checking that the implementation of Foo() replies by calling tp_svc_some_interface_return_from_foo().

  • Similarly, we generate wrapper functions to emit signals, to get type-safe signal emission.

Not all interfaces are stable enough to be included in telepathy-glib's stable API and ABI, so some of our other projects include a copy of the telepathy-glib code-generation tools, and generate their own "mini-telepathy-glib" (traditionally in a /extensions/ directory) for their implementation-specific (drafts of) interfaces. This isn't yet as polished as it ought to be (mainly because we don't want to freeze the augmented-introspection-XML format that we write the Telepathy spec in, because it's rather ad-hoc and hackish in places) but it works quite well in practice. To see what this looks like in practice, have a look at the examples in telepathy-glib, or at a Telepathy connection manager like telepathy-gabble.
Mixins and base classes Another feature of telepathy-glib which makes life easier for connection manager authors is that it provides "mixins" and base classes which implement some of the GInterfaces. There's absolutely nothing magic about the base classes - they don't have any access to private implementation details of the GInterfaces, they just implement them like everyone else does. The "mixins" are only slightly more magical - they store some extra state inside the structures representing GObjects and their classes, and use it to provide default implementations of some or all of the methods on a particular interface (or two interfaces, in the case of the new TpMessageMixin). I'll leave it up to Rob to explain those in greater detail, since I seem to remember that he invented them :-P The examples and regression tests in telepathy-glib are quite good examples of how these work in practice.

Neil McGovern: Be careful what you post

In follow-up to my previous post, in which the phrase "google for gerri bird cambridge" (which turned up a rather poor lack of reply to a survey) was printed on some campain leaflets, I've come across a slightly different example of how traditional PR ideas may not work online if the people who are doing it don't understand the technology.

The Cambridgeshire County Council have placed on their transport page a link inviting people to view some videos of their new mis^WGuided Busway on YouTube. They've even gone so far as to create a YouTube account for the purpose. Unfortunately, the comments on their first video aren't too favourable, with the vast majority of people hating the idea. Not content with this, a second video was also posted, with exactly the same effect. In retaliation, it seems that their PR department has tried to fake some good reviews, but have been caught out.

So, some simple lessons:
  1. Don't trust user generated content to do your job for you
  2. Don't trust sites you have little control over
  3. Don't try and stuff the ballot by writing like a PR person

    Biella Coleman: My Special Topics Course on Hacking

    So I am nearly done with teaching this year, which is a relief, not because I don’t like it but because after a full year of teaching, one naturally wants a break. But before it is even over, you have to start thinking about your fall courses, mostly so you can order books in time and because developing a syllabus also requires more than a few days or weeks of work. Next fall, I am re-teaching a first year course Introduction to Human Communication and Culture as well as a new course on computer hacking. While I have an old version of the syllabus, I am going to spend the next few weeks revising, reshuffling, and updating. If there is anything you think should be included in this type of course drop me a line. Below is the new description of the course.
    The Culture and Politics of Computer Hacking. This course takes as its object computer hackers to interrogate not only the ethics and practices of hacking, but to examine more broadly how hackers and hacking have transformed the politics of computing and the Internet more generally. We will examine how hacker values are realized and constituted by different legal, technical, and ethical activities of computer hacking for example, free software production, cyberactivism and hactivism, cryptography, and the pranksih games of hacker underground. We will pay close attention to how ethical principles are variably represented and thought of by hackers, journalists, and academics and will use the example of hacking to address various topics on law, order, and politics on the Internet such as: free speech and censorship, computer gaming, privacy and security, and intellectual property. This will allow us to critically 1) problematize thinking on computer hackers as a socio-cultural group guided by a singular ethic and set of practices 2) examine the multiple ways hackers draw on and reconfigure dominant ideas of property, freedom, and privacy through their diverse moral codes and technical activities 3) broaden our understanding of politics of the Internet by evaluating the various political effects and ramifications of hacking.

    MJ Ray: Better Free Software Organisations?

    Another zero-day announcement of a Software in the Public Interest online meeting has been posted. The announcement mentions "one motion has been raised" but the agenda doesn't include any motions, so I'm not sure what. Also, the agenda lists "Debian logo licence" as up for discussion but I thought last month's meeting resolved that. I watch SPI fairly closely and I'm confused. How about the rest of you? SPI isn't alone in this: many software organisations seem to suffer from similar problems. For example, I ranted in passing about Free Software Manchester yesterday, who just posted the notes from their own zero-day meeting and I've suggested possible ways of reforming debian's currently-stalled New Maintainer process more than once. Are free software users particularly bad at the basics of running an interest society (like welcoming and expiring members, calling meetings, publishing routine communications, and so on), have I been spoiled by cooperatives with their friendly Member Services departments or secretariats, or what? Is this why so many free software orgs seem to include self-perpetuating leadership groups? Is this a serious problem if, as reported, Software Development is a Team Sport [etbe]? Are there fully-working free software mass participation groups out there? I feel a lot of these problems are caused by attempting to order our inherently entropy-filled world completely and insisting everything follows petty rules, such as refusing to answer a question because the "wrong" member asked it. The world will not become less random just because hackers try to impose arbitrary rules. Sometimes it's good to put down minimum standards (because calling zero-day meetings is a mostly-avoidable way of excluding some members) but it will always be a poor alternative to trying to do the best you can for others. How do we get past this? My pro-cooperation-and-better-business platform for SPI board went pretty badly and I've had some anti-cooperative flames back from someone starting another free software group this month, so I don't think I can fix these existing organisations any time soon. About 1 in 6 people in the UK are members of a cooperative, so even if that is reflected among hackers (and I think it's lower), all of them would not be enough to reform much. One of the most common memes in free software is "show us the code" and the few other free software cooperatives I've seen have mostly failed, with a few surviving but hitting a size limit. As a result, I'm currently negotiating the start of a new free software cooperative. So far, I'm really happy with how that's going. Many cooperators learn at the feet of large consumer cooperatives like the Cooperative Group who run training courses for new members about putting cooperative values and principles into practice, which we smaller groups couldn't run ourselves. As a result, most cooperators already know how to work well together. Should large software societies like SPI try commissioning similar courses?

    15 April 2008

    MJ Ray: Talk with People who want to Discuss

    I spend too much of my time trying to talk with people who don't want to discuss, yet somehow I won't stop. I keep hoping that things like suggesting good advice on meeting scheduling will avoid them repeating old mistakes. The most extreme life-and-death example is probably trying to help with Kewstoke Toll Road, where people still speed and someone crashed off again last night (although I don't know what caused last night's crash - could have been a simple accident). Of course, it's better to talk with people who have asked questions and want to hear the answer. I'm currently involved in several groups like that and it makes me much happier [4HWW]. I've even made a confidentiality agreement [Network Blogging article] about one group because I really like the organiser and want to help them, but I've yet to see changes happen because of it and that's probably about all I can write here, which does rather suck. But talking of changes that make me happy, I spotted that Bristol Wireless has now gone further than TTLLP by deciding to change people away from Microsoft Windows when they find it:-
    "all Windows stuff must be gone from the premises [...] no longer help do callers favours with broken Windows machines, apart from fixing them properly and permanently by installing Debian"
    at their March meeting. Well done, BW! After a request, I finally put four photos from Social Source South West (which was hosted by BW) online, which reminded me to subscribe to watfordgap's travels. Disappointingly, on my first read, it promotes the Suppliers Directory developed by Lasa. That directory is a big problem because it creates a silly barrier to entry which hinders new social enterprises and cooperatives. At a time where most non-profit software is unsustainable and needs to change, requiring three referees is a way to obstruct change. Also, persuading three people to support their work is no substitute for supplier evaluation. Any non-profits who want to lead their sector should approach ICT suppliers directly. The article also mentions Experts Online which is even more short-sighted about computing: "both PC and Mac" indeed! What about GNU/Linux, thin clients, and other changes which are already making a big difference to some non-profits...? But here I go again, talking to a brick wall. I expressed these concerns when that Directory started and it didn't do any good then, so I doubt they'll change it now, near the end of its life. So I'm going to move on. There are lots of people emailing who want to hear from me, so it's time to concentrate on talking with people who do want to listen. If you want to discuss this with me, visit my website for the comments form (click the title or look for a "view original post" link, depending what site you're reading).

    Russell Coker: Buying Old PCs

    I install quite a number of internet gateway machines for one of my clients. While eventually he will probably move to using an ASUS EeePC [1] or something similar, the current plan is to keep using desktop PCs (unfortunately server-class machines make too much noise). P4 machines use an unreasonable amount of power and don’t seem worth getting second-hand. P3 machines are the most power efficient machines that have been commodity PCs (with keyboard and monitor), see my computer power use document for an example [2]. So what I would like is to get a dozen name-brand P3 systems at a reasonable price. Currently the only companies I can find selling such machines are charging ridiculous prices (such as $300) with the aim of reaming corporations who want to complete a set of old machines rather than upgrading them all. Most online auctions etc are selling P4 and Celeron 2GHz as the minimum hardware. The Celeron machines are better than P4 (at around 50W) but not as good as P3 (at less than 40W). If you read this post and are in Melbourne, Australia and have a name-brand P3 desktop system that you want to sell then let me know. If you have any suggestions of how to buy such machines then I’m interested in advice.

    Stefano Zacchiroli: not my president

    he is not my president I hereby declare that he is not my president, no matter of what (country, largest television group, football team, ...). While I'm at it, I also declare myself SABDPTIC (Self Appointed Benevolent Dictator Pro Tempore of the (Debian) Italian Cabal). Empowered by this charge I also declare that he is not the president of the various members of the (Debian) Italian Cabal either. I will not accept any counter statement, as I'm now the dictator. Now that I've declared this, I can happily resign from my role of dictator. Thanks for all the support messages I've received today by fellows around the world. Saddened. PS (ikiwiki permitting) the hyperlinks to my new non-president in this post are a good example of the usage of the rel="nofollow" microformat.

    Next.

    Previous.