Search Results: "spa"

26 October 2007

Clint Adams: zrockboxing out

Someone gave me an MP3 player. He did so for somewhat devious reasons, but that's outside the scope of this post. It is a SanDisk Sansa stop thinking of pulp fantasy and it has roughly 500 gigs too little storage space. The FAQ says such cute things as The original firmware is recommended for charging at this time, and Rockbox does not currently provide either functionality so you will need to continue using the original firmware (for now) in MSC (UMS) mode to add music to the Sansa. Please don't tell my Sansa about this FAQ because it thinks that both of those claims are false. I want my newfangled device to participate in the spyware fiesta known as last.fm, and Rockbox has an option for supporting that; if you turn it on, it fills a /.scrobbler.log file with tab-delimited lines for your parsing pleasure. Rather than download additional software to cope with my new lifestyle, I whipped up the following script (released under the gnocchi-ng license) which converts the log to something you can just stuff into your ~/.zomg/cache file and send up to the submission server at your next appropriate invocation of zomg. I'm afraid you'll have to handle any timestamp sorting by hand, but if you script it, be a dear and implement some kind of cache file locking. 
#!/bin/zsh
# Copyright (C) 2007  Clint Adams.  All rights reserved.
# This program has no name and is released under the terms of
# the gnocchi-ng license.
audioscrobbler_urlencode()  
  if (( $+options[multibyte] )); then
    setopt localoptions extendedglob nomultibyte
  else
    setopt localoptions extendedglob
  fi
  input=( $ (s::)1  )
  print -- $ (j::)input/(#b)([^A-Za-z0-9_.!~*\'\(\)-])/%$(([##16]#match)) 
 
audioscrobbler_constructquery()  
  local sid="$1"
  local artist=$(audioscrobbler_urlencode "$2")
  local track=$(audioscrobbler_urlencode "$3")
  local album=$(audioscrobbler_urlencode "$4")
  local mbid=$(audioscrobbler_urlencode "$5")
  local length=$(audioscrobbler_urlencode "$6")
  local ttime=$(audioscrobbler_urlencode "$7")
  local source="$8"
  local tracknum="$9"
  reply=("&s=$ sid " "&a[0]=$ artist &t[0]=$ track &i[0]=$ ttime &o[0]=$ source &r[0]=&l[0]=$ length &b[0]=$ album &n[0]=$ tracknum &m[0]=$ mbid "$'\0')
 
while read -r line
do
  local -a field
  field=($ (ps:\t:)"$ line " )
  if [[ $field[6] == L ]]; then
    audioscrobbler_constructquery "" "$field[1]" "$field[3]" "$field[2]" "" "$field[5]" "$field[7]" P "$field[4]"
    print "$reply[2]"
  fi
done <<(grep -v '^#' .scrobbler.log)

25 October 2007

Enrico Zini: axi-query-similar

apt-xapian-index: searching for similar packages I've recently posted: Today I'll show how to abuse Xapian to show a list of packages similar to a given one. This time I'll try just linking to the code in wsvn and showing in the blog only show the most important bits. So, we have a package name, and we want to show what are the packages similar to that one. To do it, we simply build a big OR query with all the terms indexed for that package: Xapian will show us the packages whose terms are most similar, and that does the trick. This works because Xapian gives us the best results first, therefore even if no package except the given one will give an exact match, we still get the nearest matches first. In order to get the list of indexed terms given a package name we need to do two things:
  1. Get the Xapian document for the package.
  2. Get the termlist of the document.
To get the Xapian document we search for a term that only that document can have. In the index, the package name is indexed with the special prefix "XP", so we can search for that: 
def docForPackage(pkgname):
    "Get the document corresponding to the package with the given name"
    # Query the term with the package name
    query = xapian.Query("XP"+pkgname)
    enquire = xapian.Enquire(db)
    enquire.set_query(query)
    # Get the top result only
    matches = enquire.get_mset(0, 1)
    if matches.size() == 0:
        return None
    else:
        m = matches[0]
        return m[xapian.MSET_DOCUMENT]
Then we build the big term list, by iterating the termlist of the document: 
# Build a term list with all the terms in the given packages
terms = []
# Get the document corresponding to the package name
doc = docForPackage(pkgname)
if not doc: continue
# Retrieve all the terms in the document
for t in doc.termlist():
    if len(t.term) < 2 or t.term[:2] != 'XP':
        terms.append(t.term)
Note that it's trivial to fetch terms from more than one document, if you want to query "all packages a bit like this one and a bit like that one", although that's less of a useful feature. Lastly, we build the final query: 
# Build the big OR query
query = xapian.Query(xapian.Query.OP_AND_NOT,
            # Terms we want
            xapian.Query(xapian.Query.OP_OR, terms),
            # AND NOT the input packages
            xapian.Query("XP"+pkgname))
I add an AND_NOT part with the input package name so that we don't get in the output the package that we asked for. This is it: 
$ ./axi-query-similar.py debtags
20309 results found.
Results 1-20:
33% debtags-edit - GUI browser and editor for Debian Package Tags
27% tagcolledit - GUI editor for tagged collections
25% libtagcoll2-dev - Functions used to manipulate tagged collections (development version)
24% tagcoll - Commandline tool to perform operations on tagged collections
19% packagesearch - GUI for searching packages and viewing package information
18% doodle - Desktop Search Engine (client)
18% doodled - Desktop Search Engine (daemon)
18% libept0 - High-level library for managing Debian package information
18% upgrade-system - system upgrader from Konflux
18% libept-dev - High-level library for managing Debian package information
17% ept-cache - Commandline tool to search the package archive
16% tracker-utils - metadata database, indexer and search tool - commandline tools
[...]
You can use the wsvn interface to get to the full source code and the module it uses. Next in the series: adaptive quality cutoff.

Matthew Garrett

Linux-hero wrote about how Ubuntu kills your hard drive. The situation is somewhat less clear than you might think from the article, but the basic takeaway message is that Ubuntu doesn't touch your hard drive power management settings by default. In almost all cases, it's more likely to be your BIOS or the firmware on your hard drive.

The script that's executed when you plug or unplug your laptop is /etc/acpi/power.sh. The relevant sections are:
function laptop_mode_enable  
...
    $HDPARM -S $SPINDOWN_TIME /dev/$drive 2>/dev/null
    $HDPARM -B 1 /dev/$drive 2>/dev/null
That is, when the laptop_mode_enable function is called, we set the drive power parameters. Now, by default laptop_mode_enable isn't called:
if [ x$ENABLE_LAPTOP_MODE = xtrue ]; then
    (sleep 5 && laptop_mode_enable)&
fi
because ENABLE_LAPTOP_MODE is false in the default install (check /etc/default/acpi-support). This means that, by default, we do not alter the hard drive power settings. In other words, the APM settings that your drive is using in Ubuntu are the ones that your BIOS programmed into it when the computer started. This is supported by the fact that people see this issue after resuming from suspend. We don't touch the hard drive settings at that point, so the only way it can occur is if your BIOS or drive default to this behaviour.

If you enable laptop mode, then we will enable aggressive power management on the drive and that may lead to some reduction in hard drive lifespan. That's a fairly inevitable consequence of laptop mode, since it only makes sense if the laptop enages in aggressive power management. But, as I said, that's not the default behaviour of Ubuntu.

There's certainly an argument that we should work around BIOSes, but in general our assumption has been that your hardware manufacturer has a better idea what your computer is capable of than we do. If a laptop manufacturer configures your drive to save power at the cost of life expectancy, then that's probably something you should ask your laptop manufacturer about.

24 October 2007

Enrico Zini: axi-query-expand

apt-xapian-index: suggesting new terms for improving the query I've recently posted an introduction of apt-xapian-index, an example of how to query it and a way to add simple result filters to the query. Today I'll show how to use a very interesting feature of Xapian: computing a list of the best terms to use to improve the query. I'll expand apt-query-pkgtype.py to write, after the results, the suggested terms. If you feel like reimplementing my examples in another language, let me know and I'll include it to the post. Let's start with a few definitions taken from the Xapian glossary:
RSet (Relevance Set) The Relevance Set (RSet) is the set of documents which have been marked by the user as relevant. They can be used to suggest terms that the user may want to add to the query (these terms form an ESet), and also to adjust term weights to reorder query results. ESet (Expand Set) The Expand Set (ESet) is a ranked list of terms that could be used to expand the original query. These terms are those which are statistically good differentiators between relevant and non-relevant documents.
What I'm doing now is: after printing the normal result, I build an rset with all the results. For better results one can ask the user to pick the results they like (make a GUI with that: it should rock!), but starting with the top results in the RSet is a good enough default: 
# Now, we ask Xapian what are the terms in the index that are most relevant to
# this search.  This can be used to suggest to the user the most useful ways of
# refining the search.
# Select the first 10 documents as the key ones to use to compute relevant
# terms (matches is the mset returned by a normal query)
rset = xapian.RSet()
for m in matches:
    rset.add_document(m[xapian.MSET_DID])
Then I use the RSet to compute the ESet, and display the results to the user: 
# This is the "Expansion set" for the search: the 10 most relevant terms
eset = enquire.get_eset(10, rset)
# Print it out.  Note that some terms have a prefix from the database: can we
# filter them out?  Indeed: Xapian allow to give a filter to get_eset.
# Read on...
print
print "Terms that could improve the search:",
print ", ".join(["%s (%.2f%%)" % (res.term, res.weight) for res in eset])
You can also abuse this feature to show what are the tags that are most related to the search results. This allows you to turn a search based on keywords to a search based on semantic attributes, which would be an absolutely stunning feature in a GUI. We can do it thanks to Xapian allowing to specify a filter for the output of get_eset. This filter filters out all the keywords that are not tags, or that were in the list of query terms: 
class Filter(xapian.ExpandDecider):
    def __call__(self, term):
        "Return true if we want the term, else false"
        return term[:2] == "XT"
I just rerun get_eset like above, but adding the filter, to show only a suggestion of tags: 
# This is the "Expansion set" for the search: the 10 most relevant terms that
# match the filter
eset = enquire.get_eset(10, rset, Filter())
# Print out the resulting tags
print
print "Tags that could improve the search:",
print ", ".join(["%s (%.2f%%)" % (res.term[2:], res.weight) for res in eset])
sys.exit(0)
This is it. You can use the wsvn interface to get to the full source code and the module it uses. Try running it as: 
axi-query-expand.py --type=gui image editor
axi-query-expand.py --type=cmdline image editor
axi-query-expand.py --type=game image editor
you will see that as the results change, the suggestions change as well. I'm happy to see that Xapian often suggests tags: this means that tags are useful discriminators in search results, and that all the work done on Debtags has been a good work. Notice that the suggestions change as you change any of the terms, the custom filter, and the tags you put in the search. Try to imagine how such a thing could improve Synaptic. What if I told you that Xapian is fast enough that all of this can happen live while you type? Next in the series: searching for similar packages.

Russell Coker: Blog Ethics

Reporters Sans Frontiers (AKA RSF AKA Reporters Without Borders) has an interesting document about blogging [1]. They are specifically focussed on blogging as a way of reporting news. Their definition of a blog states that it is “a personal website” (there are many corporate blogs run by teams) and that it contains “mostly news” (most blogs that I read contain mostly technial information about computers and any “news” is mostly about the author). They also describe a blog as being set up with an “interactive tool” - most blogs are written using web-based tools, but some are written with plain text and a compilation system. Some of their technical information is simply wrong. For example they say that RSS “alerts users whenever their favourite blogs are updated” (this can be done through RPC notification which then requires RSS to pull the data - but a user will almost always have a tool polling their favourite RSS feeds). Their section listing the various options for blogging platforms mentions LiveJournal, Blogger, and MSNSpaces but doesn’t mention Wordpress.com which seems to be a serious omission (although it does mention civiblog.org which is a useful hosting resource for grassroots political campaigning). But these are minor problems, they are reporters primarily not programmers. Their document gets interesting when it gives links to pages about blogging ethics. One page that was not directly linked (presumably because it mainly concerns non-journalistic blogging) is Rebecca Blood’s document about blogging ethics [2]. She gives a brief coverage to conflicts of interest and gives most space to the topic of maintaining a record of changes. One thing I have been considering is having a separate instance of Wordpress for documents that change [3]. This way regular blog posts (such as this one) can be relied on to preserve a list of changes (any change other than correcting a typo will be documented) so if you link to a post you can rely on the original intent being preserved. But for posts which have ongoing value as live documents they will be kept current without a significant change-log. Items that go into the main blog would include commentary on news and predictions, and the document blog would contain technical information about Linux, the science-fiction stories that I will eventually write, etc. When I wrote my previous blog post about this issue I was mainly considering technical issues, but when considering the ethical issues it becomes clear that I need a separate blog (or other CMS) for such things. The site etbe.coker.com.au needs to be a reliable record of what I wrote while another site such as docs.coker.com.au could contain live documents with no such record. The essential factor here is the ability of the user to know what type of document they are seeing. Another really interesting point that Rebecca makes is in regard to known bad sources. Sometimes a known bad source can produce worthy data and is worth referencing, but you have to note the context (which among other things means that if the worthy data gets any reasonable number of hits it may get replaced by something entirely different). If a blogger cites a reference on a known bad site and explains the reason for the reference (and the fact that the nature of the site is known) then the reader reaction will be quite different to just a reference with no explanation. As a contrast to Rebecca Blood’s document the cyberjournalist.net code of ethics [4] covers the traditional issues of journalistic integrity, “public interest” etc. One interesting issue that they raise (but apparently don’t address) is the definition of “private people“. While it is generally agreed that movie stars and similar people deserve less protection as they have chosen to be in the public eye the situation regarding bloggers is less clear. Is a blogger with a high ranking on technorati.com in a similar position to a Hollywood star when it comes to privacy? A claim is sometimes made that blogs are unreliable because they can change (based on the actions of some bloggers who change posts with no notification and turn sections of the blogsphere into a swamp). Magazines and newspapers are held as an example of unchanging content and people who publish online magazines sometimes express horror at the idea of changing a past issue. The fact is that newspapers and magazines never changed past issues because it is impossible to recall hundreds of thousands of printed documents that are distributed around a country or the world. When it’s possible to fix old issues (as it is online) then the requirement is to document the changes not keep bad versions. It probably would be a useful feature for a blogging package to have the ability to display (and link to) old versions of a page. This would allow the users to easily see the changes and anyone who references a post can reference a particular version of it. In some situations it may make sense to use a Wiki server instead of a Blog server for some data to preserve the change history. Maybe I should consider a wiki in which only I have write access for my documents repository. No-one seems to cover the issue of how to deal with fiction. Obviously for a blog such as 365tomorrows.com (a science-fiction blog) it’s assumed that all content is fictional unless stated otherwise. The problem comes when someone includes fiction amongst the non-fiction content of a blog. I recently unsubscribed from a blog feed when the author revealed that what appeared to have been a diary entry was actually a fiction story (and a poor one at that). If your blog has mostly non-fiction then any fiction must be clearly marked, probably including the word “fiction” in the title and the permalink would be required to avoid all potential of confusion. For my documents CMS/blog I am considering having a category of “fiction” and to use the category name in the permalink address of each post. Of course science-fiction is sometimes obvious as fiction, but to avoid problems with fiction that is set in current times I think it’s best to clearly mark all such posts. The old-fashioned media seems to have dealt with this, tabloid womens magazines (which almost everyone reads occasionally as they are often the only reading materiel in a waiting room) traditionally have a fiction story in every issue which is clearly marked. Another type of blogging is corporate blogging. I wonder whether that needs to be covered separately in an ethics code. One thing that the three documents about ethics have in common (as a side-note in each) is the fact that the personal reputation of the writer depends on their ethics. If you are known for writing truthfully and fairly then people will treat your posts more seriously than that of writers who lie or act unfairly. There are direct personal benefits in acting ethically. The RSF document claims that the sole purpose of ethics standards is to instill trust in the readership which directly benefits the writer - not for any abstract reasons. Whenever ethics is mentioned in terms of writing there is always someone who claims that it can’t be enforced, well many people in your audience will figure out whether you are acting ethically and decide whether they want to continue reading your materiel. I wonder whether Planet installations should have ethics codes and remove blogs that violate them. In conclusion I think that a complete code of ethics for blogging needs to have some IF clauses to cover the different types of blog. I may have to write my own. Please let me know if you have any suggestions.

Share This

23 October 2007

Wouter Verhelst: Automounters

You know the problem? Something happens, and all your systems go down—scheduled or otherwise. When you boot them up, one box is faster than the other; and as a result, some network mounts don't really work correctly. While this doesn't really happen at work, I've seen it after a power failure at home. I thought this was a feature that Windows had but Linux doesn't. I really should've known better... 
wouter@country:~$ mount
/dev/hda5 on / type ext3 (rw,errors=remount-ro,commit=0)
tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
(...) 
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,noexec,nosuid,nodev)
automount(pid15173) on /autofs type autofs (rw,fd=5,pgrp=15173,minproto=2,maxproto=4)
wouter@country:~$ ls /mnt/stuff
(lots of private data snipped...) 
wouter@country:~$ mount
/dev/hda5 on / type ext3 (rw,errors=remount-ro,commit=0)
tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
(...) 
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,noexec,nosuid,nodev)
automount(pid15173) on /autofs type autofs (rw,fd=5,pgrp=15173,minproto=2,maxproto=4)
goa:/stuff on /autofs/stuff type nfs (rw,addr=192.168.119.23)
And after I wait a minute, that last line is gone. Automounters are cool. And not too hard to set up, either, if only a bit confusing at first. My /etc/auto.master contains the following (not counting commented-out examples): 
/autofs	/etc/auto.autofs --timeout=60
I then need to have a directory /autofs, on which the automounter mounts a special filesystem if it's started; this is the "automount" in the above mount output. The autofs needs a second automounter map, which looks like this in my case: 
cdrom	-fstype=iso9660,ro,nosuid,nodev	:/dev/cdrom
stuff	-fstype=nfs			goa:/stuff
camerab	-fstype=vfat,ro,umask=133,dmask=22 :/dev/disk/by-id/usb-GENERIC_USB_Storage-SDC_200606191719424ND-0:1-part1
floppy	-fstype=auto			:/dev/disk/by-id/usb-Y-E_DATA_USB_Floppy_Drive
Occasionally, yes, from the above lines it can be inferred that this indeed also works for my USB floppy drive, my multicardreader, and my CD-ROM drive. Finally, I need a symlink into the automount filesystem, like so: 
wouter@country:~$ ls -l /mnt/stuff
lrwxrwxrwx 1 root root 13 2007-10-23 19:21 /mnt/stuff -> /autofs/stuff
et voila: if I now try to access anything below /autofs/stuff, the automounter kicks in, transparently mounts the filesystem, and without any further delay or ado, I can get at my files. Well—unless the server is down, that is.

MJ Ray: the Cooperative Group Half-Yearly Meeting: Q+A

the co-operative half-yearly meeting, Bristol Belatedly, as promised, here are the questions from the floor at last week's half-year meeting of the Co-operative Group Kennet and Avon area. All errors and omissions are mine, of course.
How do we defend against demutualisation and private equity attacks?
Our rules are now written so that demutualisation would require assets to be transferred to other co-operatives. Please comment on other ways to the current constitutional review consultation.
How can ordinary members get informed and involved in running the group?
Get items into network, call the regional office.
The consitutional review should give more power to meetings and make it easier for motions and binding promises.
will forward this to the review.
Why did the group secretary suddenly leave?
The job had changed post-merger, with different responsibilities.
How can we stop the payments to voluntarily-retiring directors?
The area has made its point. The chief executive of CooperativesUK will look into this and there was a congress motion to look into this issue.
There's been no mortgage advisor in one local bank branch for months. Is this indicative of the bank's attitude to mortgages?
No, it's disappointing. Please send details to the area secretary.
Please can we use geographic numbers instead of the expensive 0845 for the membership enquiries?
Will forward and ask for a geographic number.
Is the constitutional review being rushed?
It's streamlined because the merger gives a pressing need, but we want member input.
Can we or do we promote credit unions in-store?
Yes, we can and do promote them to members already. Can set aside some space in-store if needed.
Does the co-operative travel insurance discriminate against over 75s?
will forward the answer when we have it.
The top management post-merger are all from United: is this a take-over?
no, 2 executives had clear intentions to retire already and the lower levels are more mixed.
Which? gave the co-operative food a D for organic produce: why and what will we do about it?
Report from today's Guardian not yet seen. Expect to respond in next few days. The report may have compared convenience stores with larger stores.
Will ex-United travel agents become the co-operative travel stores?
Yes. There's a big big change coming, to be more of a community travel agent, restructured, with more phone and internet options.
Is the energy-saving policy to include refrigeration, lighting and IT?
This is a top goal, to improve refrigeration and lighting control. It saves the earth and saves money. Nearly all electricity is already "greener" electricity.
Do we own the freehold on the land previously occupied by a store at Shirehampton?
Yes. Only a non-food retailer will be allowed in.
Why do elected representatives have to retire at 70 if still capable?
Please submit comments on it to the constitutional review?
Why was there no transport from Weston-super-Mare?
Will check the office records for requests.
Anyone see these themes appear at other meetings? Any comments on these themes?

Enrico Zini: axi-query-pkgtype

apt-xapian-index: adding simple result filters to the query I've recently posted an introduction of apt-xapian-index and an example of how to query it. Today I'll show how to add simple Debtags-based result filters to a query. I'll expand axi-query-simple.py with a --type comman line option that allows to filter for package type. As a starter I'll implement filtering for "game", "gui", "cmdline" or "editor". If you feel like reimplementing my examples in another language, let me know and I'll include it to the post. The filters are implemented using Xapian queries. Xapian allows to build queries as well as combine existing queries using a large amount of operators. This allows us to create a little database of ready made queries: 
# This is our little database of simple Debtags filters we provide: the name
# entered by the user in "--type" maps to a piece of Xapian query
filterdb = dict(
    # We can do simple AND queries...
    game = xapian.Query(xapian.Query.OP_AND, ('XTuse::gameplaying', 'XTrole::program')),
    # Or we can do complicate binary expressions...
    gui = xapian.Query(xapian.Query.OP_AND, xapian.Query('XTrole::program'),
        xapian.Query(xapian.Query.OP_OR, 'XTinterface::x11', 'XTinterface::3d')),
    cmdline = xapian.Query(xapian.Query.OP_AND, 'XTrole::program', 'XTinterface::commandline'),
    editor = xapian.Query(xapian.Query.OP_AND, 'XTrole::program', 'XTuse::editing'))
    # Feel free to invent more
It's now trivial to lookup a query and AND it to the main query: 
# Build the query by ORing together various terms
query = xapian.Query(xapian.Query.OP_OR, termlist)
# If a filter was requested, AND it with the query
if extrafilter:
    query = xapian.Query(xapian.Query.OP_AND, query, filterdb[extrafilter])
That's it. Simple, trivial feature to add to a search, but very useful and simple to use. You can use the wsvn interface to get to the full source code and the module it uses. Try running it as: 
axi-query-pkgtype.py --type=gui image editor
axi-query-pkgtype.py --type=cmdline image editor
axi-query-pkgtype.py --type=game image editor
you will see that the results change radically. What are the games that match "image editor", you would ask? 
74% xjig - An X11 jigsaw puzzle
69% wmpuzzle - WindowMaker dock app 4x4 puzzle
66% xflip - programs to mirror-image or melt your display
Jigsaws, of course :) The code is now colourised thanks to Simon Ward who pointed me at the syntax plugin for ikiwiki. Next in the series: suggesting new terms for improving the query.

Raphael Geissert: Nice badvista photo

I just came across this photo (which, by the way, is in public domain):



P.D. sorry about the size, but bigger looks better :)

22 October 2007

Holger Levsen: Small report from the meeting in Extremadura

The last five days I've spent at the combined Debian Edu, FAI and CDD meeting in M rida in Extremadura, together with 17 others. The Debian Edu / Skolelinux crowd was the biggest, followed by FAI, followed by CDD people, though many were part of more than one project. "As usual" it was great to meet many known faces and some unknown ones and to be able to discuss stuff in RL.

Besides various small different things, I worked on the Debian Edu etch r1 release, our lenny roadmap (see the bottom of our roadmap page which also covers etch r1) and the (technical side of the) german translation of the Debian Edu documentation. Non edu related the new setupharddisk2 for FAI took most of my brain activity. And I've signed all keys allready :-)

We stayed at a very nice hotel in the even much nicer city of Merida, which features a lot of roman ruins and history. For example, we walked over the biggest roman bridge in the world! The only thing that was bigger was the amount of good food we were forced to eat every day... Incredible. This was my second visit to Merida and this time I finally managed to visit the amphitheatre (with seats for 15000 people) and the Roman theatre. (Unfortunatly the english page about Emerita Augusta is a stub.) If you happen to come around that area make sure to spend some time sight seeing, it is really worth it! People living in Merida literally fear to rebuild their houses, because as soon as they start to dig, they find something historical and have to turn their property into a museum :-)

In case you have not seen it yet, you should watch this small movie from euro news about the free software efforts in Extremadura. Insightful and very good promotion, also to show around.

Many thanks to C sar, Jos Luis and the Junta de Extremdura for hosting such a great meeting! You rock!

On a unrelated note, spam has gone crazy yesterday while traveling back, 15000 mails were not catched by my so far well working setup. ARG. I still used to have a catchall setting, which was useful to give out addresses like $company@mydomain, but now I will explicitly list those addresses and disable catchall. And luckily for you I will not blog about every spam mail I got... :) Generally I dont consider spam (the source of) evil, it's money: as long as money rules the world, there will be spam. So since it's a social problem, it is hardly solvable by technology. (That is, as long as you want to receive mails from strangers, which I do.) IMO what's worse than spammers are autoresponders to spam. Most of those 15000 mails were auto-replies to spam. OMFSM.

Enrico Zini: axi-simplequery

apt-xapian-index: performing a simple query I've recently posted an introduction of apt-xapian-index. Today I'll show how to make simple queries to apt-xapian-index. If you feel like reimplementing my examples in another language, let me know and I'll include it to the post. What I'm going to build is a replacement for apt-cache search that: This is just a beginning: in future blog posts I'll show how to enhance a search with interesting advanced features. First thing, we need to import the Xapian module, found in the package python-xapian. Documentation on the Python Xapian API can be found in /usr/share/doc/python-xapian. 
import xapian
Then we open the apt-xapian-index database: 
# Instantiate a xapian.Database object for read only access to the index
db = xapian.Database("/var/lib/apt-xapian-index/index")
Now we build a query from the command line arguments. We'll assume that if an argument is in the form foo::bar, then it's a Debtags tag instead of a normal keyword. For normal keywords, we also search for the stemmed version, so that we can, for example, find "editing" when the user searches for "edit". 
# Stemmer function to generate stemmed search keywords
stemmer = xapian.Stem("english")
# Build the terms that will go in the query
terms = []
for word in args:
    if word.islower() and word.find("::") != -1:
        # According to /var/lib/apt-xapian-index/README, Debtags tags are
        # indexed with the 'XT' prefix.
        terms.append("XT"+word)
    else:
        # If it is not a Debtags tag, then we consider it a normal keyword.
        word = word.lower()  # The index stores keyword all in lowercase
        terms.append(word)
        # If the word has a stemmed version, add it to the query.
        # /var/lib/apt-xapian-index/README tells us that stemmed terms have a
        # 'Z' prefix.
        stem = stemmer(word)
        if stem != word:
            terms.append("Z"+stem)
Now we have the terms for the query, and we can create a query that ORs them together. One may ask, why OR and not AND? The reason is that, contrarily to apt-cache, Xapian scores results according to how well they matched. Matches that match all the terms will score higher than the others, so if we build an OR query what we really have is an AND query that gracefully degenerates to closer matches when they run out of perfect results. This allows stemmed searches to work nicely: if you look for 'editing', then the query will be 'editing OR Zedit'. Packages with the word 'editing' will match both and score higher, and packages with the word 'edited' will still match 'Zedit' and get included in the results. 
# OR the terms together into a Xapian query.
#
query = xapian.Query(xapian.Query.OP_OR, terms)
We then run the query. Queries are run through a xapian.Enquire object: 
# Perform the query
enquire = xapian.Enquire(db)
enquire.set_query(query)
The Enquire object returns results as an mset. An mset represents a view of the result set, and can be iterated to access the resulting documents. Here we iterate the mset and output the result of the query, looking up short descriptions with apt: 
# Display the top 20 results, sorted by how well they match
matches = enquire.get_mset(0, 20)
print "%i results found." % matches.get_matches_estimated()
print "Results 1-%i:" % matches.size()
for m in matches:
    # /var/lib/apt-xapian-index/README tells us that the Xapian document data
    # is the package name.
    name = m[xapian.MSET_DOCUMENT].get_data()
    # Get the package record out of the Apt cache, so we can retrieve the short
    # description
    pkg = cache[name]
    # Print the match, together with the short description
    print "%i%% %s - %s" % (m[xapian.MSET_PERCENT], name, pkg.summary)
This is it. You can use the wsvn interface to get to the full source code. You can run the code passing keywords and Debtags tags. Try running it as: 
    ./axi-query-simple.py role::program image edit
    ./axi-query-simple.py role::program game::arcade
    ./axi-query-simple.py kernel image
You can search Debtags tags using debtags tagsearch. In a later blog post, I'll show how to use apt-xapian-index to implement a better-than-you-would-ever-have-thought-possible tag search. Next in the series: adding simple result filters to the query.

Gerfried Fuchs: Texta - Hediwari

Alright, a bit late. But it was hell of a concert, really enjoyed it. There were two support bands: First there was Hinterland with their new album "Zwa Seiten" and then there was Die Antwort, both of them quite interesting. Finally Texta came onto the stage and gave a great concert. You might want to take a listen to the two free mp3s they have linked from their LastFM profile. The version of Hediwari isn't the best but should give you and idea about their style—austrian hip hop at its best. As added benefit you might want to watch So könnt's gehen on youtube. The interesting part of it, though propably noone cares, the guy from the first band (shy) without an instrument is a working collegue of mine. ;)

21 October 2007

Alastair McKinstry: Whither Darwin/ Terrestrial Planet Finder ?

I see from the BBC and /. that ESA has made its choices in Cosmic Vision 2015-2025. It appears that Darwin did not make the cut. Istead, PLATO and SPICA seem to be holding the banner for exoplanetary exploration. Can someone closer to the decision-making process comment? What does this mean for NASA's Terrestrial Planet Finder? There were considerations of merging Darwin and TPF; does the project live as TPF, or was the task of imaging an exoplanet left to JWST ? Tags , ,

20 October 2007

Jordi Mallach: You might get an email from me tonight

Sometime in August, I said I would watch the Inbox Zero talk later on that day. Well, I finally did today. And I'm ready to mass-murder my (now not so) fat inbox folder and start from scratch, and becoming a good boy. In fact, I've been on probation for a few weeks. While I wasn't watching the talk (which is pretty insightful and fun, and useful if you also have these horrid mail handling problems) I did roll up my sleeves a few times and worked on reducing the problem. After a few rounds of fighting, things were looking slightly better. I deleted TONS of spam which still was sitting in there. I deleted entire threads of list mail which for some reason wasn't being filtered properly. I archived a lot of random, misc email. I even replied to some job offers, for a change. I fixed my .procmailrc a little to get rid of lots of useless stuff that appears in my mail. It got better, but not entirely better. I went from the 6600~ which was probably the figure when I said Enough! to around 2580. It's still a lot, and I can still get rid of a lot more with easy pattern searches in mutt. The good news is that, for the first time in ages, the number of emails in the mailbox has stayed stable for more than a month. I tell you: I'm proud! So Merlin gets asked in the talk what to do when you've been a naughty boy for a long time, and you've ended up with this HUGE mailbox you can't handle anymore. His answer was what some people suggested in blog comments: put it aside, start from zero. Merlin calls it mail-DMZ, and that's probably what I'll do in a few hours, admittedly with a sentiment of guilt deep in my chest. And from that point, I'll have my mailbox be a TODO list. Delete. Defer. Delegate. Respond. Do. Simple! Other Planet Debian participants like joeyh commented that something that really helps is reducing the number of times you poll for email. For me, that means 
set daemon      1800            # Pool every 30 minutes
when it was 5 minutes before. I hope I won't find myself issuing awaken commands often... I remember when, more than five years ago, having more than 100 mails made me feel bad and go cleanup. After some vacation, it went up to 150. Then Christmas came along, 300, until I found myself nearing 7000 last summer. Before moving my junk to a demilitarised mailbox, I'm having some fun replying to some email. The first one in my mailbox is from a member of a Catalan "Mallach" family. 
From: Conchita Broquetas <familia_mallach_broquetas@yahoo.es>
Subject: Hola!
To: jordi@sindominio.net
Date: Sun, 17 Jun 2001 16:55:17 +0200 (CEST)
who discovered there was a "Jordi Mallach" other than his brother in the Internet. Apparently we had an exchange on where our families came from (Mallach is all but a common surname... anywhere, and my family has always wondered where it came from). So that's more than 6 years ago. I think I'd love to get a reply to some email sent by me years ago which has been sitting for years in a mailbox, because "I need to reply to this sometime". I think the Mallach-Broquetas are getting one tonight. If you think I'm dumping random thoughts on a vim buffer, it's probably due to me feeling sad today. Sorry, but I feel like typing, and I don't have a typewriter with me. Speaking of sad, nothing beats the next email which sat for some dramatic 6 months in my messy inbox until I found out in the worst of the possible scenarios. Let's go back to late February, 2004, when I had no job, and I didn't have a clue on what to do with my life. 
From: Mark Shuttleworth <mark@hbd.com>
Subject: New project to discuss
To: Jordi Mallach <jordi@debian.org>
Date: Sun, 29 Feb 2004 18:33:51 +0000
[...]
I'm hiring a team of debian developers to work full time on a new
distribution based on Debian. We're making internationalisation a prime
focus, together with Python and regular release management. I've discussed
it with a number of Debian leaders and they're all very positive about it.
[...]
I'm not sure if I totally missed it as it came in, or I skimmed through it and thought WTF?! Dude on crack or I just forgot I need to reply to this email , but I'd swear it was the former. Not long after, no-name-yet.com popped up, the rumours started spreading around Debian channels. Luckily, I got a job at LliureX two months later, where I worked during the following 2 years, but that's another story. I guess it was July or so when Ubuntu was made public, and Mark and his secret team organised a conference (blog entries [1] [2] [3] [4] [5]), just before the Warty release, and I was invited to it, for the same reasons I got that email. During that conference, probably because Mark sent me some email and I applied a filter to get to it, I found the lost email, and felt like digging a hole to hide for a LONG while. I couldn't believe the incredible opportunity I had missed. I went to Mark and said "hey, you're not going to believe this", and he did look quite surprised about someone being such an idiot. I wonder if I should reply to his email today...

Jan Wagner: wordpress plugin and widget packages

Yesterday I was going through the plugins and widgets, if there are any updates or such. I realized again, that updating of plugins (and maybe searching for the relocated download source) makes me mad. Also testing and switching back is annoying much.
So I decided to package plugins and widgets I use. Since some plugins and widgets are not maintained long enought for any debian release cycle or getting obsolet with new wordpress releases, I decided to maintain the packages outside of the debian project. Anyways … some people are also scared about the security situation of wordpress. I guess this isn’t much better for wordpress extensions. Next pity, some extensions or any equivalents aren’t available for wordpress 2.3 (yet). If you run multiple blogs via the stock debian package and wan’t to make use of the new wordpress version in any of your sites, you have update all at once … which maybe not a way you want to go. I decided to repackage wordpress 2.3.0 as wordpress23, which allows you to install wordpress (<<2.3.0) and wordpress23 at same time. If you want to use debian plugin, widget or theme packages or use these from your normal wordpress installation (/usr/share/wordpress) with wordpress23, you need to symlink the dirs. For example:

# mv /usr/share/wordpress23/wp-content/plugins/ /usr/share/wordpress23/wp-content/plugins.orig
# mv /usr/share/wordpress23/wp-content/themes/ /usr/share/wordpress23/wp-content/themes.orig
# ln -s /usr/share/wordpress23/wp-content/plugins /usr/share/wordpress/wp-content/plugins
# ln -s /usr/share/wordpress23/wp-content/themes /usr/share/wordpress/wp-content/themes Remember that package updates of wordpress or wordpress23 may overwrite parts of /usr/share/wordpress/wp-content/[plugins themes] in such situation. BE CAREFULL! You can get the stuff via apt-line:

deb ftp://ftp.cyconet.org/debian wordpress main non-free contrib Or directly from: http://ftp.cyconet.org/debian/archive/wordpress/ The actual package list is: wordpress-plugin-af-extended-live-archive
wordpress-plugin-ald-spa
wordpress-plugin-all-in-one-seo-pack
wordpress-plugin-exec-php
wordpress-plugin-google-sitemap-generator
wordpress-plugin-landingsites
wordpress-plugin-lightbox-2
wordpress-plugin-now-reading
wordpress-plugin-page-links-to
wordpress-plugin-photopress
wordpress-plugin-related-posts
wordpress-plugin-rolemanager
wordpress-plugin-sideblog
wordpress-plugin-spam-karma
wordpress-plugin-stats
wordpress-plugin-ubernyms
wordpress-plugin-utw
wordpress-plugin-utwgooglesitemaps
wordpress-plugins
wordpress-widget-category-cloud
wordpress-widget-samsarin-php
wordpress-widget-utw-cloud
wordpress-widgets
wordpress23 If you want a plugin or widget included and it makes sence to do so, drop me a mail, where to grab. Be aware that UTW isn’t supported since wordpress 2.3 anymore. A tag system was included into wordpress and you need to search replacements for all the nice UTW extension! Stay tuned! , , , , ,

Adam Rosi-Kessel: Multitasking and Classical Music

Two recent print-media articles begging to be linked: First, the Autumn of the Multitaskers in this month’s Atlantic. Despite the age-of-Internet theme, the full article is only available to subscribers. The thesis is one I’ve seen before (e.g., in GTD). Namely: although we might feel like we’re accomplishing more when multitasking, we actually end up less productive, and eventually we’ll lose our minds if we continue to overload our sense. The writer takes the idea further than I’ve seen before, incorporating several astonishing indicators from the zeitgeist, including this NSFW quote from Jennifer Connelly that I won’t reprint, and several foreign policy and domestic political examples of how multitasking is failing our culture and our country. There is a generational divide at work here. Most in previous generations (say, people over 40) didn’t come online until the Internet was nearly full-blown with spam, entertainment news, popups, advertising, streaming stock quotes, rich media services, IM, Facebook, Flash, etc.. These people generally deal poorly with information and sensory overflow because they had no incremental introduction. The younger generations (people under 25) don’t remember the world before the Internet and multipurpose portable devices, and thus think it entirely normal to watch TV, surf the web with a few open windows, talk on the phone (perhaps VoIP), and IM at the same time. At least. And they’re bored if they don’t have at least have three or four tasks going simultaneously. This isn’t to say that these gen-Yers are actually doing a good job at it — I find myself often answering computer-related questions from my younger siblings because they can’t actually read a focused page or two of dense text on the computer (e.g., a readme file) in one sitting — but rather that they don’t experience the flow as psychologically overwhelming. That leaves my (in between) generation, generally people born in the 1970’s. Those of us who grew up with early low-bandwidth wide area networks — I started on BITNET, USENET, and BBS’s in the early- to mid-1980’s — have better mechanisms for coping and prioritizing. Perhaps because our multitasking options were relatively limited, we haven’t leaped on the bandwagon to the same extent as others. I personally don’t mind two simultaneous tasks — e.g., perusing RSS headlines and listening to MP3s — but more than that doesn’t interest me. (I don’t count it as “multitasking” in the same sense where one of the tasks is just waiting, e.g., for code to compile or a large upgrade to download). Please feel free to destroy my broad generalizations here in the comments. While we’re on the subject of multitasking, I fall into the “subtle exception” category described in Getting Things Done:
The Multitasking Exception There’s a subtle exception to the one-item-at-a-time rule. Some personality types really need to shift their focus away from some task for at least a minute in order to make a decision about it. When I see this going on with someone, I let him take two or sometimes three things out at once as he’s processing. It’s then easier and faster for him to make a choice about the action required.
I find this trick particularly helpful in getting unstuck. Frequently, when drafting briefs, I’ve rewritten a sentence eight or nine times and it still doesn’t seem to get across the necessary idea efficiently and without making the reader do any hard work. Switching to another (usually simpler) task, if only for one or two minutes, is often enough to come back and do it right in one swoop. In this situation (and probably only in this situation), multitasking actually saves me time. Second, the Well Tempered Web (by Alex Ross, music critic with a blog) in this week’s New Yorker. This one is more naturally available in complete form online. Ross manages to weave together Wagner, Snakes on a Plane, the Long Tail, and Rick Rubin. Worth reading. (And while you’re at it, check out Adam Gopnik’s piece on abridgments, which unfortunately is not available online.) Lest you all think all I read is the New Yorker and the Atlantic: it’s not true. I’m just not at liberty to disclose my other sources at this time.

Share This

Andrea Veri: BehindMOTU: My interview

Original post can be found here. Interview: Today we are interviewing Andrea Veri, fresh MOTU and eager Ubuntu volunteer. Age: 18
Location: Udine, Italy
IRC Nick: bluekuja How long have you used Linux and what was your first distro?
I started using Linux at the end of 2005 using Red Hat and Fedora distros, contributing on writing several pages for Fedora documentation (mostly server docs) but mainly working on some packaging-related activities (introducing ctorrent, gtorrent-viewer and v2strip packages inside Fedora) for more than 3 months until the beginning of March 2006 when I decided to move definitely to Ubuntu after discovering it at a friend s party. Was love at first sight that made me leaving every Fedora plan and project creating my first personal wiki page on wiki.ubuntu.com some days later. How long have you been using Ubuntu?
In fact, I started using Ubuntu at the beginning of 2006, firstly getting involved inside the Edubuntu family making real the possibility to have an Edubuntu Italian support and website area inside the current Italian LoCo Team. When did you get involved with the MOTU team and how?
Right after joining the Ubuntu brigade I started checking out MOTU documentation, mainly packaging guide plus debian new maintainer s guide, trying to understand every single new word and applying directly to a source package every lesson learned during developer s world travel . After getting introduced and fascinated from an active community, I had to left the project for a while for some small problems, restarting everything on May 2007 with my first sponsored upload inside the archive. My packaging passion increased right after meeting Alexander Sack inside #ubuntu-mozillateam irc channel some days later, deciding to work directly with him as my mentor for both Debian and Ubuntu distributions. What helped you learning packaging and learning how the Ubuntu teams work?
I started with Debian New Maintainer s guide and Ubuntu s packaging guide moving then to package my first applications learning from already-packaged software and asking if needed to Alexander improving and learning every time from him or from other developers a new Ubuntu Team lesson. Favorite part of working with MOTU?
Introducing a fix making tons of users happy is one of the best things I appreciate of being a MOTU. Mentoring, sponsoring, helping out new contributors or students is something special as well. Any advice for people wanting to help with MOTU?
I always suggest to start with a package a new contributor cares about personally, that s useful to improve/fix the package itself during its maintenance.
Reading MOTU and Debian documentation is a great starting point as well to avoid any strange question on our MOTU irc channel. What packages/areas of Universe are you most interested in?
I m currently working on a vast area of packages, but I ll try to focus on p2p (Peer-2-Peer) applications both for Universe and Main. I planned to create a MOTU-p2p team really soon including it inside the existing motu-torrent team, but it will take some months to organize everything up; contributors (testers/packagers) are currently missing. Any Plans for Hardy Heron?
I ll keep working on a large number of packages but as I said before I would like to focus on having an updated situation of p2p applications, introducing libtorrent-rasterbar and its related clients like btg or linkage. Creating a working team with interested contributors and developers will be the first step to work on. Favorite quote?
As for me, all I know is that I know nothing. Socrates
What do you do in your other spare time?
I love going around with my motorbike, listening good music, playing basketball and meeting up with friends around the city centre. Pic of you, your work area, and/or your screen?

Andreas Schuldei: Dear Lazyweb: Au-Pair wanted!

So we will move back from Sweden to Germany after 7.5 years. Besides swedish nature and working culture we will surely also miss swedish broadband. On the other hand we survived swedish health care without serious repercussions and look forward to the german alternative.
So after having found a really nice house in L beck's historic center we now look for an swedish Au-Pair. One of the main goals with that would be to help our children to keep their swedish language skills. If anyone reading this knows someone interested in working as an Au-Pair during 2008 please have them contact me.

This years first Extremadura work meeting with people from Debian Edu, FAI, Debian-Science and Debian-Med takes place at time of writing. The group is 20 people (with some locals included). There is a rich supply of good spanish food so I skipped dinner yesterday in order to let my digestive system recuperate and get work done at the same time. I feel tempted to do that again today. Having stared and poked at FAIs new partition framework makes me consider partman as an alternative.

Enrico Zini: liferea-summarise

Automatic feed summaries What do you get if you merge this and this? You get a scary little filter that extract the top 5 keywords from a post content and adds them to the titles. Useful titles for all posts: how about that? 
#!/usr/bin/python
# Copyright (C) 2007 Enrico Zini <enrico@debian.org>
# This software is licensed under the therms of the GNU General Public
# License, version 2 or later.
import libxml2, re, math
tokenize = re.compile(r"\W+")
doc = libxml2.parseFile("-")
root = doc.getRootElement()
# Create an xpath context and register the namespaces
xpc = doc.xpathNewContext()
for d in root.nsDefs():
    if d.name == None:
        xpc.xpathRegisterNs("rss", d.content)
    else:
        xpc.xpathRegisterNs(d.name, d.content)
# Collect text stats to generate summaries
doc_tokens =  
for x in xpc.xpathEval("/rdf:RDF/rss:item"):
    res = x.nsProp("about", "http://www.w3.org/1999/02/22-rdf-syntax-ns#")
    tokens =  
    for c in x.children:
        if c.type != 'element' or (c.name != 'encoded' and c.name != 'title'):
            continue
        for t in c.children:
            if t.type != 'text': continue
            for tok in tokenize.split(t.content):
                if tok in tokens:
                    tokens[tok] += 1
                else:
                    tokens[tok] = 1
    doc_tokens[res] = tokens
# Aggregate token counts
aggregated =  
for d in doc_tokens.itervalues():
    for tok, count in d.iteritems():
        tok = tok.lower()
        if tok in aggregated:
            aggregated[tok] += count
        else:
            aggregated[tok] = count
def tfidf(doc, tok):
    "Compute TFIDF score of a token in a document"
    tok = tok.lower()
    return doc_tokens[doc].get(tok, 0) * math.log(float(len(doc_tokens)) / aggregated.get(tok, 0))
def top5(doc):
    # Output the top 5 tokens by TFIDF
    return sorted(doc_tokens[doc].keys(), key=lambda tok: tfidf(doc, tok), reverse=True)[:5]
# Rewrite the titles
for x in xpc.xpathEval("/rdf:RDF/rss:item"):
    res = x.nsProp("about", "http://www.w3.org/1999/02/22-rdf-syntax-ns#")
    for c in x.children:
        if c.type != 'element' or c.name != 'title':
            continue
        for t in c.children:
            if t.type != 'text': continue
            name, content = t.content.split(":", 1)
            summary = ", ".join(top5(res))
            t.replaceNode(doc.newDocText(name+" ["+summary+"]:"+content))
# Serialize the result
print doc.saveFormatFile("-", True)

19 October 2007

Steve Kemp: As I move on through the year

Bits from the Security Team I am happy to look over patches, built packages, and generally encourage people to be involved. Our team isn't huge but historically we've only added people who've done a fair bit of work first. That is both good and bad. I could write more, and probably should, but I'll stop there for now because I'm frustrated by the HPPA build machine. Again. ObRelated: Moritz is trying to get the archive rebuilt with security features from our compilers (eg. -fstack-protector) included. This would be a fantastic achievement. People interested in tested kernel patches, donating buildd machines, etc, etc should give him a ping.

Next.

Previous.