I'm considering to embed some spamtraps (i.e. email adresses that will feed all their incoming email to the spam filter) into some web pages.However, I want to prevent people from accidentially using these links or even just seeing them. So using "display: none" seems appropriate. But Google is known to punish websites 'hiding' content from users but not from robots.Some sites say, Google will just ignore the parts that are within "display: hidden", others say it will punish the site altogether.Maybe the adsense control comments will help

<!-- google_ad_section_start(weight=ignore) -->

but it wouldn't really make sense. It's meant for adsense only.Or the page could become a bit more hackish and use javascript to kill the unwanted content. Any experiences with the proper way of hiding spamtrap email links like this:

<a href="mailto:aaaaaaa-never-email-this-address@domain.tld"
>Unwanted Emails only<:/a>

Thursday evening, I joined the inhabitants of the twin town of Valga-Valka, sitting smack on the Estonian-Latvian border, to celebrate their entry into the Schengen treaty. Ever since I first crossed the border there on a roadtrip from Tallinn to Fallingbostel, in year 2001, I knew I would have to return and, sure enough, I briefly passed through during last summer with an Estonian friend, on our way to an acquaintance's birthday party. Still, that told me nothing of the town's life and left me wanting for more. Upon hearing that year 2004 EU accession countries would join Schengen in December 2007, I immediately promised myself to show up and join the crowd. As it turns out, I missed Aleks Tapin of the All About Latvia blog by very little, having I caught his last-minute e-mail the next afternoon. Aleks blogged a great article depicting the Latvian side of life and providing some background info on the town, if you're curious. I arrived late-evening on Thursday via the Tallinn-Viljandi-Valga bus and checked into my hotel, then proceeded to checkpoint Valga II around 23:30, with the intention of grabbing a drink on the Latvian side and returning just before midnight for the celebrations. Hardly anyone was in sight, except for Latvian officer abana who was completing the inspection of a Russian car with three noisy passengers. Upon presenting her with my passport, officer abana cheerfully lead me to the office and slid my passport through a slot to officer Bukss, who was visibly surprised to have any tourist show up on the last day of his job to get their passport stamped. Upon explaining to him the reasons for my presence, he pointed me to a nearby bar where I could have a drink, visibly amused by the situation. I spent the next few minutes in a local casino in the company of three hilarious Latvian truck drivers, Zintars and Aivars, two Latvians with limited English language skills and Yuri, an ethnic Russian living in Ireland who was born stateless on the Latvian side of the town and who later acquired Estonian citizenship by claiming ancestral land on the Estonian side. Upon returning to the border, just minutes before midnight, I found myself in the middle of a huge crowd of villagers, police officers, border guards and politicians from both countries - barely getting noticed by anyone. I handed my passport to an Estonian border guard who emotionally commented to a civilian friend of his nearby that, "Wow! That was the last one!", handing me my passport back just as the midnight bells rang and the mayor of the Estonian side started his speech amidst pyrotechnics lining the road. I then had a chance to chat with the Estonian mayor, who promptly handed me a bilingual certificate, signed by the mayors of both towns, attesting that I was the first to cross the open border, while introducing me to his two young daughters with whom he was about to take a stroll on the Latvian side. The next morning, an even bigger and more symbolic event took place: the demolition of the fence that had been cutting the S prus/Rai a street in half. You can see a picture of what the street looked like before on Aleks' blog article [edit: or on Jens-Olaf's blog article]. After the fence was removed, it instead looked like this: The crowd then proceeded to checkpoint Valga III, where a stage had been setup for the politicians to make their speeches. An interesting fact is that, because of the way the border was drawn, along a creek leading to Pedeli river, an Estonian main road was technically on the Latvian border and thus aptly named "Raja" (border) street. The Estonian mayor commented in his speech that the the brand new supermarket standing behind us was also technically on borderland and could not have possibly been built earlier, weren't it for the Schengen treaty matter-of-factedly eliminating borders between participating EU countries. The speeches were closed by a youth group whose choreography featured six break-dancing boys, dressed as Estonian and Latvian border guards and as border posts. The choreography ended with the teenage border guards carrying away the human posts on their shoulders, just as a choir of young girls replaced them with songs in either languages. Before catching my bus back to Tallinn, I dropped by the Valga tourist info, only to face a nervous-looking Kapo officer in plain clothes. Upon mumbling something about the tourist info and wanting to grab maps, I noticed that a press conference was taking place behind. As he finally let me pass to the tourist info side of the building, a familiar voice said to me, "Hey! Weren't you at our place last summer with Martin Ranna?" Yup, the wife of the acquaintance whose birthday party I had attended last summer is working there and greeted with me refreshments and munchies she had cooked in prevision for the presidential visit! As was my luck, she had one of the commemorative plaques that had been given to the politicians at Valga III on hand: I was glad to see the town becoming one again, even though it is ethnically divided. There are already signs of people on both sides learning one another's languages and shopping on either side of town, not to mention plans of merging the municipal bus operations of the Estonian and Latvian halves of the town, so I'm sure that they'll get there in due course. For me, Schengen brought a much more practical and quite welcome change: the end of messy border crossing stamps that were rapidly filling my almost new passport, every time I visited the head office of our company in Tallinn. While Estonian border guards mostly stamp passports in an orderly fashion, being careful to fit exactly 8 stamps in a single page and to put entry and exit stamps side-by-side, Finnish border guards apparently are incapable of doing so, instead systematically wasting pages by either stamping sloppily in a way that makes it impossible for more than 4 stamps to appear on a given page or by flipping to a brand new page, every time I had to cross the border. In case anyone ever wondered why I'm currently representing Estonian interests abroad, despite living in Finland, this example is one of the many reasons: Finnish authorities routinely display arrogant carelessness towards the population and doubly so towards immigrants. It amounts to an accumulation of idiocy that has costly consequences on people's lives. For some, it's about being denied the public institutions' support when they need it the most and their life forever going downhill thereafter. For me, it has been about countless missed opportunities (jobs, love affairs, travel plans), plus very costly passport and residence permit renewals. Given this, I simply don't see myself ever representing Finnish interests ever again. Faith no more. Besides, the Estonians are fun and easy-going people. Siski ma m tlen, kas ma peaksen n d Eestisse kolima v i?

Some more important things happened this week for Debian Installer i18n:

• First of all, Amharic was activated. The longstanding translation effort is reaching its end with Tegegne Tefera completing 87% of the core D-I translations and, more importantly, all strings for the most used packages. Amharic is the national language of Ethiopia and is indeed our second 'native' African language, after Wolof. This is indeed the first language we activate for lenny and that could be a sign of some revival in D-I i18n work.
• I also completed the tests phase of a new version of the 'synchronization script' that allows having all translatable strings for D-I package in a single (giant) PO file. Up to now, we were requiring translators to complete that file before we activate languages. However, many strings in the file are related to very rarely used features or high-end architectures. As the probably of someone installing Debian with LVM support on a s390 machine in something else than English is pretty low, we had to do something. So, very soon, the core D-I file should be splitted in five subfiles:
  • sublevel 1: default installs on common architectures
  • sublevel 2: general strings not used during default installs
  • sublevel 3: expert strings (some low prio, some for e.g. RAID)
  • sublevel 4: specific to less-popular arches (powerpc, mips, sparc) or used in experimental features
  • sublevel 5: same for high-end (hppa, ia64, s390) and hobby (m68k) arches and old stuff (partconf, partitioner, that are replaced by partman for a while)
  This will of course allow translators to better prioritize their work. We'll probably activate languages when translators complete the first two sublevels or so (that remains to be discussed). Thanks anyway to Frans Pop for the numerous good ideas he bringed when we discussed this.
• Armenian got its first commit in the D-I SVN. While the translated ratio is still very low, that's a positive sign
• Effort to translate D-I in Kashmiri should start soon as Rakesh Pandit seems very motivated to work on it
• Several languages from India should also soon raise their translation ratio: Kannada, Telugu, Marathi are expected...

All this work is meant to give some more push to D-I localization effort that paced for some time last months while I focused my attention on some other areas. It seems to show that things never work without someone to care about them, apparently..:-)...so, existing translators, watch out, I'm back on D-I i18n and will kick your asses if you don't keep your translations up to date..:-)

I sometimes look at the download statistics of SymPy and the funny thing is that people download more the Windows installer, rather than the multiplatform source tarball. Part of this can be due to the fact, that SymPy is in Debian, Ubuntu and Sage, but still.

So here is how to make a windows installer in Debian:

$ ./setup.py bdist_wininst
[...]

and a file dist/sympy-0.5.9-hg.win32.exe is created. I only have regular python tools installed. So we just upload the exe to our site and that's it -- kind of scary, since I don't have means to test it.

But one time I tried that on one of brother's computers with windows and not only it worked, but also the 3D plotting using pyglet worked out of the box! If you are curious how it looks like:



So it has it's advantages to develop pure Python programs - they really run everywhere.

I think installing things like Python and playing with SymPy on windows must be pain (maybe people just download SymPy for windows and then run away with disgust). But just the fact that they try means, that people doing science do use windows a lot. And I don't think this is going to change any time soon. I also fully agree with Michael Abshoff's recent post about this issue.

Liferea’s upstream has decided to stop puting Debian Planet as a default example feed for new installs. I agree with that change for the upstream distribution, but I’m still going to revert it for the Debian package. And, of course, that gives us the chance to do even better. I’ve added Planeta Debian to the Spanish default feed list and Planet Debian-FR to the French one. I wanted to add Debian Brasil to pt_BR, but their feed returns a 404. So, do you know about Debian planets in other languages? Please open a wishlist bug against liferea with the name, URL, the URL of the feed, the language it’s in, as well as some way I can verify the site is “official”, where that means it’s mostly published to by developers and heavy users. The languages which currently have a customized list are bg, es, nl, pt_BR, sk, ca, eu, pt, sv, de, fr, pl, ru and tr. If your language is not one of those, you’d do better to start by contributing a non-Debian specific feed list for your language, directly to upstream (and maybe translating the whole program, while you’re at it ).

With this patch, dpkg-buildpackage starts passing a new variable DEB_VARIANT into the package build process which can then be used by the package to customize itself on different systems. As already written in the bug report, the prime candidates for this are the gcc package (which currently looks into /etc if it's building on Ubuntu, and then applies a patch that is left out on Debian) and packages often used on embedded systems, which could benefit from splitting out documentation and translations, but where we don't want that in the main Debian archive. As usual, feel free to tell me that I'm on crack.

It starts off just as you'd expect.

Yes, it s that time of year again. No sooner does an important traditional religious holiday roll around than the PC-brigade feel the need to strip-mine it of its original significance, just so s no-one s feeling get upset.

However, it doesn't quite continue as you were thinking.

It is the Christians who have the most gall of all, daring to attach the name of some first-century Palestinian to a once-proud British festival. Yule I can live with, despite its being a continental bastardisation of our British pronunciation Geola , but Christmas is just wrong. You even have to mispronounce Christ to say it.

An excellent piece of satire from Nathaniel Tapley on Liberal Conspiracy, which is well worth a read for a good laugh. NP: Giant Steps, John Coltrane

Every since the Maemo Chinook beta was relased, people have been asking when we're going to make Pimlico packages available. I'll skip over the fact that Pimlico is open source, so building a package yourself is trivial (I suppose its a good thing that there are users who can't do that!). Now, Pimlico consists of Contacts, Dates and Tasks. Contacts doesn't have a Maemo port as such and the device already has an addressbook of sorts, so I tend to leave that until last. That leaves Dates and Tasks, both of which require an all-new Evolution Data Server to be built, because Nokia strip down EDS for Maemo and don't install the calendar component to save disk space. This generally shouldn't be a problem as we've been building replacement EDS packages for some time now which adds the calendar component and restores some functionality to the common libraries. So, Rob set about re-syncing our packaging with the Chinook EDS (there are several Maemo-specfic patches we obviously need to include) and started a build. Then our plan started to fall apart. To cut a long story short, the Chinook's Application Manager has an additional sanity check which wasn't in previous releases. The gist of it appears to be that a package foo from repository A cannot upgrade package foo from repository B. Or, libebook from maemo.org cannot be upgraded to libebook from o-hand.com. I can see how this can stop people accidently breaking their device by installing broken core packages, but it's also stopping us provide core packages with enhanced functionality. Never fear, we have a plan. It's not incredibly pretty and will take a day or so of tiresome recompiles to get working, but we'll get there. That said, we've all been busy and now it is the Christmas holiday... so I wouldn't bet on being able to run Dates and Tasks on your N800/N810 before the new year. As a reward for patience, however, there are Contacts 0.8 packages for Chinook in our repository. NP: Last.fm Recommendation Radio

I'm so annoyed with myself. So today was the day I was going to upgrade the disks in daedalus. I'd finally managed to convince Dell to sell me two new 140Gb disks to replace to two 70Gb disks that were fast approaching capacity. I'd organised for access to the datacentre from 10am. I arrived early. I didn't manage to get into the datacentre until about 11am, because they were having problems remotely unlocking the door, and I had to wait until an air conditioning repair guy turned up, who was fixing some air conditioning problem. I hadn't had a lot of time to plan how I was going to do the upgrade, but I figured it was going to be pretty straightforward. Each disk is partitioned into three partitions, all of which are mirrored using software RAID. The first partition is the root filesystem, which is just an ext3 filesystem, mirrored. The second partition is swap, also mirrored, and the third partition is an LVM physical volume. The first two partitions are only a couple of gigabytes. So the basic plan was to remove the second disk (disk 1, aka /dev/sdb), put in the new bigger disk, load the partition table from the original disk 1, resync the three mirrors, and then boot off the new disk 1, and rinse and repeat with disk 0. This was all going swimmingly, when right at the end (in fact after the third partition finished syncing according to the kernel), /dev/sda decided it had a few unreadable sectors. This made the software mirroring have a dummy-spit, and decide it needed to restart the syncing from scratch. I'd seen this before with a bad disk. So I wishfully hoped it was something to do with the new disk being bigger than the old disk, so tried to roll back to the original disk 1. As was pretty much expected, disk 0 exhibited the same problems at the same point. So the problem was disk 0, not the new disk 1. So it seemed like I just managed to pick the wrong half of the mirror to work from. I thought I'd try manually dd'ing the third partition from /dev/sda to /dev/sdb, ignoring the errors, but that didn't result in a usable half of a mirror either. Interestingly, these bad sectors hadn't made themselves known until this point. The RAID array had been otherwise healthy, and SMART hadn't uttered a peep. I think at this point, after spending about 3 hours fighting with it, I decided to move up a layer. I could successfully sync /dev/md0 (the root filesystem) and /dev/md1 (the swap device), and the only reason I could think of these bad sectors not giving me grief already, was because they were right at the end of the disk, and I hadn't quite used all the free physical extents in LVM yet. Lucky I hadn't decided to grow things any further. So I decided that rather than trying to convince this existing /dev/md2 to sync with the new /dev/sdb3, I'd just build a new degraded RAID-1 on /dev/sdb3 and move all the physical extents from the degraded RAID-1 on /dev/sda3 to the new degraded RAID-1 on /dev/sdb3. I tried the logical volume with /tmp on it first, since it was fairly sacrificial. It worked fine. So I tried the largest logical volume, /srv, which was 30Gb. That went fine as well, so I moved the rest. All of them went without complaint. So then I removed the now unused /dev/md2 from the volume group, and finally managed to boot with just the new disk 1. Then I was able to put in the new disk 0, and just do a standard RAID-1 rebuild onto it. Hooray for LVM saving the day yet again. And this is where I should have stopped. At this point it was about 3pm. I'd been aiming to leave by 4pm to pick up a rental car that I had to get by 5pm. I'd wanted to also upgrade to Etch while I was physically in front of the box, in case there were any nasty kernel/udev issues that made the box unbootable. The catch was, I needed to grow /var, which was quite full. The intention had been that once I'd got the two new disks in and all synced up, I'd create a fourth mirrored partition, and add this to LVM as a new physical volume, then I could just keep merrily growing things. It felt a little bit dirty having to make a new partition to get at the extra space, and I'd reading about mdadm's "grow" mode, whilst I was sitting around waiting for things to sync. So I thought to myself, I'd just delete the third partition, and recreate it using the remaining additional cylinders I now had at my disposal, then use this grow feature of mdadm to tell Linux the array was now bigger, then resize the physical volume, and go from there. This is where things went a bit pear-shaped. I deleted and recreated (larger) the third partition on each disk, and rebooted. I was then messing around with mdadm trying to update /etc/mdadm/mdadm.conf, since the UUID of /dev/md2 had changed, when I discovered that /dev/md2 had completely disappeared. It seems that recreating that partition made Linux completely fail to see /dev/sda3 and /dev/sdb3 as RAID-1 members, and so because of the fairly transparent nature of Linux's RAID-1, LVM had seen the underlying physical volumes, and just decided to run with /dev/sda3 as the physical volume, and everything had continued to work. I just had no redundancy any more. So it was about 3:20pm at this stage, and I was starting to panic, thinking I'd never get this sorted out before I had to leave, and I'd end up returning to the US with a non-redundant mess on my hands. Then I remembered that I had the backups of the partition tables of the old disks, from before I removed them. So I thought I'd go back to that state, and just have the additional cylinders unallocated. I hadn't yet attempted to grow the RAID-1 (partly because it had vanished), so in my haste, I didn't see this being a problem. The problem was, I'd booted into multi-user mode, and things had merrily been operating on just half the mirror. I restored the partitions, and rebooted, and lo and behold, /dev/md2 magically reappeared, already 100% consistent according to /proc/mdstat. I was somewhat surprised by that, but I booted back into multi-user mode again, only to discover that the /var filesystem was panicking and remounting itself into read-only mode. This was pretty weird, given that during boot, it was considered clean by e2fsck. So I was really getting bothered by this stage, so I rebooted into emergency mode (single-user mode in Debian starts way too much), and manually started up LVM, and forcibly fscked everything. There was a bit of filesystem damage to /var, /home and /srv. Thinking I'd now fixed the problem, I rebooted into multi-user mode again, but again /var, which was considered clean during the fscking/mounting phase of booting, panicked shortly after mounting, and remounted itself read-only. So I rebooted again, broke the mirror, and forcibly fscked the filesystems again (only operating on what was on /dev/sda3. Then I rebooted without incident into multi-user mode and resynced the mirror. At this point I ran away. So I'm really annoyed with myself. This should have been a blog post about how crap Linux's software RAID is at error recovery, and how LVM saved the day yet again, but it was overshadowed by me overreaching in my haste, and managing to cause silent data corruption. The morals of this story:

• If things go off the rails, and you get to a reasonable milestone, and you're approaching the end of your maintenance window, quit while you're ahead.
• Don't do stuff in multi-user mode, just because you can and you want to minimise the length and magnitude of an outage
• Don't rush to try and do something (growing a RAID-1) without researching it properly
• Linux's software RAID-1 sucks when the underlying devices are having issues (I already knew this)
• Linux's software RAID-1 can look perfectly healthy when the underlying disk has issues, as long as the bad bits aren't getting any exercise (I think newer versions of mdadmand/or Linux address this by doing a full array check on a regular basis)
• Linux's software RAID-1 can get out of sync with reality and report that it's in-sync with itself when it isn't really, which causes silent data corruption
• e2fsck can report a filesystem is clean when it's really got issues

I think that is all. I'm just so pissed off that what could have been a perfect (enough) upgrade with zero data-loss, even considering I had a bum disk, wasn't.

A discussion about commas lead to an excuse to have a play with the IA-32 processor performance managment unit (PMU). To start, take two versions of a program to count the number of commas in a text file — one in C and one in Python. The C one runs faster on the input data set of ~60MiB of random data, but why? The CPU performance monitors give are the key to getting some idea of where the programs spend their time. I like to use perfmon2 because it's what I know, but Oprofile can do it too. All the available events for IA-32 are described in the manual; I currently of no better way of finding out about them than just reading it. On Itanium I reccommend Caliper which, for the most common situations, does most of the work for you and presents it in a nice report. Intel's Vtune also does a similar thing. The first thing to investigate is if the CPU is getting enough instructions to keep busy. The IFU_MEM_STALL metric is a good place to start as it is triggered when the instruction fetch pipeline is stalled, presumably waiting on either the ITLB or the trace buffer (Icache). That works out to 0.05% of total cycles for the C version and 1.5% for the Python version, neither of which sets off immediate warning bells. If it did, we could start drilling down to things like the L2_IFETCH and ITLB_MISS events, or the BR_* branch events to try and see why the CPU is having to wait to get its next instruction. Next it is useful to find the CPI (cycles per instruction). This is calculated by the ratio of retired instructions against the number of CPU cycles; since a superscalar machine can issue more than one instruction per cycle this should ideally be much greater than 1 (for example, an Itanium can execute up to 6 instructions each cycle). This works out at a CPI of 1.59 for the C version and 0.26 for the Python version. The Python version is clearly spending a lot of time waiting, because it isn't even able to issue one instruction every cycle. At this point it seems the CPU has enough instructions to do, but it is sitting around waiting to get through those instructions. This suggests the waiting is related to getting data from the cache. The load and store requests from the L2 cache are accounted to the L2_LD and L2_ST events respectively. These have the ability to mask out cache lines in different states of the MESI protocol, but for this we don't care so just ask pfmon to show us everything. This shows us that the Python version does quite a few more stores than the C counterpart. Considering this program should simply be reading the input stream and counting the number of commas, we do not expect much store traffic at all. This suggests the Python version is doing some extra copying, for whatever reason (maybe some Python expert can pinpoint it?). We can drill down a bit more into the memory related latencies. The DCU_LINES_IN event gives the total number of lines allocated in the cache. Another event, DCU_MISS_OUTSTANDING, gives a weighted measure of the cycles spent waiting for a cache line to be brought in. Each cycle spent waiting is weighted by the number of outstanding cache misses (I think the Pentium M I'm using can have up to 4 cache miss requests outstanding at once) and has some caveats, but can be considered a rough estimate of the time spent waiting for a cache line to be brought in. Therefore divding DCU_MISS_OUTSTANDING by DCU_LINES_IN gives us an approximate metric of how long a cache miss takes. So that works out to 7.5 cycles for the C version and 23 cycles for the Python version. This seems to strongly suggest that it is memory traffic that is weighing the Python version down. That is only the initial phase; the results give a high level idea of why one program is running slower than the other. The initial analysis phase generally consists of taking the ratios of certain events to try and get some idea in your head of what the program is doing. Then comes the really hard work; drilling down to figure out how to fix it! Some useful references:

• perfmon2
• Oprofile
• Intel® 64 and IA-32 Architectures Software Developer's Manual
• Ulrich Drepper's memory guide
• Inside the machine
• comma.c

I've been looking for a decent way to read rss feeds for a bit, I've finally given in to the fact that there's actually nothing about that I feel comfortable with... so, what I've started is writting a simple rss to maildir convertor (I know there's toursst, but that appears to want a galleon bookmarks file and appears to use deprecated bits of python). At the moment it is very very basic, there's still a way to go before I'll be fully happy with it, but it's much nicer being able to read rss using mutt than Yet Another Random Program, I'll be hacking up a mutt config to make it look different to e-mail later, but for now as it's still in testing phases, I'm just testing with a bog standard mutt config. I've made the code available through a git repository that should (when DNS has finished updating) be available with a git clone git://git.sommitrealweird.co.uk/rss2maildir.git/ before that it should be available from git clone http://miranda.sommitrealweird.co.uk/~brettp/rss2maildir.git/ . The HTML -> Text parser still needs some work, I'm mostly aiming for it to take HTML and generate almost OK ReST which is easier to read in a text mail client. It still needs paragraph wrapping etc, but I'll work on those over time. If anyones interested, or wants to get involved, or has some nicer ideas for how to make it work, as always I can be contacted at iDunno@sommitrealweird.co.uk :)

Sound Juicer "Sound Sculptures In Space" 2.21.0 is finally out. Tarballs are available on burtonini.com, or from the GNOME FTP servers.

• Add a cluebar when the CD isn't in Musicbrainz (thanks Luca Cavalli, #452047)
• Fix the multiple album select dialog (thanks Rob Bradford, #500815)
• Install 48x48 PNG icon (thanks Andreas Nilsson, #502933)
• Change Deselect All shortcut to Control-Shift-A (thanks Ted Gould, #501442)
• Add Overwrite All/Skip All buttons to the overwrite dialog (thanks Michael Chudobiak, #130782)

Andreas Schuldei writes about my blog post about making it easier to contribute to Debian:

It is much more gratifying for a contributor if his effords have
immediate effects.

[…]

So what can people who want to help Debian do to achive more imminent gratification? Pick projects that let you help directly with direct svn/git/whatnot access (Security team, Debian Edu, Debian-Installer…)

While I agree that Instant Gratification is very important, I don’t think that the proposed solutions solve anything. It’s easy to have things sleep in an SVN repository instead of the BTS (lots of teams do that, sometimes for good reasons, like the lack of sponsors - hint: Games team). It doesn’t make things any better. What we need is tasks whose results will be available without too much wait in Debian unstable, for everybody. Also, I must admit that I know very little about the inner workings of the Security Team. But I have the idea that it involves quite a lot of procedures (helping with bugs tagged security probably doesn’t, but going further than that probably does). Contributing to Debian already involves A LOT of procedures. A new contributor, even very good technically, can easily get lost in all the different ways to package stuff and solve common problems. So we should identify tasks that don’t have huge requirements. Ubuntu has the concept of bite-size bugs. I started working on a page explaining the different ways someone can contribute to Debian. The goal is to provide a good entry point (not something that replaces existing documentation) and keep it at a manageable size. Its focus is mainly to replace the emails we write to people asking “I’d like to contribute, but I don’t know what I can do or where I should start!”. The page is available on wiki.d.o/HelpDebian/Start. Don’t hesitate to improve it (see also wiki.d.o/HelpDebian for a TODO list). When it will be good enough, I plan to move it out of the wiki (the content shouldn’t change much anyway) to eg http://help.debian.net/, so we can improve the design (I really like the idea of adding pictures of past Debconfs like Christian Perrier did in his FOSS.in talk).

A Debian package is uniquely identified by its name and version; there is no further information taken into account by dpkg. This keeps the packaging system rather simple, but has the disadvantage that all information that would otherwise be placed in extra data fields has to be encoded into these two fields. As the names of all binary packages generated from a single source package need to be known when building the .dsc file, no substitutions are allowed in the package name. Back in the Good Old Days, the only extra bit of information in the name field used to be the SONAME from the library in the package (thus implicitly saying that each library should have its own package), which meant that the maintainer had to adapt the debian/control file every time a library's ABI changed. This being the Good Old Days, people writing libraries actually thought about things, so ABI changes were few and far between. These days, libraries are stacked on top of each other, re-exporting the underlying libraries' ABI. The SONAME system could in theory handle these situations if the SONAME of every library whose ABI is re-exported is encoded into the SONAME of the newly built library. So if my own ABI version is "2", and some of the functions I export use types defined in glib, then after building against glib1.2, my SONAME would be somewhat similar to "2+glib1.2". For languages that don't have a well-defined ABI (i.e. anything but C), the version of the language ABI(s) used will also have to be added. Obviously, this would also have to be encoded into the package name, which is where trouble starts during transitions. The last C++ ABI transition had all C++ library maintainers add a suffix to the package name; dependent ABIs are not handled at all in Debian right now (which led to the libpng mess). The existing encoding rules are not entirely formalized, and it is not guaranteed that they are bijective. For example, packages used together with cross compilers have been following the convention of using the name that the native package would use, followed by a dash, the target architecture name, another dash and the word "cross". There is no guarantee that the architecture name does not contain a dash, however, so we cannot simply take the second to last word as the architecture name if the last word is "cross" (think "libgcc1-kfreebsd-i386-cross"), which makes it difficult to write a plug-in to apt that would automatically download the binary for the target architecture and convert it using dpkg-cross. I've decided to nonetheless go ahead with my request that the "-*-cross" part of the package name space be reserved for packages related to cross compilation; it will work for the time being.

I wanted to make 1.16.0 last release this year, but unfortunately one bug preventing reading of all SMS messages from some AT phones sneaked in, so I decided to make another quick release. Changelog is really short:

• Fixed reading of messages from AT phones using CMGL.

Other change is that I started to produce additional binary build for Windows - it does not use any external libraries (so no MySQL, PostgreSQL or Gettext support) and links statically. So you can just copy single binary and it should work anywhere. You can download from usual place: http://cihar.com/gammu/

Emdebian SVN r2618 includes my first working implementation of em_installtdeb. Barring a single annoyance message from dpkg, the script works.
The warning arises from the need for a modified "source" tarball that can be uploaded alongside the .deb, containing only the sources needed for this specific translation: debian/control, debian/changelog, po/$lang.po /* if any */ and po/$GETTEXT_PACKAGE.pot. Where a package contains more than one po directory, all POT files are included and all po files for the requested translation. This allows translations to be updated independently of the upstream package.
I would advise using a test directory if you do want to try it, not your main package building directory.
Output (using gpe-expenses as the demo package) in translator mode:

$ em_installtdeb sv
dpkg-deb: building package gpe-expenses-locale-sv' in ../gpe-expenses-locale-sv_0.1.4-1_all.deb'.
dpkg-source: warning: .orig.tar name gpe-expenses_0.1.4.tdeb.tar.gz is not _.orig.tar (wanted gpe-expenses_0.1.4.orig.tar.(?:gz bz2 lzma))
dpkg-source: building gpe-expenses using existing gpe-expenses_0.1.4.tdeb.tar.gz
dpkg-source: building gpe-expenses in gpe-expenses_0.1.4-1.diff.gz
dpkg-source: building gpe-expenses in gpe-expenses_0.1.4-1.dsc
dpkg-genchanges: including full source code in upload
Format: 1.7
Date: Fri, 07 Dec 2007 00:08:39 +0000
Source: gpe-expenses
Binary: gpe-expenses-locale-sv
Architecture: source all
Version: 0.1.4-1
Distribution: unstable
Urgency: low
Maintainer: Neil Williams
Changed-By: Neil Williams
Description:
gpe-expenses-locale-sv - sv translation for gpe-expenses
Changes:
gpe-expenses (0.1.4-1) unstable; urgency=low
.
* New upstream release
* use machine operable debian/copyright, migrate to standards 3.7.3
Files:
781a60eccc4396ee3f0f86a01150e755 316 misc extra gpe-expenses_0.1.4-1.dsc
f9dd946db27ff6077f8e85f7100d3ea4 5863 misc extra gpe-expenses_0.1.4.tdeb.tar.gz
894f5789ee0348abfa61a5e7342ed6a7 20 misc extra gpe-expenses_0.1.4-1.diff.gz
56f8d22c30cb5502633ff4d26862db78 4532 misc extra gpe-expenses-locale-sv_0.1.4-1_all.deb
Package-Type: tdeb
Changes file: ../gpe-expenses_0.1.4-1_tdeb.changes
signfile ../gpe-expenses_0.1.4-1.dsc 0x28BCB3E3

You need a passphrase to unlock the secret key for
user: "Neil Williams (Debian) "
1024-bit DSA key, ID 28BCB3E3, created 2002-01-27


signfile ../gpe-expenses_0.1.4-1_tdeb.changes 0x28BCB3E3

You need a passphrase to unlock the secret key for
user: "Neil Williams (Debian) "
1024-bit DSA key, ID 28BCB3E3, created 2002-01-27


Successfully signed dsc and changes files
neil@holly:gpe-expenses$ dpkg -c ../gpe-expenses-locale-sv_0.1.4-1_all.deb
drwxr-xr-x neil/neil 0 2007-12-19 14:37 ./
drwxr-xr-x neil/neil 0 2007-12-19 14:37 ./usr/
drwxr-xr-x neil/neil 0 2007-12-19 14:37 ./usr/share/
drwxr-xr-x neil/neil 0 2007-12-19 14:37 ./usr/share/locale/
drwxr-xr-x neil/neil 0 2007-12-19 14:37 ./usr/share/locale/sv/
drwxr-xr-x neil/neil 0 2007-12-19 16:09 ./usr/share/locale/sv/LC_MESSAGES/
-rw-r--r-- neil/neil 6419 2007-12-19 16:09 ./usr/share/locale/sv/LC_MESSAGES/gpe-expenses.mo
-rw-r--r-- neil/neil 1796 2007-12-19 16:09 ./usr/share/locale/sv/LC_MESSAGES/qofexpensesobjects.mo

Hmm, that isn't right - need to check on the ownership of those files - might be enough to use fakeroot.

$ dpkg -I ../gpe-expenses-locale-sv_0.1.4-1_all.deb
new debian package, version 2.0.
size 4532 bytes: control archive= 302 bytes.
230 bytes, 9 lines control
Package: gpe-expenses-locale-sv
Source: gpe-expenses
Version: 0.1.4-1
Architecture: all
Maintainer: Neil Williams
Installed-Size: 44
Section: misc
Priority: extra
Description: sv translation for gpe-expenses


em_installtdeb is a debhelper add-on created by Emdebian to create translation packages (tdebs). em_installtdeb is intended to separate out the individual translation files from the current Debian packages into packages without any translation files and a series of tdeb locale packages, one per translation. Generated packages use the syntax: $package-locale-$language_code_$version_all.deb
Certain language codes need to be modified to make acceptable components of a debian package name. Underscores are converted to hyphens, '\@' is converted to '+' and all codes are made lowercase. These changes only apply to the package name, the installation location is unchanged.
The locale package must use GETTEXT_PACKAGE for the eventual filename of the binary translation file - not the $dh MAINPACKAGE . GETTEXT_PACKAGE is determined by upstream, not Debian. When building the whole package, the binary translation file may be in debian/tmp/usr/share/locale/$lang/LC_MESSAGES but when in translator mode, this location is not available. Instead retrieve GETTEXT_PACKAGE from the POT filename. debian/control is needed for various dpkg and debhelper calls only.

The "source" for translators therefore includes:
debian/control
debian/changelog
po/$lang.po /* if any */
po/$GETTEXT_PACKAGE.pot

Note that no rules file exists. (i.e. dpkg-source -x will complain when trying to extract it later so a new wrapper is needed (not difficult).)
em_installtdeb will try to generate the necessary POT file(s) and then create a $package_$version_tdeb.tar.gz containing the source files.
The default action is to process all available po files.
The above example uses translator mode where LANG_CODE is passed to only handle that language. In translator mode, em_installtdeb builds a tdeb package, a source tarball, a .dsc and a .changes file (and signs the .dsc and .changes). In default mode, em_installtdeb builds the package(s) but leaves the source, .dsc and .changes to the rest of the build process.
LANG_CODE can either be the name of the locale as specified in the PO filename (fr, en_GB, sr@Ltn etc.) or the name of the locale as specified in the eventual package name (fr, en-gb, sr+ltn etc.).
If a po file already exists for this locale, em_installtdeb processes just that one translation and builds a single tdeb package.
If the po file does not exist, em_installtdeb exits with an error. (Copy the POT file from the original source to create a new po file for the new locale and edit it to include the new translation, then run em_installtdeb again.)

Output in default mode:
$ em_installtdeb
dpkg-deb: building package gpe-expenses-locale-cs' in ../gpe-expenses-locale-cs_0.1.4-1_all.deb'.
dpkg-deb: building package gpe-expenses-locale-de' in ../gpe-expenses-locale-de_0.1.4-1_all.deb'.
dpkg-deb: building package gpe-expenses-locale-pt' in ../gpe-expenses-locale-pt_0.1.4-1_all.deb'.
dpkg-deb: building package gpe-expenses-locale-sv' in ../gpe-expenses-locale-sv_0.1.4-1_all.deb'.
dpkg-deb: building package gpe-expenses-locale-vi' in ../gpe-expenses-locale-vi_0.1.4-1_all.deb'.

Next stage is to fold this into emdebian-tools so that the emdebian patch files can be cleaned up and various complex patches to debian/rules can also be cleaned up.
Finally, note that this only supports gettext and only supports program translations. This is an Emdebian script written with an eye on usage in Debian but primarily intended to meet Emdebian requirements. As outlined before, Emdebian does not want language packs or translation grouping. One locale, one package.
See also langupdate in Emdebian SVN.

I spent a rather long time today trying to figure out a rather obscure bug. I few months ago I bought a gp2x, an open portable gaming system (and media player, etc.). I've really enjoyed it so far. The open platform for development has lent itself it many fun apps, console emulators the most conspicuous I think, but it's also fun to experiment and develop with it. I recently ported an asteroids clone I wrote and the whole process was actually fairly simple. As such, I decided to take on a larger task of porting something like FreeCNC, which as far as I know, no one has done yet.

After some simple trouble with library version conflicts, I got the program compiling correctly. However, if wasn't reading files quite right. I spent a little while investigating and discovered the problem was in a function which essentially hashes filenames. In the end, it was this snippet of code which was giving the trouble:

char buffer[13];
...
calc = ROL(calc)+*(long *)(buffer+i)

In particular, this is trying to read 4 bytes from this character array as an integer. At first I though the trouble might be endian-ness, but both the gp2x (an ARM processor) and x86 use big-endian^*. I finally resorted to running gdb and sifting through assembly level instructions.

$r2 = 0xbffffd7c
$r3 = 0x41434f4c
0x0000a310 _ZN8MIXFiles7calc_idEPc+52>: str r3, [r2]

Here, the register $r2 points to buffer and $r3 is the value we're trying to save into the buffer. After this we would expect
*0xbffffd7c = 0x41434f4c
but instead see
*0xbffffd7c = 0x08db9841.
If I purposely set *0xbffffd7c = 0xffffffff and repeat this instruction we see
*0xbffffd7c = 0xffffff41.

Clearly the problem lies in this single str instruction, and was resident to the ARM architecture. So of course it was time to go to the processor documentation. Reading the fine print finally reveals what I was looking for.
A word store (STR) should generate a word aligned address. The word presented to the data bus is not affected if the address is not word aligned. That is, bit 31 of the register being stored always appears on data bus output 31.
Yep, since the buffer was an odd-sized array and stored on the stack, it started on an odd number in memory. Hence its address was not word aligned, and so as per the documentation, the store did not have the expected result. Here's a short program which shows what I'm talking about.

x86	ARM
buf=BFBF3471, buf%4=1 Num: 4C4F4341 Buf+0: 4C4F4341 Buf+1: 4C4F4341 Buf+2: 4C4F4341 Buf+3: 4C4F4341	buf=BFFFFD75, buf%4=1 Num: 4C4F4341 Buf+0: 4F434100 Buf+1: 43410000 Buf+2: 41000000 Buf+3: 4C4F4341

Perhaps the lesson to learn here is that when casting pointers, it's buyer-beware, and there are safer bitwise manipulations to use, though certainly less efficient.


^*Edit: Both x86 and the ARM processor in question are little-endian, not big-endian. Typo on my part, but notwithstanding the point is that they both had the same endian-ness, so it wasn't where the issue lay.

Yes, Bastian, there is a way to do what you want. I believe I told you from the very start: refactor reportbug and implement reportbug-ng on top of the existing code. But no, you chose to reimplement everything (a typical thing to do for GUI programmers) and now you are asking us maintainers to change the way we've been doing things for years? You also make presumptuous statements about how we deal with bug reports. If you personally hate how certain packages prepare bug reports, then maybe you should personally suck it up and use different software? Seems like a more logical and less intrusive way forward, then expecting others to "send a friendly mail" to ask for more information on a bug report. Maybe you have all the time in the world, but I don't. I can work on e.g. mdadm (which uses an interactive bug script) only on rare occasions, and then I'd rather fix and close bugs than to indulge in another iteration of bug submitter ping-pong. You are also personally annoyed by long pre-subject texts, which you doubt anyone reads? I don't even bother to challenge that, but let me say that pre-subject texts are there to prevent stupid bug reports, or bug reports filed in the wrong place, and to take load off the maintainer. So you chose to ignore them? Given the projected user-base of your tool, I'd say you should rather force people to read them by asking them to transcribe the text. If you don't want to pop up a terminal, well, why the heck didn't you integrate debconf properly, while reusing reportbug, and taking a step forward to provide a proper interface for bug scripts? Why do you even bother to whine about interface issues at this point? Feel free to take over my packages, which make use of interactive bug scripts. I see zero reason to change them, and I will not. NP: The Flower Kings: Paradox Hotel