I joined HP's Open Source and Linux Organization (OSLO) at the beginning of the month as an Open Source Community Expert. I spent the last three weeks in Ft. Collins for some training and to get to know the team, but normally I'll be working from home in Europe. The main focus of my work will be governance issues related to the deployment of FOSS in enterprise environments. However, I've already started to get involved in a number of other areas I'm interested in, such as community relations. I'm pretty excited about my new position because it is a good fit for my skills and interests.

Some time ago I've been working on an AI to classify web pages. I've spidered (and classified!) about half a million web pages I'm using for training.This totals to about 40.000 categories and 100.000 useful tokens. I need 8 bytes of storage - two float values - for each combination, which is a matrix of about 27 Gigabytes. I'm using mmap to map my data files into memory for convenient access. On a 64 bit system, I can keep them mmapped all the time.This results in the following funny top output:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
12443 erich     26  10 27.7g 309m  21m R  8.5 32.8  86:01.98 train

So according to top, the process is using 27 GB of virtual memory. That is the data files, actually. The 300 MB resident memory is the training data it's processing. At this point, all my data has been converted to 32-bit integers, so this is already condensed down a bit.It's also obvious that ext3 supports sparse files - running "du" on my database directory currently gives 1.5 GB (it just has started filling the matrix), adding the "apparent size" switch gives 29G.I hope that when the training run has finally completed (ETA is a couple of days), the AI will actually be able to classify webpages somewhat reliably. :-) You can think of it like a huge bayesian spam filter. Except that I've added some boosting that should improve results, some computational improvements for speed and scaled it up to 40.000 categories instead of just "spam".Other uses I'm planning for the AI is automatic email filtering into multiple folders (best to be employed on Google GMail, where an email can have multiple labels, not just one folder), automatic document classification for document and knowledge management systems, etc. pp.[Update: I havn't yet decided if or when I'll release the code as GPL. Right now, the code could require some cleanups. And I'm considering to start a business around this datamining and classification stuff, so I might want to keep it closed for some time at least. The business will most likely be around integrating the code though, so a public release might well be possible.]

Today, I offered myself a nice Christmas gift: I ran my first half-marathon since I resumed running last summer. The day was ideal for this: because of a cold I didn't run yesterday so I was full of energy. Quite low temperature (about -3 C) and wonderful country, entirely frozen (the temperature doesn't raise much above zero since about 2 weeks). It was indeed the perfect moment. Even Moon and Mars still visible at the beginning (see former blog posts). The last 4 kms were....awful and *very* long (whatever circuit I use here, I always end up climbing up as we're located on a plateau dominating surrounding valleys by about 80m). Thus, I completed the distance (21.5km according to Google Maps) in 2h20 which is not that great but, hey, that's the first one...:-) My legs will hurt today..:)

I'm considering to embed some spamtraps (i.e. email adresses that will feed all their incoming email to the spam filter) into some web pages.However, I want to prevent people from accidentially using these links or even just seeing them. So using "display: none" seems appropriate. But Google is known to punish websites 'hiding' content from users but not from robots.Some sites say, Google will just ignore the parts that are within "display: hidden", others say it will punish the site altogether.Maybe the adsense control comments will help

<!-- google_ad_section_start(weight=ignore) -->

but it wouldn't really make sense. It's meant for adsense only.Or the page could become a bit more hackish and use javascript to kill the unwanted content. Any experiences with the proper way of hiding spamtrap email links like this:

<a href="mailto:aaaaaaa-never-email-this-address@domain.tld"
>Unwanted Emails only<:/a>

Full moon is bright in the skies today and tomorrow. Moreover, it will share the skies with Mars, reaching its opposition point. So, for everybody around, just go outside a few hours after sunset and enjoy. Tomorrow (Dec. 24th), some will even enjoy an occultation with the Moon passing in front of Mars. See this page for occultation details and locations. In short: state of Washington, USA, British Columbia (CA) and other parts of Western Canada, Greenland, Scotland, Germany, Czech Republik, Hungary, Serbia, Greece, Turkey, Lebanon, Israel. Then some parts of Armenia, Iran, Uzbekistan, Russia (Siberia) and Aleoutians.

omniORBpy 3.1 was just uploaded. Thanks a lot to Floris for his work on getting omniORBpy up-to-date! Since some packages were renamed, the upload will require NEW processing. In the meantime you can grab these packages here.

It took them half a year, but the Belgian federal politicians finally managed to come up with some form of government. I can't say I'm impressed. Well. In a way I am impressed. After trying for 6 months, Leterme still failed to produce some form of government. Since the situation was becoming quite dire, the King gave Verhofstadt the initiative, who then promptly produced a government in only two weeks. Granted, Verhofstadt's mission was not as hard as Leterme's. Verhofstadt was told to form a government only to handle the most urgent matters, and which will be disbanded at Easter; Leterme was asked to form a government to last for four years, which is how long a Belgian federal government usually holds office. And yes, Leterme's mission has most likely been one of the hardest one in the history of Belgian formation talks. But even if all that is true, I'm baffled at the fact that the intention now still is to make Leterme the prime minister at or around Easter. In my opinion, he's had his chance and blew it. Then again, maybe I'm biased. I've never liked Leterme; and when his party CD&V decided to ally themselves with a separatist party, I decided to gracefully extend my antipathy to his entire party.

In a comment on my post about Bad Project Management [1] Don Marti [2] says “the more money you charge, the less of your time people waste, and the more seriously they take you” and “you can affect the client more, whether you’re peddling idiocy or wisdom“. That’s a nice theory, however I don’t recall seeing an example of it anywhere that I have worked. There are two broad categories of employers, large and small. Large companies have HR departments etc who decide on rough pay scales. Once the budget is approved (by senior management and HR) for someone’s contract fees or salary the amount of money seems to be of little concern to the line manager. They have a group of N people reporting to them who are all paid by money that comes from some mysterious place (with no direct connection to the quality or quantity of work done) and therefore the amount of money paid is not very relevant. Often team-leaders don’t even know how much the people who report to them are being paid! Note that this applies to people who have a similar job title, if I am paid twice as much as another sys-admin in a corporate environment I don’t expect to get more attention, but someone who has a more senior job title (or in the case of the UK has been to a more prestigious school or has a better accent) may get more attention. Small companies are rarely in a position to be silly in this way. This doesn’t mean that the most highly paid person gets the most attention, it’s more a matter of making a logical case. Most of the small companies I have worked for have been run in a sensible manner, and my ideas have often received more attention than those of others because I have had a better technical background than my colleagues. However in some cases my suggestions have received less attention because I was receiving a higher pay rate. The reason is that I was being paid a rate that the company could not afford in the long-term and therefore the people who would work there in future years would have to maintain the systems in question which meant that their ability to do such maintenance was often considered to be more important than my ideas about the best way to design the system. I have worked for a small company that had no financial pressure to do sensible things (they had some sweet deals going with major corporations which essentially gave them money for nothing). They had chaotic management which they could get away with because there were no immediate consequences for doing silly things (such as not forcing their clients to have a method of backing up servers that they installed). I can’t claim that Don’s advice in this regard never applies, but my observations of situations where it didn’t apply lead me to believe that it’s applicable to some situations in limited areas and is not a general rule. Don also refers to “a high number that you would quote to a client you didn’t want in order to get them to go away“. This is an issue that causes a lot of disagreement. One consultant I respect advises people to never make such quotes, if you don’t want a job then say that you are not available - one reason is that if a position is so bad that you want the potential client to go away then you don’t want to be tempted to reconsider if they agree to pay you a silly amount of money, another is that it may be considered ethically wrong to make an offer is not genuine. Another consultant I respect claims that his greatest career success was when he turned up to an interview in casual clothes, explained to the potential client that he would never wear a suit and demanded an excessive amount of money. The client agreed to such terms and he then went on to charge other clients similar amounts of money and continued to refuse to wear a suit. I won’t claim that it’s wrong to demand excessive amounts of money to make an unwanted potential client go away and I will not rule out the possibility of doing so at some future time. But I think it should be noted that the practice is controversial and not referred to as something that anyone might do. It is something that I don’t recall doing at any previous time in my career (so I haven’t done it in a formal manner at least - I may have done so at the informal discussion phase and forgotten). Please note that I encourage you to read my previous post with Don’s comment in it’s original context to avoid any risk of misinterpretation. The purpose of this post is not to disagree with Don (I agree with him almost all of the time) but to disagree with some very commonly held beliefs which Don represented in his comment.

• [1] http://etbe.coker.com.au/2007/12/19/bad-project-management/


• [2] http://www.linuxworld.com/community/?q=blog/6

Share This

Some more important things happened this week for Debian Installer i18n:

• First of all, Amharic was activated. The longstanding translation effort is reaching its end with Tegegne Tefera completing 87% of the core D-I translations and, more importantly, all strings for the most used packages. Amharic is the national language of Ethiopia and is indeed our second 'native' African language, after Wolof. This is indeed the first language we activate for lenny and that could be a sign of some revival in D-I i18n work.
• I also completed the tests phase of a new version of the 'synchronization script' that allows having all translatable strings for D-I package in a single (giant) PO file. Up to now, we were requiring translators to complete that file before we activate languages. However, many strings in the file are related to very rarely used features or high-end architectures. As the probably of someone installing Debian with LVM support on a s390 machine in something else than English is pretty low, we had to do something. So, very soon, the core D-I file should be splitted in five subfiles:
  • sublevel 1: default installs on common architectures
  • sublevel 2: general strings not used during default installs
  • sublevel 3: expert strings (some low prio, some for e.g. RAID)
  • sublevel 4: specific to less-popular arches (powerpc, mips, sparc) or used in experimental features
  • sublevel 5: same for high-end (hppa, ia64, s390) and hobby (m68k) arches and old stuff (partconf, partitioner, that are replaced by partman for a while)
  This will of course allow translators to better prioritize their work. We'll probably activate languages when translators complete the first two sublevels or so (that remains to be discussed). Thanks anyway to Frans Pop for the numerous good ideas he bringed when we discussed this.
• Armenian got its first commit in the D-I SVN. While the translated ratio is still very low, that's a positive sign
• Effort to translate D-I in Kashmiri should start soon as Rakesh Pandit seems very motivated to work on it
• Several languages from India should also soon raise their translation ratio: Kannada, Telugu, Marathi are expected...

All this work is meant to give some more push to D-I localization effort that paced for some time last months while I focused my attention on some other areas. It seems to show that things never work without someone to care about them, apparently..:-)...so, existing translators, watch out, I'm back on D-I i18n and will kick your asses if you don't keep your translations up to date..:-)

On January 15th, I'm going to be giving a talk on a panel at the Computing in the Cloud conference held by Ed Felton's Center for Information Technology Policy at Princeton University. The conference description says:

Computing in the cloud is one name for services that run in a Web browser and store information in a provider s data center ranging from adaptations of familiar tools such as email and personal finance to new offerings such as virtual worlds and social networks. This workshop will bring together experts from computer science, law, politics and industry to explore the social and policy implications of this trend.

I'm going to talk about the AGPL 3.0, why it's important that we put effort into figuring out what freedom for different technologies means, and what the components of freedom for web services might be. Registration is free and bags you a name-tag and lunch. Worth noting perhaps, the conference is sponsored by Microsoft.

The last time I have blogged, few people told me that it sounds somewhat patronizing or such (especially that it may sound so to people that do not know me). Well, I have to admit that it is hard to write in a decent tone — the intention and motivation behind the text is oftentimes hard to express. Certainly, better writing skills would probably help — or choosing easier topics. But that is how I am — and afterall, you never learn much when you only do things you already know how to do. So please, dear reader, take no offense in my blog… If you know me, you can probably see through my writing and understand what I mean. If you do not and still enjoy it, the better. And if you do not, please accept my apology — but in that case, it probably was not meant for you. Yes, I know the blog is aggregated in a few places, and I will probably have to do something about this — aggregate only subsets of the blog, appropriate for each of the aggregators. But pretty please, bear with me, I am lazy and it always took me ages to fix even trivial problems.

I sometimes look at the download statistics of SymPy and the funny thing is that people download more the Windows installer, rather than the multiplatform source tarball. Part of this can be due to the fact, that SymPy is in Debian, Ubuntu and Sage, but still.

So here is how to make a windows installer in Debian:

$ ./setup.py bdist_wininst
[...]

and a file dist/sympy-0.5.9-hg.win32.exe is created. I only have regular python tools installed. So we just upload the exe to our site and that's it -- kind of scary, since I don't have means to test it.

But one time I tried that on one of brother's computers with windows and not only it worked, but also the 3D plotting using pyglet worked out of the box! If you are curious how it looks like:



So it has it's advantages to develop pure Python programs - they really run everywhere.

I think installing things like Python and playing with SymPy on windows must be pain (maybe people just download SymPy for windows and then run away with disgust). But just the fact that they try means, that people doing science do use windows a lot. And I don't think this is going to change any time soon. I also fully agree with Michael Abshoff's recent post about this issue.

I promised it some time ago and now it has been done: Debian packages for the development tree (1.1 series) of Bluefish are ready for Debian users (i386 only). The packages can be installed along with the stable version, so you can safely test new features and give us feedback and bug reports. NOTE: The packages cannot be (re)built on Etch, because debhelper version 5.0.57 or higher is necessary. I will fix this part of debian/rules soon, so it can be (re)built on Etch too.

Some days ago I wrote down my experiences with packaging docbook-defguide. The main (remaining) issues I mentioned were the resources and the time the package needs to build. Even on an AMD X2 4600+ with 6GB of RAM it needs 7-8 hours. Today I met with Torsten Werner. He mentioned, that there are some move JVMs I could try. So I tested alternatives to GIJ this night. I found this short summary about free JVMs, which was some kind of interesting. I began with cacao, which seemed to be fast, but it was killed very early in the build process with an java.lang.OutOfMemory error. Even playing around with the -Xms and -Xmx switches in buildtools/saxon.sh did not help. So I dropped cacao from the list. Seems both cacao and kaffe create similar problems here and are not suitable for building the package. Second alternative I tried was sablevm. It directly throw out some warnings or errors so I directly dropped it too. Next JVM was jamvm. But it was as slow as GIJ. So I dropped it from the list of alternatives too. Then I found an interesting statement in the article I linked somewhere above. The author said, that his perfomance test time with GCJ/GIJ reduced from 433 to 9 seconds, when he compiled his application into a native executable. So I took a fast look through the docbook-defguide build dependencies and found, that Debian already provides a natively compiled Xerces package libxerces2-java-gcj. But there were no packages for libsaxon-java, libxml-commons-resolver1.1-java and docbook-xsl-saxon. So short and dirty: I downloaded the source for these packages, added the necessary stuff to get natively compiled packages too, built and installed them. Fortunately packages with native code already exist (JAXP 1.3 and Xerces) for their dependencies. And what should I say: Now building the TDG needs less then 512MB RAM and it builds in around an hour … even on my system. I will ask the Debian Java maintainers to add -gcj packages for Saxon and XML-Commons and fix my own docbook-xsl-saxon package. This will hopefully help maintaining docbook-defguide.

The current method of carrying young children (less than 4-6 years old) in cars is to have a special car seat fitted in the back seat. This has several significant problems:

• It takes significant space in the back seat. The child seat is going to add at least 10cm to the length required in the back seat and often drives the purchase of larger cars (including SUV and 4WD vehicles that are known for being unsafe - especially for children). Having child passengers in a car is a great distraction for the driver, driving a large vehicle increases the difficulty in avoiding accidents - especially when parking.


• The seat belts of the rear seats are used as part of the mechanism of attaching the child seat to the car. Seat belts are designed to stretch in a crash. It’s recommended that after a crash all seat belts that were used to secure people or objects be replaced as they will have stretched. Seat-belts that don’t stretch will cause more serious injuries. It seems likely to me that a seat belt used to tightly secure a child seat for a long period of time will stretch without a collision. Therefore if an older child is seated where they (or another child) used to have a child-seat then they may be at greater risk in the case of a collision.


• Child seats should be fitted by specially trained experts if they are to be safe. The majority of seats are not correctly fitted and put children at needless risk (the cost of getting an expert to do the installation is small).

Some car companies are offering child “booster seats” that are an optional attachment to the rear seat (I first noticed this when reviewing the specs of the latest version of the car I drive - the VW Passat [1]). This is a good idea, but it doesn’t go far enough. The best thing to do would be to provide a selection of back-seat assemblies as factory fitted options which have built-in baby and child seats. The combinations that would be most desired are:

1. Standard car back-seat for three adults (or two adults for a small car).


2. A regular seat (for an adult) at the road side of the car combined with a baby (backward facing) seat at the kerb side.


3. A regular seat (for an adult) at the road side with a young child (forward facing) seat at the kerb side.


4. A baby seat at the road side with a young child seat at the kerb side.


5. Two young child seats.

It would be quite possible to have all five of these options available from the factory. Of course there are corner cases that this doesn’t cover such as twins or parents who have two children so close together that they need two baby seats. For those cases option 2 combined with one of the current off-the-shelf baby seats would do. The number of different supported options would need to be kept reasonably small to reduce manufacture cost and to allow a reasonable market for second-hand seats. One thing to note is that it’s recommended that the first forward-facing seat a child uses is smaller than the later one. Having options for three different built-in baby/child seats (rear-facing and two sizes of forward-facing) would significantly expand the number of combinations (and thus the expense). I suspect that the safety benefits of having an ideal method of securing a forward-facing child seat would compensate for the disadvantage of having it be too large for the child when they are first placed in it. Another possibility would be to replace the rear seat with a more solid bench with bolt holes for baby and child seats. Securing a child or baby seat to a hard surface with bolts would be a much less technically demanding task than using a seat belt (and thus could be done correctly without expert assistance). Child and baby seats would have to be redesigned for this (I suspect that the safety of them relies on being attached to a soft surface), but after that I expect that safety would improve. For this option the rear seat could bold on to a hard surface that’s suitable for attaching child/baby seats so it would simply be a matter of removing the rear seat and installing the child/baby seat(s). The most common car design in Australia includes a 60/40 split rear seat (meaning that if you have a large item to store in the boot/trunk then you can fold down 40% or 60% of the back of the rear seat to allow the luggage to extend into the passenger compartment). This split could be extended to allow removing the base of the rear seat for 60% or 40% to bolt on child/baby seats. Once a car model had been designed for replacing the rear seat there would be other options available. For example replacing the rear seat with luggage storage space. While almost all cars allow folding down the backs of the rear seats to store extra luggage the option of removing seats that you don’t need to give even more space is not common at all (I’ve only seen it advertised as a feature in vehicles with 6 or more seats). I expect that if this idea was implemented it would allow a small car such as a Toyota Corolla to give an equal or greater amount of usable space for children in the rear as a larger vehicle such as a Toyota Camry. While better options for luggage storage would allow people who don’t have children to use a small car while still being able to carry the luggage that they desire. This would allow considerable savings on car purchase prices and fuel use. I expect that a reduction in fuel use world-wide could be achieved by removing the pressure on parents to buy large cars! The poor support for child seats in cars is really surprising. One of the features that could be introduced is both top and bottom mounts for such seats. There is apparently a standard for this, some (not all) cars support it, but most baby seats apparently don’t. So baby and child seats are secured at the top (to a hook that’s bolted securely to the car frame and which was designed specifically for the purpose) and at the bottom to the seat-belt which was never designed for such things. It’s a pity that some of the money spent on supposedly protecting children from drugs couldn’t be spent on making cars safer for them. The government is in the best position to force car manufacturers to improve their safety features while parents are in the best position to teach children about the dangers of drugs.

• [1] http://www.volkswagen.com.au/pdf/Brochures/Passat_160807.pdf

Share This

Every since the Maemo Chinook beta was relased, people have been asking when we're going to make Pimlico packages available. I'll skip over the fact that Pimlico is open source, so building a package yourself is trivial (I suppose its a good thing that there are users who can't do that!). Now, Pimlico consists of Contacts, Dates and Tasks. Contacts doesn't have a Maemo port as such and the device already has an addressbook of sorts, so I tend to leave that until last. That leaves Dates and Tasks, both of which require an all-new Evolution Data Server to be built, because Nokia strip down EDS for Maemo and don't install the calendar component to save disk space. This generally shouldn't be a problem as we've been building replacement EDS packages for some time now which adds the calendar component and restores some functionality to the common libraries. So, Rob set about re-syncing our packaging with the Chinook EDS (there are several Maemo-specfic patches we obviously need to include) and started a build. Then our plan started to fall apart. To cut a long story short, the Chinook's Application Manager has an additional sanity check which wasn't in previous releases. The gist of it appears to be that a package foo from repository A cannot upgrade package foo from repository B. Or, libebook from maemo.org cannot be upgraded to libebook from o-hand.com. I can see how this can stop people accidently breaking their device by installing broken core packages, but it's also stopping us provide core packages with enhanced functionality. Never fear, we have a plan. It's not incredibly pretty and will take a day or so of tiresome recompiles to get working, but we'll get there. That said, we've all been busy and now it is the Christmas holiday... so I wouldn't bet on being able to run Dates and Tasks on your N800/N810 before the new year. As a reward for patience, however, there are Contacts 0.8 packages for Chinook in our repository. NP: Last.fm Recommendation Radio

Today is last day when I'll be more or less online, so if you wanted something to me this year, it's probably already too late :-). I will somehow follow emails, but do not expect much. Tomorrow I'll spend by traveling back to Czechia (NGO - FRA - PRG), then I will just enjoy time being home and not being pushed to deliver tons of papers to somebody. This is also most likely last blog post this year, so I can not skip the merry Christmas and happy new year phrases ;-).

Today I helped a friend sign up with Virgin Media's broadband service. He was having trouble, as their front-page at http://activation.virginmedia.com/ does a brain-damaged test, and tells you that they only support Windows and MacOS X and that if you have anything else you'll need to phone them at a pound a minute so you can talk to some idiot until you lose the will to live (or some such). After a brief rummage, I noticed that if instead of doing what they tell you to do, you go to https://autoreg.autoregister.net/ you get prompted for whether you're signing up for Broadband or Dialup, and on clicking Broadband, you're lead through the sign-up procedure with no complaints about compatibility, and within 5 minutes he was online with his GNU/Linux (Ubuntu) system and Firefox browser. This makes me wonder why Virgin Media bother putting extra effort into turning users away when in fact the sign-up procedure would work perfectly if they hadn't bothered with the stupid browser test.

I love UPS, I really do.

Their autoattendant: "Please speak want you want, you can say ..."
me: "Agent"
AA: "I'm sorry, I didn't understand that. You can say ..."
me: "Agent"
AA: "I'm sorry, I didn't understand that. You can say ..."
me: "Do I have to say 'Fucking Agent?'"
AA: "One moment please, I will transfer you to an agent."

Andreas Schuldei writes about my blog post about making it easier to contribute to Debian:

It is much more gratifying for a contributor if his effords have
immediate effects.

[…]

So what can people who want to help Debian do to achive more imminent gratification? Pick projects that let you help directly with direct svn/git/whatnot access (Security team, Debian Edu, Debian-Installer…)

While I agree that Instant Gratification is very important, I don’t think that the proposed solutions solve anything. It’s easy to have things sleep in an SVN repository instead of the BTS (lots of teams do that, sometimes for good reasons, like the lack of sponsors - hint: Games team). It doesn’t make things any better. What we need is tasks whose results will be available without too much wait in Debian unstable, for everybody. Also, I must admit that I know very little about the inner workings of the Security Team. But I have the idea that it involves quite a lot of procedures (helping with bugs tagged security probably doesn’t, but going further than that probably does). Contributing to Debian already involves A LOT of procedures. A new contributor, even very good technically, can easily get lost in all the different ways to package stuff and solve common problems. So we should identify tasks that don’t have huge requirements. Ubuntu has the concept of bite-size bugs. I started working on a page explaining the different ways someone can contribute to Debian. The goal is to provide a good entry point (not something that replaces existing documentation) and keep it at a manageable size. Its focus is mainly to replace the emails we write to people asking “I’d like to contribute, but I don’t know what I can do or where I should start!”. The page is available on wiki.d.o/HelpDebian/Start. Don’t hesitate to improve it (see also wiki.d.o/HelpDebian for a TODO list). When it will be good enough, I plan to move it out of the wiki (the content shouldn’t change much anyway) to eg http://help.debian.net/, so we can improve the design (I really like the idea of adding pictures of past Debconfs like Christian Perrier did in his FOSS.in talk).