Search Results: "ag"

24 December 2006

David Moreno Garza: My ten wishes for 2007

These are my ten wishes for 2007. Since Santa didn’t arrive in this part of the world, I think I’m asking this to the Three Wise Men (in spite of they were brought here by the Spanish catolicism, but hey):
  1. Dignity, land and respect to the Mexican indigenous groups.
  2. Return and good health for Commander in Chief, Fidel Castro.
  3. Strongest antineoliberalism in the Latin American region.
  4. Right wing oppression brought down in Bolivia.
  5. Good year socially, economically and politically to Nicaragua with its new hope.
  6. An even better year in Cuban trades; they were the best growing economy in Latin America by exporting to South America and China.
  7. Stop to the free trade agreement and military bases with the US in Ecuador.
  8. Peaceful solution to the Colombia-Ecuador border conflict.
  9. Democracy established in some of those countries where it’s more needed, like Central America.
  10. More social successful programs in Venezuela.
And my bonus one:

Russell Coker: MythTV

I've just been trying to set up a MythTV system, I've had the hardware for this for a while but until my TV broke I hadn't found the time to work on it.

My planned hardware was a P3 system for both frontend and backend. However the Debian MythTV packages take over 250M of memory for each of the frontend and backend programs (more than 500M for Myth programs alone), combine that with some memory use for MySQL, for the X server, and for the rest of a functional Linux system and a machine with 512M of RAM does not perform well. Unfortunately it seems that most (if not all) desktop P3 systems only support 512M of RAM (at least none of my collection of P3 machines does). P3 server systems support considerably more RAM (I used to have a SMP P3 server with 1G of RAM), but a server class machine is not what you want in your lounge room. P4 systems support more RAM but also take more than twice the electricity of P3 systems. The extra electricity use is a waste for a machine that will probably run 24*7, and also requires extra cooling (and therefore probably louder fans which is bad for a machine used for playing audio). Given that I desire the capabilities of a P4 in terms of RAM support and electricity use that is not excessive, my options are one of the recent Intel Core CPUs or AMD64 (both of which apparently use less electricity - not sure whether they would use as little as a P3) or a CPU designed for low power use (maybe a Via). Either option costs more than I prefer to spend (I want to use a second-hand machine valued at $129).

Of course if I could get Myth to use less memory then that would allow me to run both parts of MythTV on the one machine. Please offer any advice you can think of in regard to minimising memory use for Myth on Debian.

Given that I am unlikely to be able to optimise memory use as much as I desire it seems that I will have to go to a split model and try using an existing Pentium-D server machine as the backend and the P3 as a frontend. The up-side of this is that I can turn off the frontend machine when it's not being used (the server runs 24*7 anyway). I will need to get more RAM for the Pentium-D machine, but I had wanted to do that anyway.

The next thing I will have to do is to write SE Linux policy for MythTV and find out why /usr/lib/libmp3lame.so.0.0.0 needs both execmod and execstack (and either fix the library or write policy to permit that accss). I'm happy to have a MythTV server program running on a stand-alone machine with unrestricted access to all resources, but when it's going to run on a machine that is more important to me I need to lock it down.

Now to the design of MythTV. It's unfortunate that they don't support better multitasking options. One feature I would like to see is the ability to play MP3s in the background and then have the music pause whenever TV is selected. When I watch a live show (such as the news) I would like to listen to MP3s before it starts and then at the end of the show (and maybe during commercial breaks) have the MP3 playing resume where it left off. This is exactly the traditional lounge-room functionality we are all used to, when a commercial starts you mute the TV and un-pause the CD player! The advantage of MythTV is that you could do both with a single button - or even automatically with advert recognition!

The method of selecting MP3 files to play is also a little cumbersome and the option to add a song after the current one in the play list doesn't seem to work. Also it's a pity that there is no convenient option to sort the music list by artist, genre, or sone name - there are grouping options but they are all separate and I would like to change the sorting via a single key press.

The startup sequence of MythTV regenerates some images (which takes a few seconds), it seems that this is something that could be cached for a faster startup. Also when I add new MP3 files to the store I have to manually request a re-scan. It's a pity that it can't just check the size and time-stamp of the directories which contain MP3 files and do an automatic re-scan if they change.

Evan Prodromou: 3 Niv se CCXV

We had a tough time getting out of Encinitas yesterday. It seems like everything was going wrong. I'd forgotten to order food for a sick aunt and her family in New Jersey the night before, so I was scrambling with apologies and new orders all morning. Then, when we finally got out of town, we hit traffic for about an hour near Santa Ana. Blech! Fortunately, it's a nice stretch of California highway, and we could see Catalina Island off on the horizon to the west. Our Christmas-eve-eve (or Festivus, if you will) was spent at Maj's parents' house in Long Beach, with about 30+ Jenkinses plus various offshoots and friendly relations. They have a huge house near Long Beach/Belmont Shore, with a giant tiled patio area that goes right into the house. It was a good place for a party. Amita June got some nice gifts from family, including a screaming monkey doll dressed as Superman. Yeah, we already have one screaming monkey doll with an "I LA" t-shirt, so this new one will join our growing collection of screaming monkeys with poor sartorial taste. tags:

Typewriters are broken Yesterday Amita June and I were playing with pyKeys, from the awesome pyToddler suite of tools, and I realized that typewriter keyboards are broken. I was showing her that when she pushed on the key marked "Q", a "Q" would appear on the screen. But that's not what happened: instead, a "q" appeared on the screen. That's not a bug in pyToddler, of course: that's just the way typewriters work. If you click on the button marked with a capital letter, it prints the lowercase letter on the paper or screen. To get the capital letter itself, you have to put the keyboard in another mode, or use a modifying chord key. I'm not sure why this is the case. It's clear why we use modifier keys to make capitals, since we type lowercase keys much more often. But why do keyboards show the capitals printed on the keys, then? tags:

Free Culture Christmas If there's anything that gives a bracing jolt of Christmas cheer, it's 235 Creative Commons Christmas songs. Thanks to Uwe Hermann for compiling this fine list. tags:

Enrico Zini: paiwan-character-list

Character list for the Paiwan language We mapped the available glyphs and accents for the Paiwan language. The letters in alphabetical order:
a b c d e f h i j k l m n p q r s t u v w y z   n       
No uppercase. All the characters are in Unicode except n , which already needs to be requested for the Amis script. We need to design an input method to enter the underlined letters and the n .

Julien Blache: pommed v1.0

mbpeventd has just got a new name for Christmas, and it’s now called pommed :-) I went toward a more generic name as we’re adding support for the PowerBook too (plus it’s a semi-private joke for Yves-Alexis and me). This should, by the way, make it in the next release or the one after the next. New features: gpomme is a GTK+ client suitable for all environments; you should start it when your session starts.
wmpomme is a WindowMaker dockapp, as the name shows. The next version will feature set methods in the DBus interface, and PowerBook support if it’s ready by then. Due to the name change and new packages, the package is in NEW. You can grab it on http://people.debian.org/~jblache/pkg/pommed in the meantime. This release means the mbpeventd-dbus branch is closed :-) Merry Christmas to you all.

David Welton: Apples, Oranges, and Culture

Apples and Oranges A number of people wrote irate comments after my article about Ruby's eBay API beating PHP hands down. They complained that I was comparing apples to oranges, because the Ruby library I was using was nicely crafted, whereas the PHP one was some crufty crud that just happened to be one of the first things you find when you go to developer.ebay.com. First things first, let's look at some better PHP code:
require_once 'EbatNs/EbatNs_ServiceProxy.php';
require_once 'EbatNs/EbatNs_Logger.php';
require_once 'EbatNs/GetItemRequestType.php';
require_once 'EbatNs/ItemType.php';
$session = new EbatNs_Session('config/amd.ebay.config.php');
$cs = new EbatNs_ServiceProxy($session);
$cs->setHandler('ItemType', 'handleItem');
$req = new GetItemRequestType();
$req->setItemID(5652664344);
$req->setDetailLevel($Facet_DetailLevelCodeType->ReturnAll);
$res = $cs->GetItem($req);
echo "<pre>";
if ($res->getAck() != $Facet_AckCodeType->Success)  
echo "we got a failure<br>";
foreach ($res->getErrors() as $error)
 
    echo "#" . $error->getErrorCode() . " " .
    htmlentities($error->getShortMessage()) . "/" .
    htmlentities($error->getLongMessage()) . "<br>";
 
  else  
//#type $item ItemType
$item = $res->getItem();
echo "ShippingTerms : " . $item->getShippingTerms() . "<br>";
print_r($item);
 
versus our Ruby code:
require 'rubygems'
require 'ebay'
include Ebay
include Ebay::Types
require 'localconfig'
ebay = Ebay::Api.new
begin
  response = ebay.get_item(:item_id => '110011765264')
  puts "ShippingTerms : # response.items.shipping_terms "
rescue Ebay::RequestError => e
  e.errors.each do  error 
    puts error.long_message
  end
end
So, they get closer with that, but the Ruby code is still clearer and more concise. If you look around at some PHP5 based examples, they start to be decent as well. As one tutorial writes:
These components combine to create a kick-ass tool that PHP 4 can't replicate without lots of work, if at all.
Leaving aside the fact that a lot of people are still more or less stuck on 4, I think there is something more important to be learned. Culture There are a lot of really smart people who work with, and on PHP. As much as anyone grumbles about any language, it's not easy to create one, let alone create one and make it as popular as PHP. However, as I've said elsewhere, one of the things that made PHP so successful was its ability to "scale down". Part of its value is that it made it much easier for more people with less advanced skills to create things. That is, without doubt, a very fine accomplishment, but it does bring some baggage with it. Namely, that there are a great deal of PHP programmers who do not write very nice looking code. Going back to the eBay example, the code featured on their site is not very tasteful, if you ask me. My bet is that in Ruby-land, that sort of code would not be displayed very prominently. Why? It's not the computer, it's the culture that has grown up around the different languages. Computers do not care one bit about programming languages. They interpret bits and bytes, whatever language is used, so it's really all the same to them. Programming languages matter only to people, because they are the tool we use to comunicate our instructions to machines. This means that they are a very human thing in some ways, and that human aspects of them matter a great deal. What this means is that it's important not only to design a cool language, but to create a good culture around it. To take aim at something nearer and dearer to me than PHP, one of the major failings of the Tk toolkit was that there was little to no attempt to create a good design culture around it. Anyone who was using Linux 10 years ago will surely remember some of the truly horrible things that were created with Tk. Why? Tcl and Tk made creating a GUI an order of magnitude easier than with C and Motif (or whatever windows used at the time), and turned them loose with no instructions or idea whatsoever of how to go about creating a good, intuitive user friendly one - the Tk docs themselves make little to no mention of usability, and even the best Tcl/Tk books pay almost no attention, even in the form of recommended reading, to what to make with this beautiful tool whose hows are explained very well. The unixy community also wasn't exactly a wellspring of creativity and critique in terms of good guis, either. So, yes, I know that it's possible to write good, clever, readable PHP (or Perl, or FORTRAN) code, and bad Ruby (or Tcl, or Python, or whatever) code. However, I'm also convinced that culture counts. People who value clean, beautiful code gravitate to cultures, and, where possible, workplaces where they find other like-minded people. Others, not having as many options, find themselves staring at 13000-line PHP files that grow malignantly by dint of cut and paste, wondering how to tame that which others have wrought!

Steve Kemp: I m going to give it all of my loving, its going to take up all of my time

New computer is working in a nice stable fashion although with no working USB for no reason that I can fathom. (Plug in a device and nothing happens. No kernel messages, etc) I will document this properly a little later, and update the map to match. All told the new machine is: The graphics give me 1280×1024 which is as much as I can handle on my 17″ TFT display, and didn’t require any non-free drivers. I wandered into a couple of local stores and said “I’m looking to buy a graphics card, what do you have in the PCI express flavour?”. The answers ranged from: So whilst I wanted to get an NVIdia card since they traditionally have had drivers which work for me I was happy to get the cheaper one and hope for the best. As it turns out I’m using xserver-xorg-video-ati with no problems and playing DVDs over NFS works without any visible lagging/artifacts/errors. The system has 512Mb to itself, and the rest set aside for Xen guests. Right now I have six instances of Sarge running with 64Mb each. Things are nice and snappy :) (Since debootstrap failed on the 64 bit install of Sarge I had to download pre-made Sarge images, but still a painless process.) Talking of Xen I found a weird problem where when starting up a new Xen guest the host would lose networking. I think this is an artifact of the NVidia 4 chipset - but adding irqpoll to the boot flag(s) seems to have fixed it. OK enough about my new toy(s). Happy Holidays (No complaints about my new cheesy Christmas Logo yet. Result!)

Andreas Barth: Who is lying?

One library is lying on ia64 - it tells libtool to have a link to /usr/lib/libasound.la, but there is no dependency to that on the package. Please compare http://buildd.debian.org/fetch.cgi?&pkg=gcompris&ver=8.2.2-1&arch=ia64&stamp=1166917613&file=log with http://buildd.debian.org/fetch.cgi?&pkg=gcompris&ver=8.2.2-1&arch=s390&stamp=1166227920&file=log. Now, one needs to locate the lying package and fix it.

Manoj Srivastava: Arch, Ikiwiki, blogging

One of the reasons I have only blogged 21 times in thirty months is because of the very kludgey work flow I had for blogging; I had to manually create the file, and then scp by hand, and ensure that any ancillary files were in place on the remote machine that serves up my blogs. After moving to ikiwiki, and thus arch, there would be even more overhead, were it not so amenable to scripting. Since this is arch, and therefore creating branches and merging is both easy and natural, I have two sets of branches -- one set related to the templates and actual blog content I server on my local, development box, and a parallel set of branches that I publish. The devel branches are used by ikiwiki on my local box; the remote ikiwiki uses the publish branch. So I can make changes to my hearts content on the devel branch, and the merge into my publish branch. When I commit the publish branches, the hook function ensure that there is a fresh checkout of the publish branch on the remote server, and that ikiwiki is run to regenerate web pages to reflect the new commit. The hook functions are nice, but not quite enough to make bloggin as effortless as it could be. With the movge to ikiwiki, and dissociation of classification and tagging from the file system layout, I have followed the lead of Roland Mas and organized my blog layout by date; posts are put in blog/$year/$month/$escaped_title. The directory hierarchy might not exist for a new year or month. A blog posting may also show in in two different archive indices: the annual archive index for the year, and a monthly index page created for every month I blog in. However, at the time of writing, there is no annual index for the next year (2007), or the next month (January 2007). These have to be created as required. All this would get quite tedious, and indeed, would frequently remain undone -- were it not for automation. To make my life easier, I have blogit!, which takes care of the niggling details. When called with the title of the prospective post; this script figures out the date, ensures that the blog directory structure exists, creating path components and adding them to the repository as required, creates a blog entry template, adds the blog entry to the repository, creates the annual or the monthly archive index and adds those to the repository as needed, and finally, calls emacs on the blog posting file. whew.

Erich Schubert: More on Debian popularity and use

A few days ago, I blogged on the popularity of Debian, using Alexa ranking graphs. They showed Debian ahead of the other distributions in terms of web page accesses. Ubuntu definitely was the top growing domain there, but it didn't reach Debian yet, and there was no drop to suggest that Debian was actually losing users to Ubuntu. Here is another chart for you: (Note that it has a referrer block, if you are reading my blog via some planet I'm not aware of, I can add you to the whitelist that is allowed to refer to the image. I added this because some images from my photoblog were heavily linked from myspace and similar sites.) Debian
Popcon Graph This is a chart from the Debian popularity contest statistics. Submitting data here is entirely voluntary; the data is used to determine which packages to put on which CDs in the CD set. The chart contains four lines, the top line is for bash; pretty much every system has bash, so this is very close to the number of submissions to the popularity contest database. Note that this number has doubled within one year. Sarge was released in June 2005, and thats where the graph changes significantly. Up to then, popularity contest was a rather uncommon package, but sarge promoted its use somewhat IIRC. So lets look only on the post-sarge part of the graph. Ubuntu Dapper was released in June 2006. Any effect? No. The other lines are gnome-session (green), kwin from KDE (blue) and openoffice.org-base (red). KDE is at about 25%, Gnome at about 50% and Openoffice goes up to 50%. This suggests that a significant amount of Debian installations are actually desktop machines (I remove popcon on all my servers, where I'm more concerned with not having too much software on it). For KDE and Gnome, these values have been quite constant. When only counting "active" users, Gnome drops to about 25% (same for OO.o), and KDE to around 10%. VIM would be at 35%, same for apache. I don't claim these numbers are accurate; it's probably more interesting to compare Gnome vs. KDE and similar things. The total number of submissions (represented by bash in this graph) shows that Debian is quite healthy. Oh, and of course I'd like to invite you to install popularity-contest and contribute your software choice to optimizing the CD layout for etch.

Erich Schubert: Website meta langauges

Zack asked for website meta languages for redoing his homepage. Well, I redid my homepage last february, using XML and XSLT. A monster XSLT stylesheet, because I wanted to keep my template outside of the stylesheet. I can not recommend XSLT. Using it for templating is quite messy. XSLT is okay if you want to transform one representation of the data to another; it's not if you want to add a lot of surrounding markup and things like a sitemap and similar navigation tools. This is next to impossible in pure XSLT, it gets better once you have some extensions (dubbed EXSLT, and supported by pretty much any xslt processor) or maybe with XSLT2. String manipulation is also a pain (at least with XSLT1); I gave up on generating a nice "last modified" date from my subversion tag. Supposedly XSLT2 has some functions that could make this easier (i.e. for parsing and printing datetime information), but the common approach with XSLT is to only supply the required minimum of function you need, and I'm not aware of an easy way to add custom functions, not to mention any large standard library that can efficiently be used (which is the true strength of Java, Python and C#, that they bring a huge collection of pre-written ready-to-use code with them). Of course there are some efforts to write XSLT libraries (especially for XSLT2), and this aforementioned EXSLT is some kind of standard library that even might be efficiently implemented in some interpreters - you can't rely on it to be there and to just work. XSLT isn't useless, but when it comes to presenting data to humans or writing clean, compact code it's not satisfactory at all. I'd give you my Makefile and XSLT file, if they weren't that messy... too many features; I'm generating two languages, a partially expanded navigation menu, etc. - my XSLT is 10k, my sitemap currently 4.5k, the template file is 5k. Anyway. Half a year ago, my favourite templating language was KID templating. I used it for some small projects such as my DNSoupdate tool to edit DNS zones via the DNS protocol (requires a nameserver such as bind which has support for DNS update, uses encryption). It was a perfect match for such tiny pages, but I'm no longer that convinced I still like it for larger projects. What's good about Kid: XML templating language, Python based, easy to use in whichever way you want by writing a few lines of python (i.e. easy to write a Makefile to generate a static version of you homepage) What's not good about Kid: only works with a Python interpreter. I'd prefer to have a templating language that can be used from multiple languages. However the Kid syntax relies on Python, which is really bad. Also I'd prefer to have some component-render model for more complex websites. The current setup is IMHO okay if you have one default layout and one content layout; but if you have multiple components that could be combined differently on the pages, it gets too messy. With my turbogears experiments, Kid was also not very performant (but that might as well be Turbogears fault). Right now, I'd probably still go with Kid. But I've been having a look at JSP 2.1, and JSF / Facelets in particular. There are some things about facelets that I really like (for example, that they use a proper XML syntax, instead of this bastardized almost-XML that JSP usually (ab-)uses). There is also some stuff that I don't like (e.g. the massive overengineering of everything surrounding it), and I have no idea how easy it will be to generate static pages in a scripted fashion, i.e. using it without a real webserver. Or I might just write it all in python, which is a nice language for manipulating XML, usually. Please don't just send me an email with your favourite templating engine. Like zack, I'm only interested in XML-based templating engines, which does not apply to most templating solutions out there. Clearsilver for example is another bastardization of XML. I'm aware of TAL/METAL, and find them quite interesting, but they were also not having this kind of componentization that I'm usually thinking in. [Update: some people have pointed me to Genshi, which is mostly Kid compatible. However, it still has mostly the same problems, e.g. the templates being not reusable in other languages than python, and that certain constructs are a pain to do (e.g. the page_specific_css recipe with more than one css file). Others have pointed me to smarty, but it's string-based and doesn't ensure valid XML output. (Which is very useful for e.g. generating atom or rss output) For example this is probably valid in smarty: if 0 <b> /if </b> - allowing errors is bad. Oh, and smarty is PHP, which is broken by design, a no-go. The best match to my ideas so far is XML::Template (for Perl, Python, Ruby, PHP) which is pretty close to what I've been doing manually when not using a templating solution. I don't know yet how well it handles recursion - I need recursive templates for the navigation menu on my homepage.]

23 December 2006

Russell Coker: installing Debian Etch

A few days ago I installed Debian/Etch on my Thinkpad. One of the reasons for converting from Fedora to Debian is that I need to run Xen and Fedora doesn't support non-PAE machines with Xen. Ironically it's hardware supplied to me by Red Hat (Thinkpad T41p) that is lacks PAE support and forces me to switch to Debian. I thought about just buying a new dual-core 64bit laptop, but that seems a bit extravagant as my current machine works well for everything else.

Feeling adventurous I decided to use the graphical mode of the installer. I found it a little confusing, at each stage you can double-click on an item or click on the continue button to cause the action to be performed. The partitioning section was a little unclear too, but given that it has more features than any other partitioning system I've seen I wasn't too worried (options of creating a degraded RAID array and for inserting a LUKS encryption layer at any level are really nice). The option to take a screen-shot at any time was also a handy feature (I haven't yet inspected the PNG files to see what they look like).

Another nice feature was the way that the GUI restarts after a crash. While it was annoying that the GUI started crashing on me (and would have prevented a less experienced user from completing the install) the fact that it didn't entirely abort meant that I could work around the problem.

I have not yet filed any bug reports against the installer because I have not done a repeatable install (there is a limit to how much testing I will do on my most important machine). In the next few days I plan to do a few tests of the graphical installer on test hardware for the operations that are important to me and file appropriate bug reports. I encourage others to do the same, the graphical mode of the installer and the new encryption and RAID features are significant improvements to Debian and we want them to work well.

I have realised that it won't be possible to get SE Linux as good as I desire before the Etch release, even if the release is delayed again. I'm not sure how many fixes can go in after the release (I hope that we could move to a model similar to RHEL - but doubt that it will happen). So I now plan to maintain my own repository of Etch SE Linux packages and for other packages which need changes to make them work in the best possible manner with SE Linux. I will append something like ".se1" to the version of the packages in question, this means that they will be replaced if a security update is released for the official package. Apart from the SE Linux policy packages (for which any security updates will surely involve me) the changes I am going to make will not be major and will be of less importance than a security update.

I will also add other modified and new packages to my repository that increase the general security of Etch. Apart from SE Linux all the changes I intend to host will be minimal cost issues (IE they won't break things or increase the difficulty of sys-admin tasks), and the SE Linux related changes will not break anything on non-SE systems. So someone who wants general security improvements without using SE Linux might still find my repository useful.

Steve Kemp: It s time we all reach out 4 something new

Well I have a new computer which is running Debian Etch. The process was painful though. For a start I stupidly forgot to check that the motherboard had on-board graphics (hint: no) and so I didn’t order a graphics card. Sadly the motherboard also doesn’t have an AGP slot, so I couldn’t migrate a working card from elsewhere. Instead I had to go out and buy a PCI-express card from a local store. Once that was done I figured installing Etch would be simple. Not so. Update:: Tale of woe snipped. Apparently to netboot into the Etch installer I need to add:
nosmp,acpi=off
The “nosmp” worries me since it is a dual-core processor .. but once I’ve played around and upgraded the kernel I’ll see if removing that causes problems. Update #2: 64-bit host running Xen and X.org. Took me an hour to work out that the sound being “quiet” no matter how much I raised the volume control was due to me plugging in the speaker cable to the wrong socket and not the traditional alsa weirdness I was used to! Strangly even with USB enabled I cannot mount my external USB drive - so that is NFS shared for the moment.

Manoj Srivastava: Mail Filtering With CRM114: Part 1

The first step is to configure the CRM114 files, and these are now in pretty good shape as shipped with Debian. All that I needed to do was set a password, say that I'd use openssl base64 -d, and stick with the less verbose defaults (so no munging subjects, no saving all mail, no saving rejects, etc, since I have other mechanisms that do all that). The comments in the configuration files are plenty good enough. This part went off like a breeze; the results can be found here (most of the values are still the upstream default). The next step was to create new, empty .css files. I have noticed that creating more buckets makes crm114 perform better, so I go with larger than norm crm114 .css files. I have no idea if this is the right thing to do, but I make like Nike and just do it. At some point I'll ask on the crm114-general mailing list.
    % cssutil -b -r -S 4194000 spam.css
    % cssutil -b -r -S 4194000 nonspam.css

Now we have a blank slate; at this time the filter knows nothing, and is equally likely to call something Spam or non-Spam. We are now ready to learn. So, I girded my loins, and set about feeding my whole mail corpus to the filter:
 /usr/share/crm114/mailtrainer.crm               \
   --spam=/backup/classify/Done/Spam/           \
   --good=/backup/classify/Done/Ham/            \
   --repeat=100 --streak=35000                  \
         egrep -i '^ + train Excell Running'

And this failed spectacularly (see bug #399306). Faced with unexpected segment violations, and not being proficient in crm114's rather arcane syntax, I was forced to speculation: I assumed (as it turns out, incorrectly) that if if you throw too many changes at the crm114 database, things rapidly escalate out of control. I went o to postulate that as my mail corpus was gathered over a period of errors, the characteristic of Spam drifted over time, and what I consider Spam has also evolved. So, some parts of the early corpus are at variance with the more recent bits. Based on this assumption, I created a wrapper script which did what Clint has called training to exhaustion -- it iterated over the corpus several times, starting with a small and geometrically increasing chunk size. Given the premise I was working under, it does a good job of training crm114 on a localized window of Spam: it feeds chunks of the corpus to the trainer, with each successive chunk overlapping the previous and succeeding chunks, and ensuring that crm114 is happy at any given chunk of the corpus. Then it doubles the chunk size, and tries goes at it again. All very clever, and all pretty wrong. I also created another script to retrain crm114, which was less exhaustive than the previous one, but did incorporate any further drift in the nature of Spam. I no longer use these scripts; but I would like to record them for posterity as an indication of how far one can take an hypothesis. What it did to was have crm114 learn without segfaulting -- and show me that there was a problem in the corpus. I noticed that in some cases the trainer would find a pair of mail messages, classify them wrongly, and retrain and refute -- iteration after iteration, back and forth. I noticed this when I added the egrep filter above, and was not drowning in the needless chatter from the trainer. It turns out, I had very similar emails (sometimes, even the same email) in the Ham and the Spam corpus, and no wonder crm114 was throwing hissy fits. Having small chunks ensured that I had not too many such errors in any chunk;and crm114 did try to forget a mail differently classified in an older chunk and learn whatever this chunk was trying to teach it. The downside was that the count of the differences between ham nd Spam went down, and the similarities increased -- which meant that the filter was not as good at separating ham and Spam as it could have been. So my much vaunted mail corpus was far from clean -- over the years, I had misclassified mails, been wishy-washy and changed my mind about what was and was not Spam. I have a script that uses md5sums to find duplicate files in a directory, and found, to my horror, that there were scores of duplicates in the corpus. After eliminating outright duplicates, I started examining anything that showed up with an ER (error, refute) tag in the trainer output; on the second iteration of the training script these were likely to be misclassification. I spent days examining my corpus and cleaning it out; and was gratified to see the ratio of differences to similarities between ham and Spam css files climb from a shade under 3 to around 7.35. Next post we'll talk about lessons learned about training, and how a nominal work flow of training on errors and training when classifiers disagree can be set up.

Stefano Zacchiroli: looking for a website markup language

Which Website Markup Language? Dear Lazyweb, now that I've finished my Ph.D. thesis (i.e. now that I'm officially unemployed :-) ) it's time to renew my vetust homepage. That page is written with WML, I used to be quite happy with it, but this is no longer the case (lots of reasons for that snippet). I'm now looking for an XML based markup language for re-writing my new homepage, the features I would like to see in a tools based on it are: Of course I can design my own XML format and all the needed XSLT transformations, but I would like to avoid re-inventing the wheel if possible. Actually a while ago I think I read about a couple of tools like the one I'm looking for here on planet, but right now I'm unable to find the relevant posts. Similarly I'm apparently unable to find the right set of Google keywords for finding anything useful.

Enrico Zini: amis-character-list

Character list for the Amis language We mapped the available glyphs and accents for the Amis language. The letters in alphabetical order:
    a c d f n  h i k l m n o p r s t u w y
Everyone of them can get an acute, grave or circumflex accent on top. Literally on top: i would get the dot PLUS the accent on top. The only exception is for the letter n , which gets the dot on top of the g replaced by the accents. Some accent combinations are missing in Unicode, like accented d, f, h and k, and i with dot and accents. So we need an extension of Unicode. Then we need an input method that would insert n instead of g and allow to type all the accent combinations. Here is the full character set:
    a              
    c       ?      
    d       ?  ?  ?
    f       ?  ?  ?
    n       ? n  n  [1]
    h       ?  ?   
    i       ?  ?  ? [2]
    k       ?     ?
    l       ?     ?
    m       ?     ?
    n             ?
    o              
    p       ?     ?
    r       ?     ?
    s       ?      
    t       ?  ?  ?
    u              
    w              
    y              
[1] the gliph for 'n ' is actually missing, because it needs to be a single character: since it is a letter, backspace and delete need to erase all of it in a single shot. [2] missing because of the accent on top of the dot. Question marks mark where a character is currently missing in Unicode. We have a problem because Unicode would not want anymore latin accent combinations, so we will need to use character combinations and hope that fonts and applications display them well. This raises the issue that hitting delete or backspace on a character built using combining accents would delete only half of it. For consistency of interaction, it can make sense to use combining accents for all the combinations, even if an accented letter does exist in Unicode. This would avoid backspace and delete to have different behaviors, for example, on s-grave and s-acute. The accents are not taken into account when sorting. Uppercase letters are not used.

Uwe Hermann: Updated list of more than 230 Creative Commons licensed Christmas songs

Xmas gifts Looking at my webserver logs, there seems to be a highly increased demand for my 10 + 100 Creative Commons Christmas Songs blog post from last Christmas... I wonder why... ;-) So I decided to update the list this year, adding a bunch of additional songs; in total there are now more than 230 songs listed. Of course the list is incomplete, so any further links to Creative Commons licensed Christmas music are very welcome. Just leave a comment here or directly on the old blog post and I'll update the list. I'll probably also put up some more Christmas-related songs on my Creative Commons music podcast over the next few days... Image credit: Photo taken from Wikipedia. Author: Kelvin Kay. License: GNU Free Documentation License (GFDL), version 1.2.

22 December 2006

Evan Prodromou: 1 Niv se CCXV

I'm pretty happy about the way the 'Ten Web 2.0 APIs you can really use' episode of the LinuxCast came out so well. I'm not very used to talking for recording, and like everyone I don't particularly enjoy hearing my own voice recorded (do I really sound like that?), but it was fun to do, and I think it's a nice adjunct to the Ten Web 2.0 APIs you can really use article itself. tags:

Tijuana We're on a trip in Southern California this week, so we decided yesterday to go to Tijuana for the day. Maj, Amita June and I got out of the condo we're staying at in Encinitas early in the morning -- well, early for a vacation day -- and headed down to downtown San Diego. It's a trip of about 25 miles, with one tricky turnoff at highway 805 that seems to get me every time I visit this part of the world. Going into Mexico with a car can be a bit of a hassle. Many American insurers don't cover cars going into Mexico, and you'll often get asked about insurance when you cross over. There are some quick-and-dirty insurers on the border that can give you one-day insurance or other short-term coverage, but we have a rental car right now, and most (all?) rental car companies forbid with bitter, mouth-foaming forbiddance driving the car across the border. Fortunately for us, San Diego has a great way around this problem. Their trolley system goes from downtown San Diego right up to the border in San Ysidro. So, we picked it up in the Civic Center downtown, after a brief stop at the Downtown San Diego Vistors Center (to get tourist material), a shopping foray at a stall outside of the downtown Macy's (to get Amita June a pair of purple Crocs), and a side trip into the Westin for a bathroom break. Man, the Westin in San Diego has nice bathrooms! I'd heard about these trolleys in the past, and I guess I'd expected a kind of fancy tourist train, but it's really just the Blue Line for typical light-rail trains for San Diego and its suburbs. The seats were comfortable, but the train right-of-way went through some of the least scenic parts of Southern California I've ever seen. There's something about this corner of the United States of America that's depressing and end-of-the-world; I get the same feeling about other borderlands, like around Plattsburgh and Laredo. The end of the line for the trolley is completely opaque as to how get into Mexico. We wandered around past some ugly strip malls until Maj spotted a tiny oblique walkway that said "US PROPERTY SUBJECT TO SEARCH AND FINE" and in much smaller letters "a Mexico - to Mexico". We went through a wiggly walkway and across the no-man's land between the scary metal fence at the US border and the actual Mexican customs area. Nobody was there, but we got the feeling we were being watched. At the end of this weird walk was a very nice tourist info center, where we got some maps and brochures and such. Another 10 minutes walk through the streets up against the fence, we got to Avenida Revolucion, the central way through the Tijuana/Zona Centro. It had been a while since we'd been to a city with aggressive touts, so getting yelled at by people in shop doorways or in the street was a bit of a jolt. I hate the invasion of my anonymity, and I'm pretty reflexive about saying "NO NO NO NO NO" to everyone, but Maj is a little more practical. "If everything is run by aggressive sideshow barkers, you're going to miss out on everything," she points out. "Sometimes, you have to say 'yes', even if you don't like the way the question was asked." By the time we hit av. Revolution, we were pretty tired, so we stumbled into the first restaurant we found -- El Torito, a big nightclub with go-go cages, a mechanical bull, and a tattoo parlour on premises. I was attracted by the racy Absinth Mexico posters. "Family restaurant, family place," said the touts, seeing the baby in the stroller. Sure. The food wasn't bad -- I made the mistake of ordering the fish Vera Cruz, but the waiter made a smelly-face and suggested I get the shrimp instead -- and the cocktails at 1PM were generous and inexpensive. Probably the nicest part was a guy with a guitar who came over and asked to sing. "We've got enough music," said Maj, pointing at the ceiling where loud disco tunes were booming. But he was insistent, so we let him play, and he sang a loud Besa Me Mucho that drowned out the Ricky Martin and had us, the waiter, and nearby diners singing along by the end. Amita June was fascinated by the sounds, and clapped along when he was done. After lunch we walked down av. Revolucion, which had a dense layer of souvenir shops and discount pharmacies offering Viagra and Cipro without a prescription. One weird repeated theme was donkeys painted as zebras, hooked up to carts where you could sit and get your picture taken for $2. Some of the donkeys looked pretty good in their zebra stripes, but most looked pretty bummed out. I'm not sure why tourists would expect to get their picture taken with a zebra while in northern Mexico. It got a lot nicer as we got down to the end of the block, and we kicked ourselves for being so lazy when we passed Chiki Jai, a charming little restaurant with tiled walls and rough wood furniture. Much more to our tastes than El Torito. We stopped at the monumental Jai Alai courts, with a beautiful facade and a nice statue out front. Ducking into the Tia Juana's next door, we did a diaper-change break and had some chips and salsa and pineapple juice (and Dos Equis Dark on tap for Papa). Then we started walking back to the border. On our way back, we got off of Revolucion and walked over to the city's Cathedral. It was a much more realistic part of the town -- shopping and bars and restaurants, true, but less ridiculous touristy crap. When we got to the cathedral, there was a holiday street fair going outside, with games and food stalls and people laughing and walking around with kids. It felt a little more human than the pressure-tactics-dense world of Revolucion. On our way back to the border crossing, we dipped briefly into the red light district at the head of av. Constitucion (who knew?), then made the long slog back to the USA. We didn't get much in the way of souvenirs, but Maj bought a nice set of luchador dolls for her brother, complete with a tiny wrestling ring. The line at the border was huge, and we had to wait at the trolley station for another 30 minutes before one left for SD again. By the time we got back downtown, around 5:30, we really weren't up for a dinner in the San Diego/Gaslamp Quarter like we'd originally planned. So we piled into the car, paid the $20 parking to get out, and headed right into rush-hour traffic that led to the over-populated and under-networked North County (San Diego County). Would I take the trolley to Tijuana again? Maybe. Next time, I'd probably go to one of the stops between San Diego and San Ysidro, where there are free parking lots and a little less hassle. tags:

Hazards of vacation homes I was just rifling through the fridge here at the condo in Encinitas and found what I thought was a delicious Abita beer, the fine drink you have when you can't handle another hurricane in New Orleans. But it turned out to be an Abita Purple Haze, a gross raspberry-wheat beverage. Unfair! It has probably been stuck in the back of the fridge of this condo for years. Fortunately there was still another Stone Pale Ale, from the North County local microbrewery, to satisfy my lunchtime beer needs. tags:

Gustavo Franco: I can't get you off my mind

Dear Lenny,

Since Etch is closer, i would like to see you under my tree before the 07' xmas. I know it will be hard since tanks coming from many different ways will try to stop us from work in something we like a lot. Believe in me, you still have a lot of fans with you regardless what the press is writing about us and our work lately.

They've no idea what's really going on and will do their best to spread FUD. It sells, in a world where lies told many times turns out to be the 'truth'. The real one is that hopefully in a year and some months you will raise the bar again. They will still apt-get into it, trying to understand our acts, trying to explain our community. We can't, they shouldn't.

Sincerely,
Gustavo "stratus" Franco

Christian Perrier: End of l10n NMU campaign

So, today, I finally stopped the l10n NMU campaign. It started on Sept. 25th with the "Intent to Work" on hesiod. The last package I worked on has been isync today. The goal of that campaign was closing as many longstanding l10n bug reports as possible. The main goals were decided during the i18n work meeting in Extremadura, back in September. We focused on po-debconf translations, mostly, which are an easy target for NMUs. Thomas Huriaux extracted the data from the BTS and ranked packages with regards to the number of pending l10n bugs and their age. Lucas Wall reactivated a coordination page which we already used back in early 2006 when a first version of such a campaign was run. During that campaign, I worked on 102 packages. 89 of them got all their pending l10n bugs closed, 5 of them got most of their l10n bugs closed (remaining "l10n" bugs were either not really related to localisation and strange things which I left aside), 8 packages were ignored either because their respective maintainers wanted to handle the bugs themselves or because they were tagged for removal. Meny packages have also been handled by their respective maintainers, either because the NMU announcements woke them up...or because of their natural release process. I have to point out that the level of collaboration of package maintainers has been great (except for those who are MIA, and there are quite a few) with only one very proeminent exception (just guess which one). Thijs Kinkhorst and Tobias Toedter also contributed to the campaign. Thanks to them as well. As a side effect, we also made many of these packages as lintian-clean as we could, trying to balance between too invasive NMUs and trivial fixes. IIRC, I only introduced 3 or 4 RC bugs doing so (which I fixed immediately of course). A few RC bugs were also fixed by the NMUs and even a security bug has been hunted down (for screen). Meny thanks to all package maintainers, translators and everybody who helped during that work. It has been a great time. Just look at the result. Now heading to the improvement of the next pet project of the i18n task force: the Debian i18n server.

Next.

Previous.