Search Results: "reg"

13 January 2008

Gregor Herrmann: felicidad

after neglecting her for over four months I took "felicidad" (my western guitar) out of her case tonight. at the beginning of today's date she was quite "desafinado" but after a few minutes of tuning we started to have a nice session.

& it's amazing that my fingers (although they have lost the thick skin on the fingertips) still remember more or less what to do with the strings.

Piotr Roszatycki: FL Studio 7 on WINE

Some time ago I wrote a note about VSTi on Linux. Today I tried to run FL Studio with WINE and... it works almost ;) perfectly! I'm really impressed. Now almost all VSTi plugins works seamlessly and even external plugins can be used, i.e. my favourite Superwave P8 :) I think it might be an issue with registration of full version copy but at least there are some cracks working. It's good to see the real audio software run on Linux. WINE developers, you've done an amazing job. Thank you!

12 January 2008

Aigars Mahinovs: Having a (pyro)blast with your new Ice Block

Here is something I wrote some while ago, hope this will be useful for some World of Warcraft p(l)ayers. Has no other relation to Linux or Debian, except it works on it flawlessly. Also note that 2.3.2 patch went live last week. It is not a secret anymore that mages of all talent specs will be getting a present when patch 2.3.2 hits the live realms - the Ice Block will be available from any friendly mage trainer for a nominal fee. Currently this spell requires a whopping 21 point investment into the Frost talent tree. With patch 2.3.2 Ice Block will become trainable to all mages, Ice Barrier will take it’s place in the talent tree and a new awesome talent - Icy Veins will take the place of the Ice Barrier, so that even frost mages have a new toy to gain in this 2.3.2 mage winterly fiesta. While there is a significant proportion of mages that are frost for PVP or grinding survival purposes, a large proportion of mage population have never used Ice Block or haven’t used it for a while. For that reason, the rest of this article (after the break) will explore all the great uses of Ice Block, that all of you can look forward when patch 2.3.2 hits the live realms (or sooner, if this article convinces you to re-spec frost :)). So, what does the Ice Block do? When you cast the Ice Block, it purges you from all debuffs and releases you from all snares, you get frozen solid into a large block of ice (same look as hunter’s ice trap) and for 10 seconds you are immune to all damage, but unable to do anything - no moving, casting anything, no bandages, nothing. Getting rid of: Ice Block is a buff, so you can get out of the Ice Block by right-clicking it off at any time. Another way for keyboard-addicted mages is to press Ice Block keybinding button again (after the global cooldown is done). Enemy shamans can purge it off you and priests can mass dispel your Ice Block in PVP. Cooldown: the Ice Block is on 5 minute cooldown, which can be ended early using the Cold Snap talent from the mage frost tree. To prevent mages casting two ice blocks back to back by using the Cold Snap, Blizzard added the Hypothermia debuff (in 2.1) that prevents you from casting the Ice Block 30 seconds after the last cast. The main use of Ice Block is to survive pulling aggro in groups. While we all know that “good mages don’t draw aggro”, experienced players can easily show situations when drawing aggro is not mages fault and surviving that can be the difference between a wipe and a win. When Ice Block is used, your threat is NOT reduced, so you need to stay inside the Ice Block at least until you are under 130% of the tanks threat. After you Ice Block, the mob will run to the next person on the threat list. If your tank is dead, Ice Block will send the mob to another DPS or healer giving you a couple seconds to nuke it when it will be running back to you again. I have seen a few bosses loose their last 2-3% of health running between mages with Ice Block and hunters with Feign Death after a unfortunate death of the main tank. Ice Block is a great way to get rid of almost all debuffs that exist in the game: warlock DoTs, stuns, snares, polymorph, mind control … Even boss debuffs go away in a snap - Garrote from Moroes, Demonic Chains from Illhoof and many more deadly debuffs are no threat to a mage with an Ice Block ready. Ice Block can be used as an alternative to Slow Fall if you’re out of Light Feathers. Just do not cast it too high in the air - if the 10 seconds tick by and you haven’t hit the land yet .. you could as well kiss 10% of your durability goodbye. When solo grinding there is noone to draw aggro from you, so the mobs will continue to beat on your personal icy sculpture, however during that time your Water Elemental or other battle pets could be dishing some damage, your cooldowns will tick down and also you will also regenerate some mana inside the Ice Block. The Ice Block could thus make the difference between your death and Frost Nova+Blink+Blizzard and their death. Have you had some fun experience with Ice Block? Leave a comment so that others could repeat it too!

11 January 2008

Steve Kemp: I still got the blues for you

Been a week since I posted. I've not done much, though I did complete most of the migration of my planet-searching code to gluck.debian.org. This is now logging to a local SQLite database, and available online. I've updated the blog software so that I can restrict comments to posts made within the past N days - which has helped with spam. My other comment-spam system is the use of the crm114 mail filter. I have a separate database now for comments (distinct from that I use for email), and after a training on previous comments all is good. Other than being a little busy over the past week life is good. Especially when I got to tell a recruitment agent that I didn't consider London to be within "Edinburgh & Surrounding Region". Muppets. The biggest downside of the week was "discovering" a security problem in Java, which had been reported in 2003 and is still unfixed. Grr. (CVE-2003-1156 for those playing along at home). Heres the code:
#!/bin/sh
#
#  Grep for potentially unsafe /tmp usage in shared libraries
#
find /lib -name '*.so' -type f -print > /tmp/$$
find /usr/lib -name '*.so' -type f -print >> /tmp/$$
for i in $(cat /tmp/$$ ); do
    out=$(strings $i   grep '/tmp')    
    if [ ! -z "$out" ]; then
        echo "$i"
        echo "$out"
    fi
done
rm /tmp/$$

Felipe Augusto van de Wiel: 11 Jan 2008

Bits from the Debian i18n meeting (Extremadura 2007)
From December 12th to December 15th, Junta de Extremadura hosted another one of the Debian Meetings; five i18n guys shared ideas, food, buses and fun with the Debian KDE maintainers. We would like to thank Extremadura for hosting us during the Hispalinux Meeting 2007, the event was held at Universad de Derecho (Law University) in Caceres, Spain.

These are the minutes, results and notes from our work, it is a brief description but hopefully complete of what we have done and what is still missing/pending.

Thanks to Cesar (cek) we had the chance to work on churro (i18n.debian.net) locally; the server is still running a 2.4 kernel because of some "tick" problems with 2.6 series, the last one tried was 2.6.21 and we should try newer ones, in order to support upgrades and not get stuck with 2.4, we hope Cesar will find time to test new Debian kernels.

First, let me introduce everybody to the services, robots and resources being hosted by i18n.d.n:

  • MoinMoin wiki for local and simple reference documentation, it contains all the links to the below resources. (http://i18n.debian.net/wiki/)
  • Pootle experimental server
  • dl10n scripts, aka dl10n robots (codename Lion), these scripts are responsible for the status of pseudo URLs used by some translation teams, by the Project Smith and by the NMU Priority List for i18n NMU Campaign
  • Synchronization of the i18n material used by the Debian website to generate translation statistics about PO and PO-debconf
  • Generation of Compendium PO files per-language
  • Different types of statistics
  • Other non user-visible services like a full source mirror for stable, testing, unstable and experimental, used by the scripts and robots.
  • DDTP, Debian Descriptions Translation Project
  • DDTSS, The Debian Distributed Translation Server Satellite, a web front-end for DDTP, now integrated to DDTP to use the Database back-end instead of the e-mail interface.


And, at some point, we found important to state clear the acronyms and names used in related DDTP projects/tools:
  • DDTS, Debian Description Translation Server, this is the main "back-end" used in DDTP, it tends to be the interface between translator tools (present and future ones) and the database;
  • ddt.cgi is a CGI interface that is able to provide info for specific packages or translations, including diffs, related packages and active/inactive descriptions.
  • DDTC which is the old (and still functional) command line client for DDTP.


We took the chance to organize a few things on churro, old accounts were cleaned out and removed, we moved from /org to /srv and got more GBs of space to the "playground". Old files were also removed and some are schedule to deletion on early 2008. With the reallocation of /org we also find some more space to /home and /var, we reorganize some of the links on the web space (specially to remove services from people's accounts), and we changed the mirror script to also synchronize the Packages and Contents files.

Grisu and Martijn worked mainly on DDTP and DDTSS integration. DDTSS now provides statistics for stable, testing and unstable, we are also working with Debian Med to provide support and infrastructure to a specific audience, like packages related to Medicine. The conversion to talk directly with DDTP/DDTS database also provided:

  • Fetching new translations is almost instantaneous and marks translation as requested (avoiding duplicated works via the e-mail interface).
  • After sufficient reviews occurred, the upload is instant
  • Committed DDTS / DDTSS / DDTP website generation into SVN
    • Added READMEs for the above directories


DDTSS now announces the user using authentication because of its integration with the Database backend used by DDTP. Quick trivia: DDTP is now a compound of 25 languages occupying 18 GBytes.

A few days before the meeting we had the offer to use "AUTOBYHAND" to upload a package with the Translation-* files. The package is now called 'ddtp-translations' and we worked during the meeting to create scripts to build the package and to test it on the archive side. This approach allow Debian i18n Team to upload new translations and remove old ones (or inactive ones) without bother FTP Master Team. Special thanks to Anthony Town, he has been working with us to prove tips, fixes and info on how to produce the package and the scripts. The code is available in the debian-l10n SVN under pkg-ddtp-translations:
http://svn.debian.org/viewsvn/debian-l10n/pkg-ddtp-translations

In our case, "BYHAND" processing consists of a simple tarball of the main,contrib,non-free /i18n/Translation-*, we decide to work on a set of scripts to make it easier to create new packages (ddtp-translations) in a consistent way and keeping debian/changelog up-to-date. We also made some suggestions to the script what will run on the archive side to check the tarball structure, base on the examples of debian-maintainers and debtags (tags-override).

One of our initial targets for the meeting with regards to Pootle and Debian was to try big PO files per language, fortunately, Nicolas and Friedel were able to increase Pootle performance enough to get a few languages from DDTP loaded in Pootle. Using the upstream Pootle-diet branch, which uses a database back-end for the generation of statistics, the time to browse the DDTP POs of a language (~20.000 files) went down to a dozen of seconds.

Speaking about Pootle, Friedel gave us a good picture of what is coming next in terms of Pootle's development. There are improvements planned in the areas of permissions and rights delegation, as well as file management (for projects and templates). Improved management of terminology projects is also planned.

Improvements in the QA capabilities of the translate toolkit and Pootle are planned to help with the "false positives" of the pofilter checks. Better reuse of existing translations will become possible by using better translation memory techniques. There is also work planned on formats and converters involving, for example, XLIFF, TMX, TRADOS and WordFast.

Another pending task for quite a while was the CVS migration to SVN, it is now done, with a new layout. Commits to the CVS were disabled and every single script or resource depending on CVS should be changed to use SVN. For now, we are publishing (via HTTP) the status files generated by the pseudo-urls robots until we can fix the scripts to re-enable the commit of the files. You can find them here: http://i18n.debian.net/debian-l10n/status/

We are pretty happy with the changes and results of the work during those days, but we still have some items pending on our TODO list:

  • More advertisement and usage information about PO Compendiums
    There are two use cases are identified:
    • Filling new PO files.
    • QA work to find inconsistent translations.
    Maybe Eddy would love to do that? :-)
  • Extend the duration of the statistics history. (Nekral)
  • Debian packages of the services running on churro
    • DDTP (Grisu)
    • DDTSS (Martijn)
    • dl10n (Nekral)
  • DDTP: add some scripts to handle packages with version in the description (e.g. kernel and kernel modules) (Martijn)
  • DDTP: Standard generation of the translation tarballs (faw)
  • DDTP: document the bracketed stats on the main page (faw)
  • DDTC: should be updated to match the current features. Documentation to ease integration with procmail. (Nekral - low priority)
  • Implement mail service for translation teams with their own robots (e.g. Dutch) (faw)
  • Collect data from http://www.debian.org/devel/website/stats/ (Nekral)
  • http://www.debian.org/intl/l10n/po/,
    http://www.debian.org/intl/l10n/po-debconf/
    are built based on the churro material. It would make more sense to build these statistics on churro (Nekral)
    • We could "fork" the page and add some fancy new features on these pages (Nekral)
    • Add information from the coordination page to indicate that a translation is ongoing. (Nekral)
  • Pootle: missing review indication. Hard with PO back-end. (Friedel)


There are a couple more reports to be sent but they are more focused on i18n specific questions, tools and plans for 2008. So, probably those will be sent only to debian-i18n mail list. If you are interested, please, stay tuned. :-)
Posted on d-d-a: http://lists.debian.org/debian-devel-announce/2008/01/msg00002.html
And a big thanks to Nicolas François (aka nekral), he helped me a lot making notes, preparing the text and reviewing it; and was patient enough to wait for the report while I was solving some personal problems.

Anthony Towns: LCA Sponsors

An article by Sam Varghese appeared on ITwire today, entitled linux.conf.au: What is Novell doing here?:
A GNU/Linux system does not normally load modules that are not released under an approved licence. So why should Australia’s national Linux conference take on board a sponsor who engages in practices that are at odds with the community? What am I talking about? A company which should not be in the picture has poked its nose in as a sponsor. Novell, which indicated the level of its commitment to FOSS by signing a deal with Microsoft in November 2006, will be one of the supporting sponsors for the conference.
Novell was also a minor sponsor of the 2007 conference, and Sam wrote an article in January expressing similar thoughts, which included this quote from Bruce Perens:
“I’d rather they hadn’t accepted a Novell sponsorship. It wasn’t very clueful of them, given Novell’s recent collaboration with Microsoft in spreading fear and doubt about Linux and software patents,” Perens said.
Ultimately, I think that’s a mistaken view. Linux.conf.au is what it is thanks to the contributions of four groups:
the organisers
who create the conference, get a venue, organise a schedule of events, help the speakers and attendees to get there, and generally make it easy for everyone to just get immersed in cool Linux stuff
the speakers
who provide the core of the schedule, the reason for attendees to go, and a core depth of awesome technical knowledge and ideas
the attendees
who fill in the organisational/content gaps that the organisers and speakers miss, who make for fascinating corridor and dinner conversations, who make side events like the miniconfs, the hackfest or open day interesting, and who pay the rego fees that lets the conference happen
the sponsors
who provide a chunk of money to fill out the conference budget letting us commit to venues and events earlier (when we might otherwise have to wait to see how many people come), and let us do extra things that registration fees alone wouldn’t cover
Obviously sometimes you have to exclude people from participating, but that’s mostly only if they’re actually causing trouble for the event. For sponsors, that pretty much means trying to interfere in the conference itself, or not paying on time. Otherwise, if you’re contributing to the conference, and not causing problems, you certainly should be recognised for that, as far as I can see. For me, the same thing would apply if Microsoft was offering to sponsor the conference – if they’re willing to contribute, and not cause problems, I’m all for it. If they happen to not be doing anything constructive in Linux-space anywhere else, well, it seems perfectly fine to me to start contributing by helping make linux.conf.au awesome. In Microsoft’s case that would be hard, because all the people going “oh my gosh, Microsoft, Linux! Wolves, sheeps! Hell, snow!” along with possible mixed messages from Microsoft and our long-term major sponsors HP and IBM about the future of Linux and whatnot could really distract us from all the cool technical stuff the conference is fundamentally about. I don’t think there’s anything Microsoft could offer to justify that much disruption, but having more of the world’s software companies involved in free software would probably be worth a bit of hassle, if the disruption could be minimised. Ultimately, I guess my disagreement comes down to these couple of comments from Sam’s article:
Asked whether it was right that Novell should be allowed to be a sponsor for a conference such as this - which, in my view, is a privilege - […] […] Novell, obviously, is hoping that, as public memory is woefully short, it will be able to wriggle its way back into the community. Providing such leeway is, in my opinion, a big mistake.
In my opinion, the ability to contribute to open source isn’t a privelege, it’s something that should be open to everyone, including people who’ve made mistakes in the past: and that’s precisely what the “free” in free software is all about. OTOH, if you want to see who’s been participating most in the Linux world lately, you’re much better off looking at the list of speakers than sponsors. Novell (or at least SuSE) folks giving talks in the main conference this year seem to include John Johansen and Nick Piggin. Interestingly, the count of HP folks seems a bit low this year, with only two that I can see, which leaves them not only merely equalling Novell/SuSE, but beaten by both Intel and Catalyst. Tsk! I guess we’ll have to wait and see if that changes when we can see the list of attendees’ companies in the booklet this year…

10 January 2008

Alexander Schmehl: About Apache, Basic Authentication, Location and regular expressions

The other day, I had an interesting problem, regarding Apache, the LocationMatch directive and regular expressions. I'll put it here, so I'll be able to find it again... Actually it was a quite simple problem: A simple web page, where all web pages are protected with a password. No big Problem: Use a <location /> and AuthType basic, problem solved.... or isn't it? Some documents might be needed to be available without the authentication... for example robots.txt... or the funky custom error document, which explains how to get the authentication data, in case the authentication fails. Looking at the documentation, found a solution: LocationMatch exists, where you specify a regular expression, which must match, for the following configuration to apply. What isn't mentioned in the documentation, that the obvious solution... something like <LocationMatch !"^/(robots.txt my401handler.html"> doesn't work. See apache bug 10932. Needed quite some time, to find this out... Luckily Meike found the bug report above, which has a workaround for that problem: Use <LocationMatch "^/(?!robots.txt my401handler.html)"> Oh... and if your customized error handler references some pictures or style sheets, don't forget to add them to the regular expression above, too.

9 January 2008

David Welton: Nefarious Network Solutions

  1. Go to http://www.networksolutions.com/
  2. Type in a silly domain name (not something you actually care about!).
  3. In a few minutes, they will have registered that domain themselves, locking you into buying it from them if you actually want it.
They're overpriced anyway, but that's really sleazy.

7 January 2008

David Nusinow: Finally

A few years back I asked Branden Robinson for access to the X Strike Force SVN repository in order to improve the X server's autodetection, with the goal of stealing what I could from knoppix. At the time, users were constantly wandering in to #debian saying that they had used knoppix to create an XF86Config-4 and then they used that file on their Debian installations. They were also constantly whining about how Debian wasn't doing as good a job. So I decided to do something about it, and took a look at what knoppix was actually doing so much better than us, and I was surprised to find that they pretty much just wrote a skeleton configuration file and let the X server fill in the details. I had no idea the server was capable of such things. So Branden graciously gave me SVN access and I began comparing the knoppix method to the script called by "dpkg-reconfigure xserver-xfree86" and realized that we probably couldn't adopt the same method because we had tons of checks for portability. So I put the problem on the backburner for few weeks and worked on something else for a while.

Well, somewhere in between then and now I got sucked in to transitioning Debian to X.org (using the aforementioned SVN access) and then working on all the things that went along with maintaining X in Debian, some well and others less so. Between transitioning to Xorg, and then transitioning to a modular Xorg, even with the ability to steal Ubuntu's packages it still took about two full years with me being a rookie and all that. There were tons of people who worked on this with me, but it was just a damn big job. Eventually though, etch released with modular X packages, and we were running at a pretty good pace with upstream, so it was time to revisit the problem of configuration again after the two year detour.

The problem of configuration had been reframed for me by having looked at what knoppix was doing and discussions with upstream. Upstream was starting to come to the conclusion that we shouldn't have a config file at all, and that the server should be smart enough to do everything by itself. It was already partially there, it just needed a push in the right direction. This was a major shift in how I thought of it. Coming from a Debian background, where the answer is to always just regenerate or edit your config file, having the server work things out for you was a totally alien idea. But I have my deepest roots in the Macintosh world, so immediately fell in love with it. The problem was that the server had a whole body of code to use if you had no config file at all, and another, with far less automagic goodness, if you had a config file, even if it was a 0 byte file. The goal became to have the server work really well with a minimal config file, so you could override what you don't like in the defaults and let the server figure things out for itself at boot. The way forward was to translate as much of the logic in our configure script in to the X server itself as possible.

Ubuntu had put in a lot of work in to the configuration setup that we automatically benefitted from, so given that most users were happy with things as they were, I was able to carve away at the problem without disrupting anything. And there was a lot to carve out, as the script that runs the config is an absolute mess that was slated by Branden for a rewrite all those years ago. Early on I picked off the low hanging fruits like the font path and modules. Similarly, Redhat's Adam Jackson had also been working on this problem, and he killed off the ServerLayout section as well as putting in lots of critical fixes all over the place elsewhere. More recently, I've gone and cut explicit modesetting out of the configure script. Hardcoding this information is generally a bad idea in the randr 1.2 world, and most of the drivers will do as good or better job of figuring out the modes than our config script would. This let us jettison using the xresprobe program to ask the monitor for the settings to use. This cut out a lot of the code that we had to deal with, simplifying the configure script and letting us all benefit from upstream's work. This leaves a big gaping hole in user configuration, which is something I'm looking to address in a few weeks, but for now there are workarounds like editing your xorg.conf manually to make things work.

Finally, yesterday I was able to upload a version of the script that no longer uses the discover program to figure out what driver to load. I've patched the server to do this at runtime. If there's no driver listed it simply scans the PCI bus, picks out your primary video card, and loads the first driver that claims to support that PCI ID. This let us jettison the last external dependency that the configure script had, so now we have a relatively small chunk of shell script with no external C code and a simplified setup. At this point we're now shipping a more skeletal config file than knoppix ever did simply because it wasn't possible to ship something like this before and have the server work at all. This is a huge milestone for me because finally, after over three years, the problem I originally set out to work on with X is done. There's still bugs to uncover and fix with all of this, but I'm convinced that what we have now is superior to our old method. Eventually, with any luck, xorg.conf will just fade away for most people. There's a lot to work on before that happens though, but I'm happy to have finally gotten here.

Adam Rosi-Kessel: Political Blogging

The Frontal Cortex provides an astute explanation as to why he shies away from political blogging in this election season:
My hypothesis is that political judgments are like moral judgments. When you see a candidate, you experience a visceral, instinctive, inexplicable response. Your brain generates an emotion - Obama is uplifting, Hillary is commanding, McCain is honorable, etc. - and then the rest of your brain goes about explaining your emotion. The inner interpreter gathers together bits of evidence, post hoc justifications, and pithy rhetoric in order to make our automatic reaction seem reasonable. But this reasonableness is just a fa ade, an elaborate self-delusion.
Personally, I haven’t done much (any?) such blogging lately for lack of time, but I also generally agree with his conclusions. “A man hears what he wants to hear and disregards the rest.” Technorati Tags: , ,

Peter Eisentraut: Problems with newer kernels

It has become apparent to me that the Linux kernels in Debian testing and unstable (2.6.2x) have all kinds of problems compared to the one in stable (2.6.18), especially when it comes to working with virtualization. I have been trouble booting newer kernels as guest systems in VirtualBox (bug #434723) and QEMU (and bug #449085). There is some chatter in the Gentoo forums to the same effect. I have also had trouble compiling the host system kernel modules of VMPlayer and VirtualBox with newer kernels. If your job requires regular juggling of virtual machines and operating systems, this creates big problems. Plus my UMTS card stopped working (bug #433750), also confirmed by Gentoo chatter. :) I have also tried to build pure upstream kernels myself, but they have the same problems. So my advice to those running Debian post-stable is to stick to kernel 2.6.18 from stable if you are seeing funny issues.

6 January 2008

Andrew Pollock: [life] Post-op, Day 3

Slow day today. I slept at home, and got in at around 9:40am. Sarah didn't sleep well again overnight, her O2 saturation levels kept dropping below 90% because she wasn't breathing deeply enough in her sleep, and that causes an alarm to go off. So she was pretty flat when I first got in, but she picked up as the day went along, and was fairly chirpy by around 11am. The doctor came by and explained the photos that we've been given, so I'll have to try and regurgitate all of it and update the captions. Her lungs are still looking a bit suboptimal. The third lobe on the right lung isn't inflating well apparently. They said something about there being some fluid as well, although the nurse said she can't hear anything. All that said, she spent a lot of time in a chair today, and her O2 saturation levels have been the best they've been since she was in the ICU. So good, that they're trialing having her off oxygen while she attempts to take a nap as I write. Her blood sugar levels are settling down. I think they're still slightly elevated, but not to the point that insulin is required. She only needed insulin with lunch yesterday, so it definitely looks like it's improving. She's already been on her three walks today, by 4pm. An occupational therapist came by, and went through the sternal precautions again, and showed us how to get in and out of bed without putting undue pressure on the sternum, and Sarah's been getting out of bed unassisted from horizontal since. Her pain has been quite manageable. So while today started out looking like it was going to be a small backward step, I think it's still a good move forward. Sarah's continuing her breathing exercises, so I'm sure her lungs will sort themselves out in the next day or two. I guess they'll take another chest x-ray this afternoon, although it's getting a bit late in the day. Sarah's starting to get a bit of cabin fever, but I think today's been pretty slow, in part because we deferred having visitors on account of her being not up to it in the morning.

Christoph Berg: mutt-patched

There have been several long-term wishlist bugs on Mutt to include the (in?)famous "sidebar" patch. It adds a panel on the left side of the screen to list mailboxes with message counts. We do ship several patches with Debian's Mutt package that are not considered for inclusion by the upstream authors, but this one is different. It touches the core of the UI renderer and adds extensive counting of messages. There's been quite a hype about the patch, it was even included in the (now discontinued [1]) mutt-ng fork. Unfortunately, the patch was technically questionable, it was written for the Mutt tarballs and didn't easily apply to CVS, and most versions floating around contained lots of cruft like backup files and temporary editor files. Furthermore, it is far from bug-free and even segfaults occasionally. Some months ago we decided we would build a second binary package to put the patch in so we don't destabilize the standard package. Thanks Dann Frazier, we got a clean patch, Adeodato had the right idea on how to build two binaries from the same, differently-patched source (make install would run havoc, the trick was to build the patched version before the regular one), and I resolved some more involved conflicts with the maildir-mtime patch. So, finally, there it is: mutt-patched. Disclaimer: As said, there are bugs. YMMV. On related news, there's now also a -dbg package now for the victims of the IMAP and header cache segfaults that we still see sometimes. Update: The sidebar patch was not the reason for the creation of mutt-ng, but many mutt-ng users were using it because of this patch. Update 2008-01-08: [1] mutt-ng used to be a Mutt fork. It is now maintained as a patch collection for Mutt.

5 January 2008

Dirk Eddelbuettel: RQuantLib 0.2.8

Version 0.9.0 of QuantLib was released a couple of days ago on Christmas Eve; as usual, Debian packages for QuantLib were uploaded right away following a few earlier pre-releases. As QuantLib is approaching its 1.0 release, a few API changes requires updates to basically all of RQuantLib's C++ source files. Luckily most changes were minor. At the same time, we also generalised the Binary (aks Digital) option pricer to allow for a 'binType' argument (with values 'cash', 'asset' or 'gap' for CashOrNothing, AssetOrNothing or Gap digitals) as well as an 'excType' argument to switch between European and American exercise. Dominick made a small change to the DiscountCurve object to seamlessly pass a switch variable indicating whether we have 'flat' curves or not. Another change was the addition of formal unit tests using the RUnit from CRAN (which we happened to have added to Debian recently in the wave of new RMetrics packages). We use the scheme initially proposed by Gregor Gojanc and extended by Martin Maechler for RMetrics that allows the unit tests to be run from both the source and the installed package which is nice. As QuantLib itself has a massive amount of unit tests in its code; I am hoping to add more and more of those into RQuantLib itself as we add more functionality. On that front, more exciting news: RQuantLib is now hosted on R-Forge. Potential contributors are encouraged to register at R-Forge and to get in touch -- this is a great way to learn about combining C++ and R. To wrap up, the new version 0.2.8 is currently in the queue at R's master CRAN host and should hit the CRAN mirrors shortly; likewise the Debian package has been uploaded and should also propagate to Debian mirrors in due course. As usual, source are also available locally on my site.

4 January 2008

Wouter Verhelst: Broken C code

A few weeks back, someone asked on debian-68k about an unexpected result of some piece of software when it was compiled on ARAnyM, an m68k emulator that can run Linux. His initial idea was that this could perhaps be a bug in ARAnyM, since it only occurred to him inside ARAnyM, and not on any of the other architectures he tried. In fact, it was a bug in his code. Let's have a look at what's going on here. The example program which Sergei provided and which exhibits the problematic behaviour is spread over two files. The first file contains this:
#include <stdio.h>
void *f();
main()  
	void *a;
	a = f();
	printf("%d\n", (long)a);
 
while the second contains this:
#include <stdio.h>
long f();
long f()  
	long a = 84;
	printf("%d\n", a);
	return a;
 
Many people familiar with the C programming language will intuitively expect that this program, when run, will output the following:
84
84
When in fact, when compiled and ran on an m68k machine, the output is as follows:
84
-1072577264
What's going on here? In order to understand that, we have to take a look at the assembly code. This is done by use of "objdump -d", and gives (amongst others) the following output:
800004c4 <main>:
800004c4:	4e56 0000	linkw %fp,#0
800004c8:	61ff 0000 001a	bsrl 800004e4 <f>
800004ce:	2f08		movel %a0,%sp@-
800004d0:	4879 8000 05aa	pea 800005aa <_IO_stdin_used+0x4>
800004d6:	61ff ffff feec	bsrl 800003c4 <printf@plt>
800004dc:	508f		addql #8,%sp
800004de:	4e5e		unlk %fp
800004e0:	4e75		rts
800004e2:	4e75		rts
800004e4 <f>:
800004e4:	4e56 0000	linkw %fp,#0
800004e8:	4878 0054	pea 54 <_init-0x800002f0>
800004ec:	4879 8000 05aa	pea 800005aa <_IO_stdin_used+0x4>
800004f2:	61ff ffff fed0	bsrl 800003c4 <printf@plt>
800004f8:	7054		moveq #84,%d0
800004fa:	4e5e		unlk %fp
800004fc:	4e75		rts
800004fe:	4e75		rts
So what's happening here?
  • main acquires some stack space, and then (0x800004c8) immediately jumps to a subroutine 26 bytes ahead; our function f.
  • f also acquires stack space (0x800004e4), then pushes two addresses on the stack (...4e8 and 4ec), and jumps to the printf function. The first of these two addresses is the constant 84 that is hardcoded in the binary; the second address is our format string.
  • f then moves the constant value 84 to the D0 register (...4f8), frees its stack space (4fa), and returns (4fc; the second rts can be ignored, that's a harmless quirk in the compiler).
  • main now copies whatever is in the A0 register to the stack (4ce), pushes the exact same format string to the stack (4d0), and again jumps to the printf function (4d6; the difference in hexadecimal values is due to the fact that the bsr opcode uses processor-indirect addressing).
  • Finally, main clears up what it's been doing, and exits
The difference should be clear: the f function stores its result value in the D0 register, but main goes looking for it in A0. The reason for this discrepancy very simple. The m68k processors has three sets of registers: one set is for integer values (D0 through D7), one set is for address values (A0 through A7), and one is for floating-point values (FP0 through FP7). The m68k ABI specifies that integer return values should be stored in integer registers, that address return values should be stored in address registers, and that floating-point return values should be stored in floating-point registers. This makes sense, since register-indirect addressing modes require the address to be stored in an address register; and calculating values requires the value to be stored in either a floating-point or integer register. When main, then, tried to look for a return value in A0, it found something, but obviously not what it should have found... This issue would have been a non-issue had this function been written the way it should have, like so:
#include <stdio.h>
void* f();
int main()  
	long* a;
	a=(long*)f();
	printf("%Ld\n", *a);
	return 0;
 
void* f()  
	static long a=84;
	printf("%Ld\n", a);
	return &a;
 
To close up: there's a reason why compilers emit warnings if you do something strange or "clever". In this particular case, the warning was suppressed, since both files contained a declaration of an f() function, even if the declaration was different. This is a horrible hack that tries to work around those warnings, rendering them utterly useless. Please, pretty please, with sugar on top: don't do something like that. If you write C code, please compile it with -Wall -Werror, and make sure it compiles that way. If you want to access a function in a different file, create a common header file that both will #include, so that the compiler can notice differences between declaration and definition of a function; and don't expect that something will work because you fixed it for a common and well-known case, because there will often still be other places where your bug will still trigger. In this case, by declaring the function as one returning a long value, they made sure it could not break on 64-bit architectures. However, as shown above, that doesn't make the bug go away...) Writing clean and portable code is way more fun. Trust me.

Benjamin Mako Hill: Laptop Liberation in Nara

I'm going to be giving a reprise of the Laptop Liberation talk I gave at Cornell University in November at the Nara Institute of Science and Technology in the Kansai region of Japan on January 7th. If you around, please feel free to show up. If you are in Osaka, Kyoto or Nara and would like to have lunch or dinner, please email me and we can try to arrange something. Details on the talk is online here in English and Japanese (thanks Mika!) although the talk itself will be in English.

Torsten Werner: Open Source licensed scientific software



Software becomes more and more important in science as in other areas of life. Scientist have a tradition to publish their work very openly but that does often not include the source code of the software that was developed to carry out simulations which has some obvious problems such as:
- other scientists cannot check the software for errors,
- other scientists cannot fix the bugs and easily reproduce the results,
- other scientists cannot base their new research on already existing software and have to write it completely from scratch again and again,
- software package from different authors cannot be combined easily.

But things are getting better. One field of scientific research where we can see some improvement is machine learning - http://en.wikipedia.org/wiki/Machine_learning which is a broad subfield of artificial intelligence and concerned with the design and development of algorithms and techniques that allow computers to "learn". S ren Sonnenburg et.al. wrote a paper about "The Need for Open Source Software in Machine Learning" which is available at http://jmlr.csail.mit.edu/papers/v8/sonnenburg07a.html. They even created a portal with the goal to support a community creating a comprehensive open source machine learning environment at http://mloss.org.

An increasing number of software package are available in Debian like
- some simple-to-use utilities to apply compression techniques to the process of discovering and learning patterns: http://packages.debian.org/sid/complearn-gui
- a python package for convex optimization: http://packages.debian.org/sid/python-cvxopt
- a library for support vector machines: http://packages.debian.org/sid/libsvm2
- a machine-learning library: http://packages.debian.org/sid/libtorch3-dev
- an object-oriented programming language designed for researchers, experimenters, and engineers interested in large-scale numerical and graphic applications: http://packages.debian.org/sid/lush
- a large scale machine learning toolbox: http://packages.debian.org/sid/shogun-python-modular
- a data mining software in java: http://packages.debian.org/sid/weka

I'd like to know if you are using some of the packages or some other scientific software in Debian. Feel free to leave comment. Or maybe you are missing something in Debian?

If you are an author or user of some free software related to the topic of machine learning please consider registering it at http://mloss.org.

Masayuki Hatta: Japan's Big Push To Regulate The Internet

Once again Techdirt carries an excellent summary for the situation we Japanese are currently facing. Also I recommend the original article Techdirt refers.

Tag this post with del.icio.us!

3 January 2008

MJ Ray: MSN Santa Sex Scandal

I missed this one when it happened earlier last month, but I'm still putting it on my 2007 page of Bad Technology. Thanks to a colleague for pointing it out. Apparently MSN's Santa-bot made inappropriate comments to children and was exposed in Microsoft's sex-obsessed RoboSanta spouts filth at children The Register After that, it was almost inevitable that it would be taken off-line, even if just for a short while, so I think the next headline could have been predicted:- Microsoft kills Santa Claus The Register What surprised me was the ham-fisted PR attempt described in Microsoft accuses kids of bullying Santa into sex chat The Register If it was a mistake, then admit it. How did it happen? Unruly programmers wanting easter eggs at Christmas, did someone make the Santa-bot by modifying a porno-bot, or was it a learning bot finally corrupted by all the abuse it was sent - like Arnold on UEA/tsw's #pubfood IRC? Truly bizarre.

Ond&#345;ej &#268;ert&iacute;k: SymPy/sympycore (pure Python) up to 5x faster than Maxima (future of Sage.calculus?)

According to this test, sympycore is from 2.5x to 5x faster than Maxima. This is an absolutely fantastic result and also a perfect certificate for Python in scientific computing. Considering that we compare pure Python to LISP.

Ok, this made us excited, so we dugg deeper and ran more benchmarks. But first, let me say a few general remarks. I want a fast CAS (Computer Algebra System) in Python. General CAS, that people use, that is useful, that is easily extensible (!), that is not missing anything, that is comparable to Mathematica and Maple -- and most importantly -- I want it now and I don't care about 30 years horizons (I won't be able to do any serious programming in 30 years anyway). All right. How to do that? Well, many people tried... And failed. The only opensource CAS system, that has any chance of becoming the opensource CAS, in my own opinion, is Sage. You can read more about my impressions form Sage here. I am actually only interested in mathematical physics, so basically Sage.calculus. Currently Sage uses Maxima, because Maxima is old, proven, working system and it's reasonably fast and quite reliable, but written in LISP. Some people like LISP. I don't and I find it extremely difficult to extend Maxima. Also even though Maxima is in LISP, it uses it's own language for interacting with the user (well, that's not the way). I like python, so I want to use Python. Sage has written Python wrappers to Maxima, so Sage can do almost everything that Maxima can, plus many other things. Now. But the Sage.calculus has issues.

First, I don't know how to extend the wrappers with some new things, see my post in the sage-devel for details, it's almost 2 months old with no reaction, which shows that it's a difficult issue (or nonsense:)).

And second, it's slow. For some examples that Sage users have found out, even SymPy, as it is now, is 7x faster than Sage and sympycore 23x faster and with the recent speed improvements 40x faster than Sage.

So let's improve Sage.calculus. How? Well, no one knows for sure, but
I believe in my original idea of pure Python CAS (SymPy), possibly with some parts rewritten in C. Fortunately, quite a lot of us believe that this is the way.

What is this sympycore thing? In sympy, we wanted to have something now, instead of tomorrow, so we were adding a lot of features, not looking too much on speed. But then Pearu Peterson came and said, guys, we need speed too. So he rewrote the core (resulting in 10x to 100x speedup) and we moved to the new core. But first, the speed isn't sufficient, and second it destabilized SymPy a lot (there are still some problems with caching and assumptions half a year later). So with the next package of speed improvements, we decided to either port them to the current sympy, or wait until the new core stabilizes enough. So the new new core is called sympycore now, currently it only has the very basic arithmetics (and derivatives and simple integrals), but it's very fast. It's mainly done by Pearu. But for example the latest speed improvement using sexpressions was invented by Fredrik Johansson, another SymPy developer and the author of mpmath.

OK, let's go back to the benchmarks. First thing we realized is that Pearu was using CLISP 2.41 (2006-10-13) and compiled Maxima by hand in the above timings, but when I tried Maxima in Debian (which is compiled with GNU Common Lisp (GCL) GCL 2.6.8), I got different results, Maxima did beat sympycore.

SymPyCore:

In [5]: %time e=((x+y+z)**100).expand()
CPU times: user 0.57 s, sys: 0.00 s, total: 0.57 s
Wall time: 0.57

In [6]: %time e=((x+y+z)**20 * (y+x)**19).expand()
CPU times: user 0.25 s, sys: 0.00 s, total: 0.25 s
Wall time: 0.25

Maxima:

(%i7) t0:elapsed_real_time ()$ expand ((x+y+z)^100)$ elapsed_real_time ()-t0;
(%o9) 0.41
(%i16) t0:elapsed_real_time ()$ expand ((x + y+z)^20*(x+z)^19)$ elapsed_real_time ()-t0;
(%o18) 0.080000000000005


So when expanding, Maxima is comparable to sympycore (0.41 vs 0.57), but for general arithmetics, Maxima is 3.5x faster. We also compared GiNaC (resp. swiginac):

>>> %time e=((x+y+z)**20 * (y+x)**19).expand()
CPU times: user 0.03 s, sys: 0.00 s, total: 0.03 s
Wall time: 0.03


Then we compared just the (x+y+z)**200:

sympycore:
>>> %time e=((x+y+z)**200).expand()
CPU times: user 1.80 s, sys: 0.06 s, total: 1.86 s
Wall time: 1.92
swiginac:
>>> %time e=((x+y+z)**200).expand()
CPU times: user 0.52 s, sys: 0.02 s, total: 0.53 s
maxima:
(%i41) t0:elapsed_real_time ()$ expand ((x + y+z)^200)$ elapsed_real_time ()-t0;
(%o43) 2.220000000000027


Where GiNaC still wins, but sympycore beats Maxima, but the timings really depend on the algorithm used, sympycore uses Millers algorithm which is the most efficient.

So then we tried a fair comparison: compare expanding x * y where x and y are expanded powers (to make more terms):

sympycore:
>>> from sympy import *
>>> x,y,z=map(Symbol,'xyz')
>>> xx=((x+y+z)**20).expand()
>>> yy=((x+y+z)**21).expand()
>>> %time e=(xx*yy).expand()
CPU times: user 2.21 s, sys: 0.10 s, total: 2.32 s
Wall time: 2.31
swiginac:
>>> xx=((x+y+z)**20).expand()
>>> yy=((x+y+z)**21).expand()
>>> %time e=(xx*yy).expand()
CPU times: user 0.30 s, sys: 0.00 s, total: 0.30 s
Wall time: 0.30
maxima:
(%i44) xx:expand((x+y+z)^20)$
(%i45) yy:expand((x+y+z)^21)$
(%i46) t0:elapsed_real_time ()$ expand (xx*yy)$ elapsed_real_time ()-t0;
(%o48) 0.57999999999993

So, sympycore is 7x slower than swiginac and 3x slower than maxima. We are still using pure Python, so that's very promising.

When using sexpr functions directly then 3*(a*x+..) is 4-5x faster than Maxima in Debian/Ubuntu. So, the headline of this post is justified. :)

Conclusion

Let's build the car. Sage has the most features and it is the most complete car. It has issues, some wheels need to be improved (Sage.calculus). Let's change them then. Maybe SymPy could be the new wheel, maybe not, we'll see. SymPy is quite a reasonable car for calculus (it has plotting, it has exports to latex, nice, simple but powerfull command line with ipython and all those bells and whistles and it can also be used as a regular python library). But it also has issues, one wheel should be improved. That's the sympycore project.

All those smaller and smaller wheels show, that this is indeed the way to go, but very important thing is to put them back in the car. I.e. sympycore back to sympy and sympy back to Sage and integrate them well. While also leaving them as separate modules, so that users, that only need one particular wheel, can use them.

Next.

Previous.