Search Results: "cas"

23 September 2010

Dirk Eddelbuettel: R Project and Google Summer of Code: Wrapping up

As this year's admin, I wrote up the following summary which has now been posted at the R site in the appropriate slot. My thanks to this year's students, fellow mentors and everybody else who helped to make it happen. GSoC 2010 logo

Projects 2010 As in 2008 and 2009, the R Project has again participated in the Google Summer of Code during 2010. Based on ideas collected and disussed on the R Wiki, the projects and students listed below (and sorted alphabetically by student) were selected for participation and have been sponsored by Google during the summer 2010. The finished projects are available via the R / GSoC 2010 repository at Google Code, and in several cases also via their individual repos (see below). Informal updates and final summaries on the work was also provided via the GSoC 2010 R group blog.

rdx - Automatic Differentiation in R Chidambaram Annamalai, mentored by John Nash. Proposal: radx is a package to compute derivatives (of any order) of native R code for multivariate functions with vector outputs, f:R^m -> R^n, through Automatic Differentiation (AD). Numerical evaluation of derivatives has widespread uses in many fields. rdx will implement two modes for the computation of derivatives, the Forward and Reverse modes of AD, combining which we can efficiently compute Jacobians and Hessians. Higher order derivatives will be evaluated through Univariate Taylor Propagation. Delivered: Two packages radx: forward automatic differentiation in R and tada: templated automatic differentiation in C++ were created; see this blog post for details.

A GUI for Graphics using ggplot and Deducer by Ian Fellows, mentored by Hadley Wickham. Proposal: R puts the latest statistical techniques at one's fingertips through thousands of add-on packages available on the CRAN download servers. The price for all of this power is complexity. Deducer is a cross-platform cross-console graphical user interface built on top of R designed to reduce this complexity. This project proposes to extend the scope of Deducer by creating an innovative yet intuitive system for generating statistical graphics based on the ggplot2 package. Delivered: All of the major features have been implemented, and are outlined in the video links in this blog post.

rgeos - an R wrapper for GEOS by Colin Rundel, mentored by Roger Bivand. Proposal: At present there does not exist a robust geometry engine available to R, the tools that are available tend to be limited in scope and do not easily integrate with existing spatial analysis tools. GEOS is a powerful open source geometry engine written in C++ that implements spatial functions and operators from the OpenGIS Simple Features for SQL specification. rgeos will make these tools available within R and will integrate with existing spatial data tools through the sp package. Delivered: The rgeos project on R-Forge; see the final update blog post.

Social Relations Analyses in R by Felix Schoenbrodt, mentored by Stefan Schmukle. Proposal: Social Relations Analyses (SRAs; Kenny, 1994) are a hot topic both in personality and in social psychology. While more and more research groups adopt the methodology, software solutions are lacking far behind - the main software for calculating SRAs are two DOS programs from 1995, which have a lot of restrictions. My GSOC project will extend the functionality of these existing programs and bring the power of SRAs into the R Environment for Statistical Computing as a state-of-the-art package. Delivered: The TripleR package is now on CRAN and hosted on RForge.Net; see this blog post for updates.

NoSQL Interface for R by Yasuhisa Yoshida, mentored by Dirk Eddelbuettel. Proposal: So-called NoSQL databases are becoming increasingly popular. They generally provide very efficient lookup of key/value pairs. I'll provide several implementation of NoSQL interface for R. Beyond a sample interface package, I'll try to support generic interface similar to what the DBI package does for SQL backends Status: An initial prototype is available via RTokyoCabinet on Github. No updates were made since June; no communication occurred with anybody related to the GSoC project since June and the project earned a fail.

Last modified: Wed Sep 22 19:39:43 CDT 2010

Julien Danjou: Gnus news is good news!

As I already wrote too many times, I've started to use Gnus 6 months ago, and never looked back. At that time, I joined the ding mailing list in order to ask some dumb questions and, once, send a patch. There were very low activity on that list. Until Lars, the original Gnus author, came back. Three weeks ago, he started to wrote a new wash function to render HTML mails properly, with pictures. It's named gnus-html', and is (for now) based on w3m (but not on emacs-w3m, which is not part of Emacs). Last week, I've sent a set of patches to replace the usage of curl by the standard url-retrieve' function to fetch images, plus various enhancement. It seems that my work was good enough that Lars offered me write access to the git repository. I can therefore mess up the Gnus entirely. Hurrah! I've continued to work on gnus-html' and recently merged a set of patches improving image retrieval (which is now done in parallel) and starting to use url-cache' to cache image for a defined period of time. Of course, I found a bunch of tiny bug and special case while reading RSS feeds and various HTML mails, and fixed them all along. Lars added a libxml binding for Emacs 24, providing the html-parse-string' function. His future plan seems to be the abandon of w3m in favor of a native parsing via libxml to render HTML, and therefore, HTML mails. I should also mention the new nnimap' back-end; Gnus has been designed to read NNTP newsgroups, and not mails. Consequently, it had a very poor behaviour when used with a back-end such has IMAP. Lars took a week to rewrite entirely our dear nnimap' back-end, and make it act in a more expected way. There's still a bunch of bug and code to write, but it is at least usable and seems faster than the old code. Last thing I did was to rewrite the icon support in the group buffer. When I started to use Gnus, I was curious and tried to configure this. I never managed to make it work, and now know and understand why it was broken. So I ended rewriting entirely, and now it works. I never though I would understand, fix, and commit this code when reading the Gnus documentation this winter, but hell yeah, I did. Now I've still several little project to improve things in all sort of area. We'll see what I'll do next. :-) Flattr this

22 September 2010

Vincent Sanders: I like driving in my car. It is not quite a Jaguar

I work from home, this is a good thing. I benefit from a 20 metre commute, comfortable working environment and generally low carbon lifestyle.

Except on Wednesdays, on Wednesday I have to get up early and go to the office, this is not usually too much of a chore and takes around 90 minutes each way.

Today was different, I made a small five minute diversion to collect a colleague to whom I was giving a lift and then due to a little problem near the M6/M61 junction spent a fun filled three hours sitting in traffic crawling along the M56. At least I had company instead of being on my own.

Before I did the return journey I decided to check the traffic news sites. Oh dear now the M60 was stuffed, I altered the route and only had to queue on the M56 for twenty minutes or so. I dropped my colleague off at his place (avoiding the worst bits of the M62/M66 junctions by use of a rather convoluted back route) and proceeded to queue on the M62 for a while for no apparent reason.

Basically I have spent almost seven hours in the car today to do a 150 miles or a little over 20 miles per hour average. I was just going to rant about the dreadful lack of any redundancy or resilience in the UK road system which often grinds to a complete and utter halt if there is a single failure.

However a different thought has wandered across my travel weary mind. It has occurred to me that this average speed is faster than anyone could reasonably expect to do this trip for the majority of human existence.

In 1810 and indeed for all time before, your best possible speed by good horse, for 150 miles, would have been two days (and your horse would have probably been very poorly afterwards) This assumes your horse could do the 75miles (120km) each way in times consistent with modern world endurance trials... across a mountain range! Yes the Pennines are only tiny but even so!

A hundred years later, in 1910, the British railway network was nearing its zenith in most measurable terms. The influence across the north of England was profound and pushed the industrial revolution ever faster towards its climax before the first world war. Even at this point in time my best reading of the available timetables says I would have needed to change trains four times each way, purchased eight separate tickets from six different companies and taken around nine hours to make the journey allowing for hanging around on platforms.

Another fifty years on, in 1960, the trans-pennine car journey would have been on poorly maintained trunk routes through the decaying cores of the declining post-industrial northern cities. The route would probably have involved the A646, A59 or the A58 which at this time were not the well maintained (if slightly shabby) roads of today but instead were dangerous twisty and, from the looks of the archive photographs, positively heaving with traffic. On these pre-motorway strips of tarmac the 150 mile round trip would have taken in excess of seven hours (even today's mapping systems suggest over four and a half hours would be needed)

So instead of being frustrated that my commute took an extended period today I have instead decided that I shall enjoy the fact it was faster and certainly more comfortable than at any time in the past. Well that and I need to get the cars air-conditioning fixed ;-)



Gunnar Wolf: Damage control: Cleaning up compromised SSH keys

This morning, my laptop was stolen from my parked car while I was jogging. I do not want to make a big deal out of it. Still, even though I am sure it was not targetted at my data (three other people at least were reporting similar facts in the same area), and the laptop's disk will probably just be reformatted, I am trying to limit the possible impact of my cryptographic identification being in somebody else's hands. GPG makes it easy: I had on that machine just my old 1024D key, so it is just matter of generating a revocation certificate. I have done that, and uploaded it to the SKS keyservers - Anyway, here is my revocation certificate: 
-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: A revocation certificate should follow
iHIEIBEIADIFAkyaOZwrHQJBIGNvbXB1dGVyIGNvbnRhaW5pbmcgdGhpcyBrZXkg
d2FzIHN0b2xlbgAKCRDYDvNai7UnrzWAAKC34eF76JQjxrZqSjNwcC0dU/5VbACg
gMIMmYg91Sl3y8KsZXdGj/rV7UE=
=rdlT
-----END PGP PUBLIC KEY BLOCK-----
But What worries me more is access to the computers my ssh key works for. Yes, the ssh key uses a nontrivial passphrase, but still SSH keys cannot be revoked (and this makes sense, as SSH should not add the delay, or potential impossibility, to check with a remote infrastructure whenever you want to start a session). So, I generated a new key (and stored it at ~/.ssh/id_rsa.new / ~/.ssh/id_rsa.new.pub) and came up with this snippet: 
  1. $ OLDKEY=xyHywJuHD3nsfLh03G1TqUEBKSj6NlzMfB1T759haoAQ
  2. $ for host in $(cat .ssh/known_hosts   cut -f 1 -d \  cut -f 1 -d ,  
  3.                 sort   uniq); do 
  4.     echo == $host
  5.     ssh-copy-id -i .ssh/id_rsa.new.pub $host && 
  6.     ssh $host "perl -n -i -e 'next if /$OLDKEY/;print' .ssh/authorized_keys"
  7. done
Points about it you might scratch your head about: Caveats: Oh, by the way: If you noticed me knocking on your SSH ports... please disregard. Possibly at some point I connected to that machine to do something, or it landed in my .ssh/known_hosts for some reason. I currently have 144 hosts registered. I am sure I triggered at least one raised eyebrow. And I will do it from a couple of different computers, to make it less probable that I miss some I have never connected from while at the particular computer I am sitting at right now. So... Any ideas on how to make this better?

Bernd Zeimetz: refactoring the guruplug server plus - part two

Although the modifications described in my last blog post about refactoring the GuruPlug resulted in a well working GuruPlug, I've decided to give it some more air to breath at the top of the case. The large round piece of plastic in the middle asked to be removed - and now a shiny 60mm fan grill protects the board from the bad world outside of the case. GuruPlug Server Plus case with fan grill (1/2) GuruPlug Server Plus case with fan grill (2/2) So far its working fine and not too hot. Guess I'll see what happens next summer.

Luca Bruno: Report from the Debian/Ubuntu Community Conference, ITA 2010

From the 17th to the 19th of September in Perugia, Italy, it took place the 5th edition of the Italian Debian Community Conference, which has been attended by many contributors and users. For the first time, the event has been organized in collaboration with the Italian Ubuntu community, as to create a new joint conference in order to foster shared contributions and emphasizing the large common ground of our projects. This new experimental kind of mini-conference was then labeled DUCC-IT, to reflect both the local profile and the mutual collaboration. After the initial social night, spent discussing of several ongoing free software efforts and having dinner all together, the conference official opening started on Saturday, temporarily housed at University of Perugia, with a series of talks and hands-on session aimed at recruiting new contributors to work on development, translations, documentation writing and marketing. It has been a good opportunity to celebrate the Software Freedom Day too, in collaboration with FSUG Italia and the participation of some local schools.

DUCC-IT '10 Group Photo The event been also attended by some members of the Debian Women and Ubuntu Women teams (whose goal is to promote women participation in both projects), who organized a round-table debate taking the Italian panorama as a study case. The discussion embraced different topics, ranging from the wide difference in numbers, to the deep causes of this phenomenon and how to improve the situation. With the help of the hacklab staff (hosting the debate), an audio/video streaming has been made available in real-time, and many remote participants joined us with comments and questions. This new kind of collaboration between our communities was found to be really positive and more events has already been drafted for the next year, including a translation sprint and a contributors meeting. We encourage worldwide local communities to try and engage in a similar experiment: organizing and joining a DUCC event will be pure fun. A detailed report of the conference will be soon available, completed by photos, participants comments, video records and slides for the talks.
We d really like to thank the Math Department of University of Perugia, the Projectz On Island hacklab, FSUG Italia, the Ubuntu-it community and everyone who contribute to this event.

21 September 2010

Russ Allbery: Lightweight DNS servers

Since I got a lot of responses to my previous journal entry about tinydns, I wanted to share the answers with everyone else (and with search engines) in case anyone else was wondering the same thing. The general consensus choice for a simple authoritative DNS server was NSD (which is also packaged for Debian, currently as nsd3). I took a brief look and indeed it would do what I want. It's still a bit more complicated than I would prefer, but nowhere near as bad as BIND, and it handles all modern DNS features (IPv6, DNSSEC, etc.). Several people mentioned unbound, which looks like a great solution for a different problem than the problem that I have. It's a caching DNS server rather than an authoritative DNS server, although it supports some authoritative overrides. There was also one recommendation of PowerDNS (Debian package pdns-server and friends), which I'd heard of before and which I think I'd turn to if I was looking for a full-featured DNS server. I think it's overkill for my tiny problem, but it has the neat feature that you can run an arbitrary command to provide DNS responses. That means that it could potentially replace lbnamed, should we need something with more features for some of the DNS tricks that we do with Stanford. Finally, tinydns development hasn't completely stopped since djb stopped working on it, and there are maintained forks that have patches to support new record types. Still no DNSSEC support that I'm aware of, but continuing to use it with patches to support SRV and AAAA records is quite appealing, since I much prefer the zone file format (the "standard" zone file format is a horrible bodge) and, of course, I'm already familiar with it. If a new version were packaged for Debian, I'd probably just keep using it. And as a bonus follow-up from my original post, on the topic of the divergent handling of iptables for IPv4 and IPv6, Martin Krafft recommended ferm, which looks very interesting but which I haven't yet investigated further.

Bastian Venthur: Weird Konsole Split

Can someone explain me the purpose of (KDE s) konsole s ability to split?
You can split horizontally or(!) vertically but the resulting terminals are just mirroring the activity of the first one. What is the usecase for that? Yakuake does splitting right. You can split several times horizontally and vertically and the resulting terminals are independent. You can also switch the focus of the terminals easily via keystrokes which makes yakuake an invaluable tool when working on the terminal under KDE. So what s the point of konsole s variant of splitting?

Richard Darst: The DebConf travel sponsorship process

This is a guest post by Michael Schultheiss, DebConf10 travel sponsorship team lead.DebConf doesn't just provide sponsored food and accommodation to attendees, it can also provide travel reimbursement to 20-30 attendees each year, from all over the world.The travel sponsorship team has historically been formed by a call to debian-devel-announce for developers to review sponsorship requests. This has resulted in one of the most transparent review teams of DebConf, which is by design since this is one of the hardest decisions to make. A team of about ten people will review applications. The travel sponsorship team is typically comprised of both people with past travel sponsorship team experience as well as new members to help make the process as fair as possible. The team will consider the listed workplans, the answer to the question "what do you do for Debian", their general knowledge of Debian contributors, and web searches of Debian lists, debian.org, and the web in general, looking for Debian and free software related activities.Applicants are rated on a 1-5 scale of contributor level and amount requested. These are averaged and sorted, and this is used as the basis of travel sponsorship - but is subject to human review for special cases. The team will then decide the cutoffs for the different queues - for example, DebConf10 used queue A which had money released first, queue B which was released after more fundraising, and queue C which was told "no". After these queues are decided, most of the selection team doesn't need to be as active, but anyone from that team can work with the organizers to decide percentages of funding of the queues and when to commit money to them, based on our budget.Attendees submit documentation for actual flight costs after the conference, and can be reimbursed for min(actual_costs, amount_requested).There are various problems with this system, which are pretty well known but not so easy to fix:These questions are hard to answer. I'm encouraged that there are some discussions on debconf-team which are working towards a better system. Most importantly, these people want to start early, which is almost guaranteed to result in a better outcome for everyone. If you are interested in working to improve the travel sponsorship process, I encourage you to get involved in the team and start early.Tags: debconf

Raphaël Hertzog: Everything you need to know about conffiles: configuration files managed by dpkg

The Debian policy dictates that package upgrades must take care of preserving user changes to configuration files. This article will explain you how most packages ensure this. This is important knowledge for anyone who has to manage upgrades: knowing how it works lets you easily automate most of it and deal correctly with the fallout. How dpkg manages configuration files Most packages rely on dpkg to properly install configuration files. Dpkg keeps a checksum of the last installed version of configuration file. When it must install a new version, it calculates the checksum of the currently installed file and if it doesn t match anymore, it knows that the user has edited the file. In that case, instead of overwriting the configuration file, it asks the user what to do. You probably already have seen those questions, they look like this: 
Configuration file  /etc/bash.bashrc'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** bash.bashrc (Y/I/N/O/D/Z) [default=N] ?
In this specific example, if you answer Y or I (for yes or install ), dpkg will install the new version of /etc/bash.bashrc but it will also backup the current version in /etc/bash.bashrc.dpkg-old. If you answer N or O (for no or old ), dpkg will install the new version in /etc/bash.bashrc.dpkg-dist and /etc/bash.bashrc is left untouched. The two other answers allow you to examine the differences before taking a decision. Note that if you choose to start a shell, the new version is currently available as /etc/bash.bashrc.dpkg-new (and since Squeeze there are convenient environment variables $DPKG_CONFFILE_OLD and $DPKG_CONFFILE_NEW in case you want to create a custom review script). All configurations files managed by dpkg are called conffiles because that s the name of the field where they are recorded in the dpkg database. You can display the list of conffiles for any package: 
$ dpkg --status bash
[...]
Conffiles:
 /etc/skel/.profile ecb6d3479ac3823f1da7f314d871989b
 /etc/skel/.bashrc 2afdd6c53990f2387a7ef9989af0bc07
 /etc/skel/.bash_logout 22bfb8c1dd94b5f3813a2b25da67463f
 /etc/bash.bashrc 5b3c3bc73d236e4e1b6f9b6c1ed5964e
[...]
The command dpkg-query --showformat='$ Conffiles \n' --show bash can give you the same information if you need to retrieve only that field. The 32 characters after the filename are the MD5 checksum of the original configuration file provided by the package. Avoiding the conffile prompt Every time that dpkg must install a new conffile that you have modified (and a removed file is only a particular case of a modified file in dpkg s eyes), it will stop the upgrade and wait your answer. This can be particularly annoying for major upgrades. That s why you can give predefined answers to dpkg with the help of multiple --force-conf* options: If you use Apt, you can pass options to dpkg with a command-line like this: 
$ apt-get -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" dist-upgrade
You can also make those options permanent by creating /etc/apt/apt.conf.d/local: 
Dpkg::Options  
   "--force-confdef";
   "--force-confold";
Bringing up the conffile prompt at any time The conffile prompt is only displayed when dpkg detects that the package provides an new version of the conffile. Thus reinstalling the same package will not bring up the prompt. But you can instruct dpkg to ask nevertheless with the --force-confask option. This is a new feature in Debian Squeeze. It will only ask for files that are locally modified.
If you want to read more articles like this one, click here to subscribe to my free newsletter. You can also follow me on Identi.ca, Twitter and Facebook.
Flattr this Share/Bookmark 4 comments Support my work

20 September 2010

Marco Silva: Handling spam with SpamAssassin + Bogofilter + Procmail + Sup

This is mostly inspired on this article. The main difference is that I talk about sup instead of mutt, and use SpamAssassin together with bogofilter. Having all these utilities installed and the sources for sup configured, you only need to touch procmail and sup configuration files. My ~/.procmailrc contains a lot of comments, but what it does is basicly the following: If the message is smaller than 250 KiB, it calls SpamAssassin with it. This will create the headers X-Spam-Level and X-Spam-Status in this message. Only messages smaller than 250KiB are considered because most spam isn't bigger than that, and considering big messages would overload SpamAssassin. Then, if SpamAssassin consider this message as spam, tell bogofilter about it. Next, call bogofilter, which will add the X-Bogosity header to the message. Now we have to configure sup. We need to tell it to update bogofilter when we manually mark a message as spam. For this we can use ~/.sup/hooks/mark-as-spam.rb. We must consider two cases: the case where bogofilter thought it was not spam, and the case where bogofilter was not sure about its status. In the first case, we must tell it to forget that the message was not spam and to register it as spam now. In the second, just register it as spam. Also, we don't want spam on inbox. So we can edit ~/.sup/hooks/before-add-message.rb to look for the headers generated by SpamAssassin and Bogofilter and remove the :inbox label. It's important to mark unsure messages with a label so that we don't forget to mark it as spam if it is, and as not spam if it's not. So we need a way to mark messages as not spam. I included a new keybinding in ~/.sup/hooks/startup.rb. Unfortunatelly, I don't know how to include a new keybinding in sup without repeating all the keybindings for the mode I'm dealing with, so that's how I've made it. The command removes the :unsure label and mark it as non-spam. It also sets the undo command, unregistering the message in bogofilter. The case where the message was previously marked as spam, and not unsure, is also handled. I'm sure there's a lot of room for improvement, specially in the Ruby part, but it seems to be working.

19 September 2010

Gregor Herrmann: RC bugs 2010/37

thanks to Lucas' recent archive rebuild & bug filing we have new stuff to work on :) my activities in the last week:

Iustin Pop: Debian Thanks

There are many ways that people express their thanks to Debian: mails, nice comments in a bug report, patches, so on. But to receive them in person is one of the most pleasing ones. I recently visited Bologna at the invitation of another DD (hi Guido!). The trip was nice, my first trip to this city, and a significant part of it was devoted to the local food (of course!). One evening, after a late dinner which included an absolutely fantastic Mascarpone desert, as we were paying at the cashier, one of the guys from the restaurant came forward and shook my hand while telling me something. I was tired and the few Italian I know didn't help at all, so I was at a loss on what's happening. Now, if you've ever been in a foreign country without knowing the language, you know this feeling: there is something strange going on, people are talking in a language you don't understand, and you might have just made a very big social mistake without realising it. Or it's just a silly, harmless misunderstanding. The only question is whether staying silent and just smiling is the right thing to do. In other words, you're at a complete loss. This all happens usually in just one or two seconds, before someone (if anyone, that is) manages to translate for you. In this particular case, the translation initially didn't make any sense at all: It's your t-shirt ; I usually wear t-shirts, and people never congratulate me for it. Then I realised: I was wearing one of my Debian t-shirts. My companions told him that I'm indeed involved with Debian, and he was extending his thanks. So, from a bolognese fan whose name I unfortunately didn't catch, to the Debian Project: thanks!

Joerg Jaspert: FTPMaster meeting, III - discovering new Restaurants for fun and profit

And there, another day come and passed, still no real breakage, only some small things here and there. We are doing something wrong, definitely. Ok, granted, providing everyone who has experimental in their sources.list with a free upgrade path due to us suddenly forgetting that apt actually really wants this NotAutomatic: yes flag in the release instead of just looking it up in our database, later on doing the same to all of the backports user (accidently, happens when you have identical codebases), does count enough for a small icecream. But we didn t have anything serious, which is great. Alex finished patching dak rm in a way that we can now actually close all bugs attached to a (source) package we remove and was thinking of enhancing it to actually also be able to close wnpp bugs attached to it (think of O: bugs for example). Mark had design work on g-p-s, which actually is how do we generate packages and sources and how to move the config sanely into a database . Sounds easy, but trust me, it isn t, when you consider all the special cases Debian has. He also jumped in helpfully whenever there was a question around some coding and stuff, at one time turning up with the insanity/genius/younameit code that allows us to transfer even more of dak.conf into the database. This gets us a good way along the way that should end up with a dak.conf consisting of only the database details and nothing else. My job today looked similar like yesterday. Merge code I got, eat, remove lots of values from dak.conf. This weekend besides totally missing out on Baklava, had us change our dak code quite a bit: 46 files changed, 982 insertions(+), 1060 deletions(-). Besides that we do have about 18 points on our meeting notes, a mail going off to one debian lists sometime the next days summarizing it a bit more.

18 September 2010

Lucas Nussbaum: UDD bugs search news

Many changes to the UDD-backed Debian Bugs Search: Enjoy!

Lucas Nussbaum: Helping improve Ruby on Debian and Ubuntu

After my last blog post on Ruby in Debian and Ubuntu, I got a few offers for help, which is excellent. So I ve decided to update this wiki page with pointers to the various team resources, and some examples of tasks that newcomers can do. I d like to stress that many pending tasks are really doable by someone not very qualified in Debian packaging. They are not done just because nobody has the time to do them. Some good examples are the libinotify-ruby, libimage-size-ruby, libffi-ruby RC bugs.

Robert Collins: Maintainable pyunit test suites fixtures

So a while back I blogged about maintainable test suites. One of the things I ve been doing since is fiddling with the heart of the fixtures concept. To refresh your memory, I m defining fixture as some basic state you want to reach as part of doing a test. For instance, when you ve mocked out 2 system calls in preparation for some test code that represent a state you want to reach. When you ve loaded sample data into a database before running the actual code you want to make assertions about that also represents a state you want to reach. So does simply combining three or four objects so you can run some code. Now, there are existing frameworks in python for this sort of thing. testresources and testscenarios both go some way towards this (and I and to blame for them :) ), so does the zope testrunner with layers, and the testfixtures project has some lovely stuff as well. And this is without even mentioning py.test! There are a few things that you need from the point of view of running a test and establishing this state: Large test suites or test suites dealing with various sorts of external facilities will also often want to optimise this process and put the same state into place for many tests. The (and I m not exaggerating) terrible setUpClass and setUpModule and other similar helpers are often abused for this. Why are they terrible? They are terrible because they are fragile; there is no (defined in the contract) way to check that the state is valid for the next test, and its common to see false passes and false failures in tests using setUpClass and similar. So we also need some way to reuse such expensive things while still having a way to check that test isolation hasn t been compromised. Having looked around, I ve come to the conclusion we ll all benefit if there is a single core protocol for doing these things, something that can be used and built on in many different ways for many different purposes. There was nothing (that I found) that actually met all these requires and was also tasteful enough that folk might really like using it. I give you fixtures . Or on Launchpad. This small API is intended to be a common contract that all sorts of different higher level test libraries can build on. As such it has little to no policy or syntatic sugar. It does have a nice core, integration with pyunit.TestCase, and I m going to add a library of useful generic fixtures (like temporary directories, environment isolators and so on) to it. I d be delighted to add more committers to the project, and intend to have it be both Python 2.x and 3.x compatible (if its not already my CI machine isn t back online after the move yet, I m short of round tuits). Now, if you re writing some code like: 
class MyTest(TestCase):
    def setUp(self):
        foo = Foo()
        bar = Bar()
        self.quux = Quux(Foo(), Bar())
        self.addCleanup(self.quux.done)
You can make it reusable across your code base simply by moving it into a fixture like this: 
class QuuxFixture(fixtures.Fixture):
    def setUp(self):
        foo = Foo()
        bar = Bar()
        self.quux = Quux(Foo(), Bar())
        self.addCleanup(self.quux.done)
class MyTest(TestCase, fixtures.TestWithFixtures):
    def setUp(self):
        self.useFixture(QuuxFixture)
I do hope that the major frameworks (nose, py.test, unittest2, twisted) will include the useFixture glue themselves shortly; I will offer it as a patch to the code after giving it some time to settle. Further possibilities include declared fixtures for tests, and we should be able to make setUpClass better by letting fixtures installed during it get reset between tests.

17 September 2010

Jo Shields: Mono mythbusting, September 2010 edition

There are corners of the Internest where foolish people congregate, and invent stories. These foolish stories are then read as gospel by trusting people, and reposted, until the original made-up source is concealed from view. As an attempt to stem this flow of disinformation, here are some commonly held but incorrect beliefs about the Mono framework, and an explanation of the reality of the situation, as far as I understand it. The next Mono version is co-developed with Microsoft There is a grain of truth behind this one, but it s a gross mischaracterisation. Mono 2.8, when it ships, will bundle, for convenience, a number of Free Software libraries, which are released by Microsoft under a license considered Free Software by the Free Software Foundation, the Ms-PL. These are: So, to summarise, there are five Free Software libraries written by Microsoft under the Ms-PL included in Mono 2.8 but only two of those five are new, and none of them were co-developed in any meaningful sense. Mono development is dead There have been reports that Mono development has ended, on the basis that no commits have been made to Novell s Subversion server for a few weeks. However, these reports miss one minor detail Mono has moved to Github.com. There were 35 commits from 9 different people to the main mono.git repository (of dozens of repositories under the main Mono project) in the last 2 days, at time of writing far from dead. Mono lacks features found in Microsoft.NET Mono lacks libraries found in Microsoft.NET, such as the Windows Presentation Framework GUI toolkit. In terms of features, i.e. things implemented at a compiler or runtime level, Mono is typically more advanced. This is helped in no small part by Mono s Free Software development model, allowing experimentation and what if changes to the core runtime which a sluggish corporate behemoth like Microsoft cannot accommodate. To give three real-world examples, Mono allows embedding of its compiler as a service, and provides a REPL shell this is a planned feature for .NET 5.0, but has been available for years in Mono; Mono.Simd provides a number of data structures which will run on any version of Mono or Microsoft.NET, but will use optimized CPU extensions like SSE when run on a sufficiently new version of Mono, on an appropriate architecture as far as I m aware, there is nothing like this available or planned for Microsoft.NET. Mono is able to produce fully compiled, static executables which do not need to JIT anything at runtime this is used for iPhone compilation, for example, where JITters are not permitted. There is no comparable feature in Microsoft.NET. Clearly, Microsoft.NET can only be thought of as more featureful if one defines features in terms of does it have lots of libraries? in terms of functionality, Mono is ahead. Mono can sneak onto your system without your knowledge If you don t have the mono-runtime package installed, you can t run Mono apps. It is possible to install some Mono apps alongside the awful-yet-popular mononono equivs package, since the popular equivs script fails to place Conflicts on the correct packages (F-Spot will be blocked, Tomboy will not). No package in Debian or Ubuntu embeds its own copy of the Mono runtime, and we have no plans to make any changes to packaging which would allow execution of any C# application without mono-runtime installed. If one is using a different OS, then things may be different e.g. The Sims 3 for Mac & PC uses embedded Mono, which you wouldn t know about without looking. Canonical are pursuing a pro-Mono agenda, and are responsible for it being pushed in Debian Mono development has been happening in Debian for longer than Canonical has existed the first upload was made in April 2002. Ubuntu is made primarily from software already available in Debian deemed of sufficient quality, and when F-Spot and Tomboy became parts of the default Ubuntu desktop system in 2006, both pieces of software were already available in Debian and deemed of sufficient quality. Nobody working on Mono in Debian is paid by Canonical actually, that s not entirely true, three packages related to accessibility are officially maintained by someone who was hired by Canonical after doing the initial packaging work when he worked for Novell. But the Mono runtime itself, Canonical have no influence over its direction in Debian. As for a pro-Mono agenda , they ve always taken an extremely pragmatic approach to language choice, never showing any real preference for one language or another when it comes to app selection. They don t exhibit any overt anti-Mono policies, which is not the same thing. The names of people contributing to Mono in Debian are not secret check the pkg-mono team page on Alioth. The System.Windows.Forms namespace is protected by Microsoft patents The truth is, nobody knows for sure if SWF, or any other part of Mono, or any other framework such as Vala or Python, is covered by Microsoft patents. The way the US patent system is contrived, it is actively dangerous to check whether something contains any patents when you write it, as you are liable for triple damages should it later emerge that there WAS a patent, even if your searching missed the fact. You cannot patent an API or namespace only a specific implementation of a software concept, in those countries where software patents are permitted. There has never been any evidence shown that Mono s implementation of SWF, or indeed any part of Mono, infringes any Microsoft-held patents, because if that were the case, then the code would be rewritten to avoid the issue much like the approach taken by Linux kernel developers when a patent becomes apparent. The belief within the Mono community is that the core parts of Mono as defined in ECMA334/335 are safe (they are covered by the Microsoft Community Promise patent grant); any of the Ms-PL libraries mentioned above is definitely safe (Ms-PL includes an explicit patent grant); and the rest of the package is likely safe too (on the basis that it is a named component of the Open Innovation Network s list of protected software and on the basis that there s unlikely to be anything patentable in one of many implementations of basic ideas like database connectors). Nobody knows for sure, because that s how the system works. But, again, nobody knows for sure that Microsoft patents don t apply to other frameworks such as Python there is simply a belief and an assumption that they do not. MonoDevelop contributors removed GPL code from MonoDevelop, in an attack on the GPL This is somewhat disingenuous, given MonoDevelop is LGPL. The MonoDevelop team excised the remaining GPL code (and there wasn t much of it) in order to grant greater Freedom to developers. Previously, the entire MonoDevelop IDE was a GPL combination, meaning any add-ins for the IDE also needed to be GPL, regardless of developer choice. Now, any developer can write an add-in for MonoDevelop, using the license of their choice, whether another Free license like the Mozilla Public License, or a proprietary closed-source add-in. They are still welcome to produce GPL add-ins, if they want to, as well. Mono won t run random Microsoft.NET apps anyway, so isn t really cross-platform Actually, this one is often true. When developers write apps for Windows primarily, they rarely take the time to think is this the correct way to do it? and will often plough on with an assumption about the Windows way of doing things. It might be simple things like filesystem assumptions (assuming a case-insensitive filesystem, or assuming a backslash is used to separate directories, not a forward slash). It might be more involved, such as using P/Invokes into Windows-specific C libraries, when a more cross-platform alternative is possible. So, often, random applications for Microsoft.NET won t run on Mono. The reverse is also true F-Spot or GNOME Do are fairly heavily tied to Linux (or to UNIX-like OSes with X11, at any rate), through the libraries they invoke being fairly platform-specific. You can write platform-specific Java, with one quick piece of JNI, too. It should be noted, however, that Mono makes the chance of .NET applications being ported to Linux (and/or Mac) much more likely, since even in the worst case scenario, a company only needs to fix a portion of their source to make it cross-platform. The Mono team have a tool called MoMA, which will scan an application and its libraries, and give you detailed reports on the app s portability. This info is used at both ends by app vendors who want to become more portable, and by the Mono team who want to fill in the most frequently encountered blanks. And, it should be stressed, writing cross-platform apps is entirely possible if one desires it e.g. the IRC client Smuxi is pure cross-platform C#, and the executables compiled on Mono on Linux will run fine on Microsoft.NET on Windows. Portability in this direction is important too consider how many people have been introduced to Free Software thanks to availability of Free apps on Windows like OpenOffice.org, Firefox, Pidgin or GIMP. You can download Tomboy for Windows, and work is ongoing on fixing Banshee portability issues (which are mostly caused by gStreamer). Mono on Android is a terrible idea which will be super slow There are two efforts to enable developers to write Android applications with Mono a paid proprietary product from Novell called MonoDroid, and an untitled porting effort by big-name Android hacker Koushik Dutta. I have no insight into MonoDroid since it hasn t been released yet, but Koush did some benchmarks of his work which exhibited some remarkable figures compared to Dalvik (and even compared to Sun Java for ARM). Those numbers haven t been updated to reflect the Froyo Dalvik JITter, but Mono on Android is still very exciting from a performance perspective. Mono isn t really Free Software The source code is all available under Free Software licenses. So, um, yes it is. We don t need Mono because we have $LANG By the same logic, we don t need $LANG because we have Mono. By all means, use the language of your choice other developers will use the language of their choice too. If you want to use Vala or Python or Java, then by all means, go ahead. It doesn t mean there s no room for more languages, better suited to other usage patterns. Haskell is great for some types of development, so is Fortran, and so is C#. You might not need Mono because of $LANG, but there are others in the world with different needs. Mono apps are all slow, and crash your computer, and stuff No userland app can crash your Linux system, unless there s a bug in the kernel, or you have some severe problems with your hardware. If you ve ever observed a crash as a result of running a Mono app, then it s really coincidence that Mono is tickling whichever part of your system is busted. As for slow startup time may well be slower than for apps written in C or Vala. There s a delay due to the JITter needing to compile all the libraries used by the application (we may AOT the most common libraries in a future Mono package version, at upstream s recommendation). Once an app is running, it should be fast compared to a Python app, and memory-light compared to a Java app. The new garbage collector in Mono 2.8 should also offer significant improvements to performance, especially under heavy workloads. Ubuntu s default GTK+ themes require Mono This is my favourite recent nonsense to emerge from the Internest. There are reports mostly restricted to IRC and blog comments, that (paraphrasing here) removing Mono removes the Ubuntu themes . Here s the reality: in Ubuntu 10.04, a new visual style was implemented throughout the distro. As part of this, small icons (e.g. notification area icons) were set to be black-and-white by default, and colourised when attention is needed (e.g. the power-off icon turns red when a reboot is needed, the messaging indicator turns green when there s a new message). All of these new monochrome icons are in a package named ubuntu-mono . Removing the ubuntu-mono icon package will also remove the new Ubuntu GTK+ themes, in the light-themes package. So there s your explanation: the themes have a dependence on some monochrome icons, not on the Mono framework.

Matthew Garrett: USB runtime power management

I've just committed some patches to the rawhide (not F14) tree that re-enable USB autosuspend on some devices. This set includes a workaround in the bluetooth input code that should handle the case where people were seeing their input devices become laggy when autosuspend was enabled, but there's still some chance that other bluetooth devices will behave slightly oddly. If that's the case then try:

echo on >/sys/class/bluetooth/hci0/device/power/control

and see if it improves things. If so then please file a bug and include information about the device you're trying to connect to.

The other thing this patchset does is enable USB HCD runtime power management by default. This means that if you have a host controller with nothing connected, or with connected but suspended devices, the host controller will be powered down. It'll automatically power back up if a device is connected or a connected device sends a wakeup event. The obvious failure mode here is that your USB ports stop working. If that happens, please file a bug and include the output of lspci and dmidecode (you may want to edit the dmidecode output to make sure that your serial numbers are removed). In both cases, assign the bug to me (mjg at redhat.com).

Once this seems fairly stable I'm going to be looking at powering down more mostly unused PCI devices. SD readers and firewire controllers are typically unused and both support generating wakeup events, so they're the next step.

Russell Coker: Preferring Low Quality and Microsoft Software

Is Low Quality in Italian Academia related to the choice of Low Quality Software? Diego Gambetta and Gloria Origgi wrote an interesting paper titled L-worlds: The curious preference for low quality and its norms [1]. The paper describes how in Italian universities (and large portions of Italian life) there are covert agreements that both parties in a transaction (the employee and the employer or the buyer and the seller) will deliver less than agreed while pretending that they are offering the agreed exchange. People who offer high quality in exchanges are discriminated against because they make people who offer low quality in exchange feel guilty. Nathan suggests that this is the explanation for people choosing to pay for inferior software from Microsoft instead of getting superior software for free [2]. Now it does seem quite plausible that someone who is offering low quality goods in the manner of Italian academia would refuse to consider software from any company other than Microsoft, after all the easiest way of selecting software is to phone a MS representative and be told exactly what to buy. But I don t think that this explains even a significant fraction of the people who refuse free software. There is no direct analogy between bilateral agreements to produce low quality and the choice of MS Software because MS Software is quite expensive (they demand what would be considered a high quality trade in the jargon of the paper). If someone was to buy one of the cheaper laptops in Australia (around $650 new) and upgrade to the full version of Windows 7 along with purchasing the home version of MS Office then the cost of software would be almost as much as the cost of hardware. If they wanted one other MS product then the cost of MS software would probably be greater than the cost of hardware. Hardware costs are steadily falling and MS prices are only increasing, for people who use MS software we should expect that soon the MS tax will be the majority of the costs of running a typical PC. Good Reasons for Choosing MS One thing we have to consider is that there are people who have good reasons for using MS software. One example is the companies that depend on proprietary applications which are central to their business. When the entire company s data is stored in an undocumented proprietary database it s really not easy to change to a different application even when everyone in the company knows the software to be of amazingly low quality. If the vendor of the proprietary application in question decides to only support MS Windows then it s customers (victims?) have no choice about which OS to use. One interesting thing to note about such companies that are locked in to proprietary software is that the amount that they spend per year on license and support fees is usually greater than the cost of hiring one good programmer. If a few such companies formed a consortium to develop free software to manage their business where each company paid the salary of one programmer then after a couple of years of development they could move to the free software and reduce their operating expenses. Another category of users who have a good reason to choose MS is the people who play games seriously. If you want to play games then MS Windows does offer some real advantages. The price of games will usually be a fraction of the hardware cost (the serious gamers spend a lot more on hardware than most people) and MS Windows is apparently the best PC OS for commercial games. Personally I ve found that there are more than enough free games on Linux to waste my time, Warzone 2100 [3] is one that I currently play, and I ve tried Battle for Wesnoth [4] in the past and found it too time consuming and addictive. How a Preference for Low Quality could lead to Microsoft I think that everyone who has any significant experience in the computer industry has encountered companies that have large areas of low quality. This generally tends to be in large corporations as small companies can t afford the waste. In some large corporations Linux on the desktop is never considered, even when people are hired as Linux sysadmins and there are obvious productivity benefits to having the same OS on the desktop as on the servers (even if two desktop PCs are required so that proprietary software can be run on MS-Windows). Major wrote a good satire of the corporate IT non-working culture with a comparison to medical work [5], it illustrates the principle of a coalition to ensure low quality. He later documented how he was sacked by the low quality coalition at a company that uses a lot of Microsoft software [6]. So it does seem that when customers don t care at all about the quality of the result it does help drive some sales for Microsoft. But that doesn t explain the market share that they have. It takes a lot of work to get Market Share without Quality Microsoft has spared no effort in gaining market share. Every possible effort including buying out small competitors, aggressively FUDing competition, using all manner of legal attacks (including the threat of patent suits), and deliberately breaking standards has been used. I think it s reasonable to assume that the MS senior management are not entirely stupid, they do so many things that are unethical and possibly illegal because they know that they need to do so to maintain their market share. The result is that the market capitalisation of MS is almost as high as that of Apple and Apple makes vastly superior products. Given the amount of effort that MS uses to keep market share it seems apparent that they aren t just relying on customers not caring about quality. Should users have to Understand Computers? My observation is that most users don t want to know much about how their computers work. The desire to understand computers seems to be about as common as the desire to understand cars, people just want to buy one that looks good and have it work. The difference is that cars are very compatible while computers aren t. Cars have the same controls in the same places and large parts of the design are specified by law so that they can t differ between models. Proprietary software is usually intentionally incompatible with other software (both open and proprietary) to try and gain a competitive advantage. Hardware is often incompatible due to the rapid developments in technology and the requirements for new interfaces to take advantage of new features. In concept it seems reasonable for someone who is about to spend $30,000 on a car and $1000 on a computer (for hardware and software) to spend 30 times longer considering which brand of car to buy. One could argue that more than 30 times as much consideration should be given to the car as most people can t afford to discard a car that they don t like. As people spend a few minutes considering which brand of car to buy they can be excused for spending a few seconds considering which type of computer to buy. But once a choice has been made about which software to use it s very difficult to change to something else, while in comparison it s easy to drive a car that was manufactured by a different company. So a poorly informed choice made at an early stage can have costly long-term affects when buying software. If we had mandated open standards for file formats and data interchange then users would be able to make choices that don t result in their data being locked in to some proprietary format. Such standards could be set through government tender processes, if every government agency was to only buy software that complies with open standards then the proprietary software vendors would scramble to make their products less incompatible. The result would be that bad choices in purchasing software could become learning experiences that result in better purchases in future instead of being a lock on users that forces them to keep using the same software that doesn t satisfy their needs. Conclusion I think that the best thing about the paper by Diego Gambetta and Gloria Origgi is that it highlights the issue of low quality. No-one wants to be considered a loser, so maybe this can encourage people to strive for high quality (or at least try to make their work suck a little less). Regardless of the conclusion they eventually reach, it s probably good for people to occasionally wonder do I suck? .

Next.

Previous.