It has been more than a month since I last wrote about my work on update-manager during this year s Google Summer Of Code and I am somewhat ashamed I wasn t able to provide you with updates more regularly. So first of all, yes, I did do some work and yes, there has been quite some progress. Basically both private and university stuff have kept me from writing and that s why I d like to start with this series of weekly updates today.
This series are meant to summarize what has happened during a week of writing code and give you an overview of what s happening. This first issue however will sum up the past month. So let me begin explaining what has happened since my last post. update-manager bazaar repository All code I have written so far is available through a public bazaar branch on launchpad.net. My branch s page can be found here and provides you with its history and of course instructions on how to obtain the code. The location is only temporary though, as I am going to move hosting over to alioth.debian.org. This is on my task list for next week. modular design I have ripped apart nearly all of update-manager and put it together in a more modular way, which should implementing new frontends or backends more easy, whilst also simplifying code maintenance. The new design consists of four major parts:

• The application class is responsible for parsing command line arguments, initializing all other components correctly and coordinate communication between the frontend and the backend.
• The backend itself is defined through the UpdateManager.Backend.BackendBase class and each implementation subclasses BackendBase. It is responsible for interacting with apt.
• The frontend is again defined through a base class, UpdateManager.Frontend.FrontendBase. This part of update-manager provides the userinterface, handles user input and starts operations accordingly.
• Lastly there is the distribution-specific part, which lives inside the UpdateManager.DistSpecific Python module and is defined by its own base class, DistBase.

backend implementation When I started working on update-manager it heavily relied on synaptic and used it to do the dirty-work. However, together with my mentor, mvo, I decided to drop synaptic support and rather concentrate on using python-apt. This means that the only backend implementation right now is a python-apt backend. The python-apt backend is currently a work in progress, but already includes some basic functionality. Right now it can (re-)load the package cache and package lists and is able to provide a list of packages which are upgradable to the frontend.
Whilst implementing these functions I noticed some shortcomings of python-apt itself, fixed those and got mvo included in his python-apt branch at launchpad. frontend implementation I started re-implementing the Gtk frontend as provided through current update-manager and right now it visualizes the package cache reloading process and provides users with a list of upgradable packages. However, that s pretty much all of the functionality it includes right now, which is why implementing more functions is pretty much on the top of my todo list. Additionally I have ported the text frontend, as included in Ubuntu, to the new modular system, and this frontend s code really shows how easy adding a frontend with the new modular design is. This frontend contains the same functionality as the Gtk frontend. distribution specific code The core described above does not include any distribution specific code anymore, which is the main focus of this project. The implementations of distribution-specific functionality contains classifiers for update categories for both Debian and Ubuntu, whilst I focused on getting things right with the Debian implementation for now. These classifiers allow the frontend to let the user know which kind of update they are about to install, like a security update, a recommended upgrade or a third-party (unofficial) upgrade. documentation As update-manager was poorly (read: hardly at all) documented I started documenting the API using sphinx. However, right now the generated documentation cannot be found anywhere yet. This should change as soon as an alioth project for update-manager has been created. next week s tasks I would also like to provide you with my task list for the coming week. The list, ordered by priority, is:

• Register an alioth project and move the bazaar branch to bzr.debian.org
• Generate an HTML version of the API documentation and put it on alioth.
• Implement changelog-fetching for the Debian-specific module and make use of that from within the code and the Gtk frontend.

As you can see this list is rather short. This can mainly be attributed to a few university assignments, and instead of providing a long list of tasks which I probably won t be able to finish I rather keep the list short and hopefully get things not on this list done too.

... you might be bitten by this Java bug rendering arcs as straight lines at large zoom levels.It looks like a classic to me: in order to improve rendering performance, you approximate arcs with straight lines at small resolutions (if it's just 2 pixels big, nobody will be able to tell the difference). Except of course, when you end up doing the same approximation at a large zoom value - of course a 100-pixel circle looks different from a 100-pixel diamond.Reported in 2005, still not fixed in current Java (we're in 2009 now).Sun is really slow at fixing Java bugs.See also a related Apache Batik bug report. Fortunately, this only applies to Java rendered graphics - SVG export, PDF, Postscript are all fine.

I m on vacation the 1rst two weeks of June in Manerba del Garda at Lago di Garda. If some dudes are around for beering / chatting / keysigning, feel free to contact me til Friday.
In case of urgent actions (for example #530271), I m without regular internet access there. Feel free to NMU if needed, but try to avoid breaking things. For secuirty related issues please coordinate with team@security.debian.org.
Since I m probably most of the rest of the week busy with organising our camping, please send a short ping, if I m not responding by time, please go ahead.

From my friends around who read my "running" posts, here are some news. Since my first marathon, I had a full week rest, then gradually began running again, first starting with 10 to 15 kilometers runs and then slowly reclimbing up distances to reach 23km 3 weeks after the marathon and 26km on May 3rd. Both these were done at a 5:35 to 5:40 mins/km pace, which seems to be the pace I can keep for a long time at this moment. On later weeks, I diminished efforts slightly, cutting off on distances, to concentrate on speed, so running distances between 7 and 12km, with a faster pace (between 5:15 and 5:30). That culminate in a very sharp 6km at 5:06 on a week day, after work (and that loop includes a very steep slope). The target for this "preparation" was indeed May 19th. I was on holidays all week long and that seemed to be a great occasion to try out something I wanted to do for quite a long time: run a marathon....alone. I completed it, moreover with a very great loop in Rambouillet forest, running the 42.2 kilometers in 4h22. Not that a great time (6:12/km), compared to the 4h10 of the Paris Marathon, but fairly interesting as that was done by running lonely...and also by running in woods, partly on very muddy paths. What's even more interesting is that I ended up nearly without cramps and had only to walk on the steepest slope....at km41 (about 20% slope for 200 meters). Note that I made a stop around km25 in order to eat dry fruits as, contrary to official marathons, in lonely marathons, you need to have your own food and beverage with you. So, well, the full time is around 4h30. Even more interesting: I was not tired at all the day after and, accordinng to my wife, much more "alive" than after Paris marathon..:). That probably explains why I could run again only 4 days after (a small but fast 6km) as well as today, 5 days after, for a 14km loop at 5:25/km despite the heat (my very best time on such a distance). So, Debconf runners, bring on your shoes in C ceres. I'll be there and ready..:-)

It would appear that debian.org rejected mail for maybe up to twelve hours from late yesterday afternoon (Central timezone) to some time shortly after I got up this morning. Things appear to be back to normal, so a big Thanks to the mail admins. If you happened to have sent me mail to my debian.org address during that time period, you may have gotten a hard reject ('550 Administrative prohibition') as did a test mail of mine. In this case mail may not be respooled, so please do send it again. My alternate address, formed by my first name followed by the family name and the commercial top-level domain, remained functional as a fallback.

Take the following URL:

http://my.server.tld/wow/summary/2009/05/21/?style=boxed

and imagine the following Python function:

def summary(request, year, month, day):
    response = HttpResponse()
    // do something
    return response

If you see a relation between the two, then Django is for you. The missing part is a URL mapping, a list of regular expressions, which transforms a URL into a keyword list, and maps it to a function (a HTTP view, as it s called):

(r'^wow/summary/(?P<year>\d 4 )/(?P<month>\d 2 )/(?P<day>\d 2 )/$',
    mysite.wow.summary,  'additional':'information' )

You do not have to use keyword arguments, positional parameters will work as well, and you can do anything you like within the powers of regular expressions, but the result is always along the same lines: When someone visits the above URL, Django calls the above summary( ) function (defined in the modules mysite/wow.py) and passes an object encapsulating the request, along with the components from the URL as parameters:

summary(request, 2009, 05, 21,  'additional':'information' )

It then renders the return value. The function can also throw an exception if it need to communicate e.g. a 404 error. Very clean, and there are even generic views if your data doesn t need any massaging, of if you just want to do standard things like generate PDFs. This is the core idea of Django, which I ve gathered from one talk and four hours of invested time. It s a truly thin layer, and its elegance made me bouncey. The QUERY STRING of the request will be passed in the request object, along with all the other CGI variables you re familiar with. Django provides a large number of helpers and shortcuts to save you from having to do the ugly work, including topics such as internationalisation, syndication, authentication, file uploads, and caching. In addition, Django gives you an object-relational_mapper to map the data into, and manage the data definition within your favourite RDBMS, a templating language designed to fit in with the whole philosophy, and widgets to work with forms. You do not have to use any of those. You are free to use any other Python module for the task, just as you can use all of the other features of and modules written for Python. This is particularly powerful in the context of the middleware layer. There s also a large number of pluggable applications and snippets for re-use. The only concern I have at this point is whether features like tagging are going to find their way into the framework, because tracking numerous external plugins for a site and keeping them working across versions reminds me of the nightmares I had with Zope and Plone (I stopped working with (and on) those before I had a chance to dive into version 3 of each). The fact that Django leaves data storage to other tools (in fact, it s entirely up to you) makes those nightmares seem further distant. All in all, I am excited to have had a chance to take a brief glance at the software, and I am looking forward to doing more with it. Unfortunately, it looks like it will take quite some time to wade through existing plugins and design the rest of an application that can finally replace the horrific dung-pile that is my homepage.

Ever since I bought this laptop I was quite content with it running Debian GNU/Linux (Lenny) and, except the sleep functionality not working (bug reported, but I have no answer for any new approaches) I have no other major issues.


Actually, the problems are partially solved since I am in a "pick your favourite bug" situation:

• with the 2.6.26 kernel from Debian the information about the power source is incorrect (kernel bug which goes away for me right since 2.6.27)
  
• while with a newer (>=2.6.27) pristine kernel, power information is accurate, but the entire bluetooth stack (init.d scripts and apps) needs a restart after a hibernate/resume cycle to work again; I added my info in the corresponding upstream bug and I hope it gets fixed

Still, I am really curious, since the Debian Kernel Team policy is to accept only patches accepted upstream, theorecally that would mean newer upstream kernels should work wrt that bluettoth issue, assuming this isn't a regression (it doesn't look like one from my experience).

So what is present in Debian kernels that isn't in the pristine ones that makes BT work afer resume? If you know the answer, please add it to the bug report.


On the Debian side of the kernel, which are the chances that newer Lenny kernels will include the power fixes necessary for MSI laptops to report power related info correctly? I know, I know, I should probably report a bug, but I want to know first if there will be a lenny-n-half release, otherwise it doesn't make sense.

In Myanmar, Aung San Suu Kyi and 2 000 monks and activists are being detained, as the miltary junta in the country presses new, ridiculous charges against Suu Kyi a few days before the end of her house arrest. Suu Kyi, recipient of the Nobel Peace Prize and proponent of democracy, is leader of the opposition and the greatest threat to the junta s grasp on power. Given the inhumane conditions imposed upon the Burmese, and the ruthless slaughter of those who do not play by the military s rules, it is not hard to imagine that she would win any fair election, as soon as she would get a chance to run. However, if the junta are successful in pushing charges Suu Kyi is accused of breaching the house arrest after an American man sneaked into her house uninvitedly she could be locked up until after the elections in 2010, effectively disarming the threat she poses to the military regime. She is threatened to be kept in the notorious Insein Prison without medical care. As she is suffering from serious illness, this puts her life at risk. I do not usually campaign for political topics, but I ve been to Myanmar and this issue touches me deeply. The people in Myanmar need our support. Even though the 2007 Burmese anti-government protests seriously weakened the military and brought hope to the lives of millions that have been oppressed for years, the events quickly vanished from the media and allowed the junta to regain control. We must not let that happen again. It is of utmost importance to show our support and not look away. One way of doing so is by increasing visibility. If you care, please write to your newspapers, publish on your blog, and do everything else you can to raise awareness. Another way to help the Burmese people is through signing the avaaz.org petition to UN Secretary General Ban Ki-moon. Best to do all of the above! Unfortunately, avaaz.org force-subscribes everyone who signs a petition to their newsletter, which may well not be what you want. However, please do not let it stop you from signing the petition. You can later unsubscribe from the (infrequent) newsletter. I have informed the staff members of this concern. If you would like to voice your opinion too, write to info t avaaz dot org and encourage them to make the newsletter subscription optional. But make sure to also sign the petition, please.

• Marco d Itri about adding support for yet-unknown-to-kernel-drivers devices. More specifically: how to make udev tell ftdi-sio that it should accept a certain VendorID/ProductID combination in addition to previously supported ones.
• Thomer M. Gil about ICMPTX, a combination made from proxy and server that tunnels IP through ICMP. Nice thing if you need a connection from some NATed and heavily firewalled location I guess. (Thanks go to Runa Sandvik for finding this.)
• Non-technical in a way, but still a nice link: Image of the Shuttle transiting the sun, thanks to Pablo Lorenzzoni for this link.
• Another post by Marco d Itri that explains how to install Debian on a SheevaPlug. Not that I need this info now, but it might come in handy some day.

That s it for now, will update the post if I find more interesting links in the next few days.

Is there any way to provide an alternate CSS stylesheet for GoLive CS2 only, not for regular browsers? Because there are some things in that layout that are too difficult for the GoLive renderer, it doesn't display them right. The pages are still editable (just plain XHTML), it's just not looking right in GoLive (advanced CSS).The site already has alternate stylesheets for browsers such as the broken Internet Explorers, so if I could convince GoLive to use their stylesheet it might be looking a lot better in the editor, too ...I am aware that GoLive CS2 has been abandoned in favor of DreamWeaver. Still it's going to be used in a project I help with the web templates.(Other options would be Kompozer and Amaya, but none of them seem really fit for production use: Amaya was just removed from Debian because it had some security issues and the maintainers had the impression the code was such a mess that there will be much more such issues. And Kompozer seemed to be a mostly dead branch of a Gecko hack (although there has been a new alpha release this year) ... is there some reliable opensource non-source HTML editor that I'm missing?)P.S. Sorry, no comments in this blog. Use Email: erich AT debian ORG

So, today I turn 31. Fortunately I've had a year to learn that the thirties change nothing, and looking back, I can easily say I've enjoyed one of the best years I remember. Today, a bit of protesting in the Pla a de l'Ajuntament against the old menaces of the Valencian Botanical Garden, and just after that, beer time around the Cedre area. The amount of email, Facebook stuff and calls I've been getting today since I woke up is impressive. Thanks everyone! ;)

To the extent possible under law, Antti-Juhani Kaijanaho has waived all copyright and related or neighboring rights to
Asynchronous transput and gnutls. This work is published from Finland. GnuTLS is a wonderful thing. It even has a thick manual but nevertheless its documentation is severely lacking from the programmer s point of view (and there doesn t even seem to be independent howtos floating on the net). My hope is to remedy with this post, in small part, that problem. I spent the weekend adding STARTTLS support to the NNTP (reading) server component of Alue. Since Alue is written in C++ and uses the Boost ASIO library as its primary concurrency framework, it seemed prudent to use ASIO s SSL sublibrary. However, the result wasn t stable and debugging it looked unappetizing. So, I wrote my own TLS layer on top of ASIO, based on gnutls. Now, the gnutls API looks like it works only with synchronous transput: all TLS network operations are of the form do this and return when done ; for example gnutls_handshake returns once the handshake is finished. So how does one adapt this to asynchronous transput? Fortunately, there are (badly documented) hooks for this purpose. An application can tell gnutls to call application-supplied functions instead of the read(2) and write(2) system calls. Thus, when setting up a TLS session but before the handshake, I do the following:

                gnutls_transport_set_ptr(gs, this);
                gnutls_transport_set_push_function(gs, push_static);
                gnutls_transport_set_pull_function(gs, pull_static);
                gnutls_transport_set_lowat(gs, 0);

Here, gs is my private copy of the gnutls session structure, and the push_static and pull_static are static member functions in my sesssion wrapper class. The first line tells gnutls to give the current this pointer (a pointer to the current session wrapper) as the first argument to them. The last line tells gnutls not to try treating the this pointer as a Berkeley socket. The pull_static static member function just passes control on to a non-static member, for convenience:

ssize_t session::pull_static(void * th, void *b, size_t n)
 
        return static_cast<session *>(th)->pull(b, n);
 

The basic idea of the pull function is to try to return immediately with data from a buffer, and if the buffer is empty, to fail with an error code signalling the absence of data with the possibility that data may become available later (the POSIX EAGAIN code):

class session
 
        [...]
        std::vector<unsigned char> ins;
        size_t ins_low, ins_high;
        [...]
 ;
ssize_t session::pull(void *b, size_t n_wanted)
 
        unsigned char *cs = static_cast<unsigned char *>(b);
        if (ins_high - ins_low > 0)
         
                errno = EAGAIN;
                return -1;
         
        size_t n = ins_high - ins_low < n_wanted
                ?  ins_high - ins_low
                :  n_wanted;
        for (size_t i = 0; i < n; i++)
         
                cs[i] = ins[ins_low+i];
         
        ins_low += n;
        return n;
 

Here, ins_low is an index to the ins vector specifying the first byte which has not already been passed on to gnutls, while ins_high is an index to the ins vector specifying the first byte that does not contain data read from the network. The assertions 0 <= ins_low, ins_low <= ins_high and ins_high <= ins.size() are obvious invariants in this buffering scheme. The push case is simpler: all one needs to do is buffer the data that gnutls wants to send, for later transmission:

class session
 
        [...]
        std::vector<unsigned char> outs;
        size_t outs_low;
        [...]
 ;
ssize_t session::push(const void *b, size_t n)
 
        const unsigned char *cs = static_cast<const unsigned char *>(b);
        for (size_t i = 0; i < n; i++)
         
                outs.push_back(cs[i]);
         
        return n;
 

The low water mark outs_low (indicating the first byte that has not yet been sent to the network) is not needed in the push function. It would be possible for the push callback to signal EAGAIN, but it is not necessary in this scheme (assuming that one does not need to establish hard buffer limits). Once gnutls receives an EAGAIN condition from the pull callback, it suspends the current operation and returns to its caller with the gnutls condition GNUTLS_E_AGAIN. The caller must arrange for more data to become available to the pull callback (in this case by scheduling an asynchronous write of the data in the outs buffer scheme and scheduling an asynchronous read to the ins buffer scheme) and then call the operation again, allowing the operation to resume. The code so far does not actually perform any network transput. For this, I have written two auxiliary methods:

class session
 
        [...]
        bool read_active, write_active;
        [...]
 ;
void session::post_write()
 
        if (write_active) return;
        if (outs_low > 0 && outs_low == outs.size())
         
                outs.clear();
                outs_low = 0;
         
        else if (outs_low > 4096)
         
                outs.erase(outs.begin(), outs.begin() + outs_low);
                outs_low = 0;
         
        if (outs_low < outs.size())
         
                stream.async_write_some
                        (boost::asio::buffer(outs.data()+outs_low,
                                             outs.size()-outs_low),
                         boost::bind(&session::sent_some,
                                     this, _1, _2));
                write_active = true;
         
 
void session::post_read()
 
        if (read_active) return;
        if (ins_low > 0 && ins_low == ins.size())
         
                ins.clear();
                ins_low = 0;
                ins_high = 0;
         
        else if (ins_low > 4096)
         
                ins.erase(ins.begin(), ins.begin() + ins_low);
                ins_high -= ins_low;
                ins_low = 0;
         
        if (ins_high + 4096 >= ins.size()) ins.resize(ins_high + 4096);
        stream.async_read_some(boost::asio::buffer(ins.data()+ins_high,
                                                   ins.size()-ins_high),
                               boost::bind(&session::received_some,
                                           this, _1, _2));
        read_active = true;
 

Both helpers prune the buffers when necessary. (I should really remove those magic 4096s and make them a symbolic constant.) The data members read_active and write_active ensure that at most one asynchronous read and at most one asynchronous write is pending at any given time. My first version did not have this safeguard (instead trying to rely on the ASIO stream reset method to cancel any outstanding asynchronous transput at need), and the code sent some TLS records twice which is not good: sending the ServerHello twice is guaranteed to confuse the client. Once ASIO completes an asynchronous transput request, it calls the corresponding handler:

void session::received_some(boost::system::error_code ec, size_t n)
 
        read_active = false;
        if (ec)   pending_error = ec; return;  
        ins_high += n;
        post_pending_actions();
 
void session::sent_some(boost::system::error_code ec, size_t n)
 
        write_active = false;
        if (ec)   pending_error = ec; return;  
        outs_low += n;
        post_pending_actions();
 

Their job is to update the bookkeeping and to trigger the resumption of suspended gnutls operations (which is done by post_pending_actions). Now we have all the main pieces of the puzzle. The remaining pieces are obvious but rather messy, and I d rather not repeat them here (not even in a cleaned-up form). But their essential idea goes as follows: When called by the application code or when resumed by post_pending_actions, an asynchronous wrapper of a gnutls operation first examines the session state for a saved error code. If one is found, it is propagated to the application using the usual ASIO techniques, and the operation is cancelled. Otherwise, the wrapper calls the actual gnutls operation. When it returns, the wrapper examines the return value. If successful completion is indicated, the handler given by the application is posted in the ASIO io_service for later execution. If GNUTLS_E_AGAIN is indicated, post_read and post_write are called to schedule actual network transput, and the wrapper is suspended (by pushing it into a queue of pending actions). If any other kind of failure is indicated, it is propagated to the application using the usual ASIO techniques. The post_pending_actions merely empties the queue of pending actions and schedules the actions that it found in the queue for resumption. The code snippets above are not my actual working code. I have mainly removed from them some irrelevant details (mostly certain template parameters, debug logging and mutex handling). I don t expect the snippets to compile. I expect I will be able to post my actual git repository to the web in a couple of days. Please note that my (actual) code has received only rudimentary testing. I believe it is correct, but I won t be surprised to find it contains bugs in the edge cases. I hope this is, still, of some use to somebody

I love drinking water, talking about the future of water, and am especially passionate about the need for clean water. I hate the idea that we have polluted our oceans, our lakes, our rivers and this water of life, is of course, indispensable. Today, I found out about this amazing project, Water Canary, a project out of NYU s ITP, which is building low cost sensors to detect toxins in water (in Africa in specific but it seems like every corner of the earth is quite eligible for such toxic sensing). All I can say is WoW. There is just so much potential for this type of project of informational visibility and I can t wait to see it develop.

I ve previously written about memory squeeze problems in a Xen Dom0 when large amounts of memory were assigned to DomUs [1]. In summary the Dom0 would have problems if started with default options and the majority of the RAM was later assigned to DomUs, but if the memory of the Dom0 was limited by the dom0_mem parameter to the Xen kernel then things would work well. Fatal server error:
xf86MapVidMem: Could not mmap framebuffer (0xfae80000,0 80000) (Invalid argument) I have since found another exciting bug with Xen. I was in the process of upgrading an AMD64 workstation to using Xen so that I could test other versions of some software in the background. The first stage was to install the Xen kernel and the Xen enabled Linux kernel and boot the machine. Unfortunately I then received the above message when trying to start the X server. I discovered the solution to this in the comments section of Different Colours blog post about Virtualisation on Lenny [2]. It seems that there is a problem gaining mmap access to MMIO memory regions in Xen and that restricting the memory of the Dom0 is a work-around. My AMD64 workstation has 3G of RAM because the motherboard can t support more than 3.25G and buying 4G of RAM to have 3.25G usable would be an expensive way of doing it. So I used dom0_mem=1500M and now X works again. I have yet to discover if anything strange and exciting happens when I create DomUs on the machine. I don t have any immediate plans for running Xen on the machine. It s main uses at the moment is torcs (a slightly realistic 3D car racing game), supertuxkart (a cartoon 3D car racing game), and mplayer so it doesn t really need a lot of RAM. I like to keep my options open and have all my machines capable of virtualisation apart from routers.

• [1] http://etbe.coker.com.au/2008/08/25/xen-and-linux-memory-assignment-bugs/


• [2] http://diffrentcolours.livejournal.com/814563.html

This post is being made from my EEEEEeeee PC, using a 3G modem plugged into the USB port. The fact that I'm sat on my sofa, within easy reach of both a network cable and multiple WiFi access points is irrelevant! I started my adventure yesterday evening, getting pretty annoyed along the way that that it wasn't just plug and go. It turns out I was suffering from two problems:

• The USB device itself alternates between being a modem and being a dumb USB storage device (full of Windows software).
• My copy of Network-Manager was too old.

In short from my Lenny installation I had to upgrade to Sid to get a copy of Network Manager with a "Mobile Broadband" section in its preferences. (I looked for backports, to no avail, and I didn't have the patience to make one mysefl). The new connection looks like this:

Number	*99#
Username	[blank]
Password	password
APN	m-bb.o2.co.uk
PIN	0000
PUK	0000

Once I added the connection discovered the USB modem device (/dev/ttyUSB0) just didn't work - and I learned about the dual-nature of the device. Thankfully switching is nice and easy "apt-get install usb-modeswitch" then:

# disable storage
usb_modeswitch -v 12d1 -p 1003 -d 1
# enable modem
usb_modeswitch -v 12d1 -p 1003 -H 1

Once that was done the connection worked almost immediately. (I just had to upgrade to a 2.6.29 kernel because I got panics on the Lenny kernel; something the upgrade installed but I'd previously ignored. Kernels: Bane of my life.) Update: I do see some kernel weirdness talking about timeouts talking to the USB-serial device. Perhaps something to investigate in the future. Anyway running a 3G O2 PAYG (pay as you go) modem on Debian, on an EEEPC is possible, it is justfiddlier than I had expected, and it required an upgrade to Sid - since Lenny didn't have a network manager with mobile broadband support. For google's benefit the modem is described by O2 as a "mobile broadband USB modem - E160". This appears under lsusb as :

Bus 001 Device 005: ID 12d1:1003 Huawei Technologies Co., Ltd.
E220 HSDPA Modem / E270 HSDPA/HSUPA Modem

Hope that helps somebody else spend less than 5 hours getting it working. I guess the friends who said "It just worked" were running Ubuntu and so had a slightly newer network manager by default - and possibly their modems didn't need to toggle between "dumb storage" and "actual modem" modes. Anyway it works now, and even though it was fiddly the issue wasn't insurmountable. I'm just a little grumpy because I've gotten used to a world in which Debian just works - the last time I struggled to get new hardware toys playing nice was .. a long time ago. ObFilm: Pretty In Pink

Nir Aides, the author of the excellent winpdb debugger, sent me the following email on September 21, 2008, so I asked him if I can copy his email and reply in form of a blog post (so that other people can comment and join the discussion) and he agreed. It took me almost a year to reply, but I made it. :)

Hi Ondrej,

How are you?

I am about to publish a new free software project, a new simple PHP framework, and I am interested in your advice.

You started SymPy and were able to make other people join you and develop it with you.
How did you do it?
How did it happen?
Did you actively call for other people or they spontaneously showed interest and joined you?
Are the other major contributor people who were your friends before you started the project?
Did you need to create or manage the project in a particular way to make it attractive to other people?
Are there things you are aware of that promote collaboration or demote it?

I was never successful in doing the same with Winpdb, which while it became reasonably popular, no one has ever joined me to develop it, except for a notable tutorial contribution by Chris Lasher which was developed independently.

Now with the new project, I am wondering what are my chances of making other people try it and take it on. On the one hand it is a new and fresh code base in an interesting field, on the other hand, why would anyone bother to spend their energy on this new project when they have Symfony or Drupal?

What do you think?

BTW, Ohloh believes you have a median of 19,000 lines of changed code per month since the start of their log. Can this be true? Is this humanly possible? According to it SymPy has over 1,000,000 lines of code? I can't understand these numbers. Winpdb has about 25,000 lines after 3 years of development. And from my experience 1,000,000 lines of code projects need about 20-50 full time developers to work on for 2-5 years which is about 40-250 man years. And as if this is not enough you are listed as owner in a dozen other projects in Google code and have enough time to become an awarded scientist. How is this possible?

http://www.ohloh.net/p/sympy/contributors/

BTW2, do you still use Winpdb? If you find yourself using it less, can you say what are the reasons, or what it would take to make it more useful?

BTW3, How is SymPy doing?

Cheers,
Nir

So my most honest answer how to run a successful opensource project is: I don't know.

But nevertheless I tried to summarize some of my ideas and experience and some guidelines that I try to follow, maybe it will be useful to you Nir, or anyone else.

First of all, there has to be a public mailinglist (easily accessible), public bug tracker, nice webpage, easy to find downloads, frequent releases (once a month is good, but in the worst case at least 4 times a year) and a set of guidelines to follow in order to contribute. So that's a must, if the project doesn't have the above, it's almost impossible to become successful. However, that is just a start, just a playground. There are still many projects that have the above and yet they totally fail to attract developers.

So I think the most important principle is that I always think how to employ other people in what I do. If I have some plan in my head how to do something, e.g. how to move some things forward, I always create exact steps and put it to issues, or our mailinglist, so that each step can be done by someone who is completely new to sympy. So I try to look at things from other people's perspective and think -- ok, I quite like this SymPy project and I'd like to get this done (for example a new release, or something fixed, or implemented), but I have no idea how to start and what exactly needs to be done.

So what I try to do if someone comes to our list and asks for something, is that I create a new issue for it and think how I would fix it if I had time. Then write the necessary steps in the issue and invite the submitter to fix it and I offer help with explaining anything and guiding. Now there are two things that can happen. Either the submitter has time and a will to go forward and in this case he starts wrestling with it and whenever he has some code or a question, I need to find time, review it and offer some way out. Or the submitter is too busy, in which case the instructions simply rest in the issues and the next time someone asks for the feature, the instructions are already there. I don't have estimates how frequent either case is.

When I am working on something myself, I try not to code privately, but also put up issues first and put the steps needed in the issues, so that it's easy for other people to join in.

In general, the most precious value for me is the fact that someone else had to sit down at his computer and wrote the patch. So I do everything possible to get new (or more) people interested in the development. Some people think that only super programmers can do a decent job and it's useless to invest time in people that may just have started with Python. They are wrong. Among the SymPy developers (around 65 people total have contributed patches so far at the time of writing this post), we have all kinds of people. We have people from high school, we have a retired US army engineer, we have physicists, mathematicians, biologists, engineers, teachers, or just hobbyists, who do it for fun. Unfortunately, we do not have many women (I think no patch that made it into sympy was contributed by a woman, but I may be wrong), so if anyone has any ideas how to get more women involved, let me know (I know we have several women fans, so that's a good start:). We have people whose first open source project they ever contributed to was sympy and people who are new to Python.

Many times the first patch that a new potential developer submits is not perfect, usually it's faster for me to write it myself, than to help with the first patch, however my rule is to always help the submitter do that. Sometimes he sends a second patch, or a third, and usually it needs less and less work on my side and it already pays off, because he is then able to fix things himself, if he discovers a bug and sympy has just won a one more contributor.

So I came to the conclusion that all that is needed is an enthusiasm. You don't even have to know Python (as you can learn all these things on the way) and you can still do useful things for us and really spare our time.

To answer another question from Nir's email, SymPy has about 130000 lines of code and another about 20000 lines of tests, so I think those stats are wrong. Also the changed lines of code is in my opinion wrong, we usually have about 250 new patches per release (this depends how often we release and other things).

Yes, I am involved in couple other projects, e.g. Debian, Sage, ipython, scipy, hpfem.org (and couple more), basically everything that has to do with numeric simulation and Python, but my activity there varies. The most time consuming thing in the last couple years was definitely school, I was finishing my master in Theoretical Physics in Prague and then moved to the Nevada/Reno and I just finished my first semester here at PhD in Chemical Physics, and sometimes it was just crazy, e.g. I finished teaching at 7pm and instead of going home and sleep, I stayed in my office, fixed 10 sympy issues that were holding off a release, finished at 1am, went home (by bike, since I don't have a car yet), slept couple hours and then did just school again for a week, other people reviewed the issues in the meantime, and then I made the release (instead of sleeping again). In the last semester it was not unusual that I got home at 1am every week day, then slept most of Saturday to catch up, on Sunday I did some laundry and shopping, and the rest of time I did grading and homeworks for all my classes and teaching, no time for anything else (e.g. no friends, no girls, no rest, no hobby, no opensource stuff, nothing). So sometimes one has to work pretty hard to get through it, but fortunately it's behind me finally, if all goes well, I should be just doing research from now on and have a real life too. Also I am sorry I didn't manage to reply sooner. :)

To answer the other questions:

Are the other major contributor people who were your friends before you started the project?

No, not a single major contributor was my friend before I started the project. Every single one of them become a developer using the procedure I described above, e.g. first showed on the list or in the issues, and maybe even the very first patch was not a high quality one (and if I was stupid and arrogant, or didn't see the big potential, I would just ignore them). But when given a chance, they became extremely good developers and sympy would simply just not be here without them.

Did you actively call for other people or they spontaneously showed interest and joined you?

I very much encourage everyone to contribute, but the initial interest must be in them, e.g. they at least have to show around the mailinglist/issues, so that I know about them. But once I know they are interested in some issue, yes, I try to invite them to fix it, with my help.

One observation I made is that I have to always think in the spirit "how to earn new money, not how to spare the money I already have", e.g. when applied to sympy, how to get new developers, how to develop the new great things etc. Even if I am super busy as I was, I still have to think this way. Once I start thinking how to conserve and preserve what we already have, I am done, finished and that's the road to hell.

If I am open, positive, full of energy, I can see people joining me and we can do great things together. It probably sounds obvious, but it was not for me, when for example some people I worked with, started their own projects, when I got busy, and started to compete, instead of helping sympy out. And I felt betrayed, after so much work that I invested into it and started to become protective. And then I realised that's wrong. I can never stop other people do what they want to do. If they want to have their own project, they will have it. If they don't want to help sympy out, they won't (and what is more important, there is nothing wrong with either of that). It's that simple and being protective only makes things worse.

There is also a question of the license that you use for the project, e.g. one should basically only choose between BSD (maybe also MIT or Apache), LGPL and GPL (there are also several versions of the GPL licenses). Unfortunately the fact is, that there are people who will never contribute a code under a permissive BSD license (because it's not protecting their work enough) and there are also other people who really want to code to be BSD (or other permissive license) so they can sell it and they don't need to consult with lawyers what they are or aren't allowed to do and also so that they can combine it with any other code (opensource or not). It also depends if one wants to combine (and distribute) other codes together. So choosing a license is also important. I believe that for sympy BSD is the best and for other projects (like Sage) GPL is the best and one has to decide on a case by case basis. For Winpdb, I would make it BSD too, since you can get more people using it.

To conclude, SymPy is a little more than 2 years old, and it has been a great ride so far and more things are coming, e.g. this summer we have 5 Google Summer of Code students and people are starting it to use in their research and we plan to use it in our codes at our group here in Reno too, so things look promising. I am really glad, we managed to build such a community, so that when I am busy, as I was the last semester, other people help out with patches, reviews and other things, so that the project doesn't stall and when I got rid of my school duties now, we can move things forward a lot.

So maybe you can get inspired by some of the ideas above. I am also interested in any discussion about this (feel free to post a comment below, or send me an email, or just write to a sympy list about what you think).

Because of the recently announced attack against the SHA-1 digest algorithm, I finally decided to move away from my old 1024-bit DSA OpenPGP key, landing to a shiny new 4096-bit RSA one. The old key will continue to be valid for some time, but I prefer all future correspondence to come to the new one. I would also like this new key to be re-integrated into the web of trust. I'm attaching a file with
the complete text of the transition, clearsigned by both keys to certify the transition. You can find the transition statement also on http://www.hanskalabs.net/key-transition_20090509.txt . You can check the validity of the file with something like:

$ gpg --decrypt key-transition_20090509.txt   gpg --decrypt

If you need to transition your key too, and you're one of my signees, then let's please coordinate, so that we have the new keys cross-signed, without spurious signatures fromgoing-to-be-revoked keys Thank you for your help.

Attachment	Size
key-transition_20090509.txt	4.03 KB

Adrian von Bidder raises an interesting discussion on why public health care is difficult. Basically, it comes back to the challenge:

How much money should be spent on this person's illness? which is a very, very bad question. We try to sidestep it by only taking about statistics etc., but no matter how you look at it, you either do 20'000  per week medications for 80 year old patients who will die soon anyway and get a system society can't pay, or you don't and you get a system where the rich are better off than the poor.

The problem is, the majority of health costs are in the last few years of a persons life: when the body is breaking down. If viewed from the problem of health-care funding vs. eg. education, you have an ethical dilemma: is it worth it ? Here there are no easy answers. Nick Bostrom wrote a very interesting and persuasive essay a while back, The fable of the Dragon Tyrant. In short, we are getting somewhere with regenerative medicine While cures are hard to come by at this stage, there is a growing realization in medicine that senesence, what we used to call "old age", is curable. But we have a lot of psychological defensive mechanisms to help us cope with the carnage of old age that make us deny the problem: if we look afresh at "old age" knowing it to be curable, any delay in doing so is abhorrent. From a public, societal perspective, we spend a fortune every year on health. But we do so in a very disjointed way: we pay colossal amounts for health care, but also for basic health science: in the US, for example, the budget for the National Institutes of Health is about 30 billion dollars. Nearly twice that of NASA. Similar figures are spent in Europe, but there is this strange gap in the middle: the drug companies and medical industry. We start the development of new drugs with public money, and we buy the drugs with public money, but the choice of what drugs and treatments are developed are left to private industry, that is, a profit motive. This results in many cases in expensive treatments to treat symptoms rather than necessarily cure the problem. Faced with the huge costs of geriatric medicine and senesence, the response should be co-ordinated: funding a cure will be expensive, but save a fortune (Think, instead of a pensioner slowly dying and 'being a drain on resources', of 'experienced citizen in the prime of their productive, tax-paying years'). Instead of funding the basic science alone, we should be funding the complete drug and treatment development publically, only farming out the actual manufacture to private industry. It is silly to spend a fortune to keep merely keep someone alive at the age of eighty or so, when we know for a larger investment (bigger than a private company can do), we can cure them properly. The solution to the dilemma, then, is to stop thinking of the elderly being a drain, but actually applying our public efforts in a co-ordinated manner to solving the carnage of old age. Tags politics, health

Cooperatives-SW (which I m currently helping to direct) has been approved for Co-operative Enterprise Hub funding for 2009. Well done to Viv and the rest of the hard-working team who have been doing miracles for co-ops in this region with relatively little funding for so long now. (But if you ve got a pot of money to donate, let me know, please.) Industrial and Provident Societies are the traditional form for registering UK Co-operatives, but there are a number of things in its 1960s law which aren t reflected in general company law (which was mostly updated since the mid-1980s). For starters, IPSes are registered with the FSA, whose website isn t as good as Companies-House WebCHeck for Limited companies. Cooperatives-UK are pressing for the law to be modernised. C-UK sent us a status update (which I can t find on their website yet) this week. Amongst other things, they re seeking to introduce provisions to allow online registration and filing for IPSes. I ve asked them to take the opportunity to press for similar provisions for LLP Co-ops, or directing Companies House to act on any such existing provisions. At the moment, it seems unfair that LLPs are excluded from online filing, so paying double for routine administrative filings and relying on the increasingly dysfunctional Royal Mail. Finally, the next Cooperatives-SW print/PDF newsletter should be out next week. Hope to see lots of you at the AGM in Plymouth on 23 June!