Search Results: "pm"

27 April 2010

Matthew Garrett: Radeon reclocking

Alex released another set of Radeon power management patches over the weekend, and I've been adding my own code on top of that (Alex's patches go on top of drm-next, mine go on top of there). I've left it stress-testing for a couple of hours without it falling over, which tells me that it's stable enough that I can feel smug. This is a pleasing counterpoint to the previous experiences I've been having, which have been rife with a heady mixture of chip lockups or kernel deadlocks. It turns out that gpus are hard.

There's a few things you need to know about gpus. The first is that if they're discrete devices they typically have their own memory controller and video memory. The second is that there's an impressive number of ways that you can end up touching that memory. The third is that they tend to get upset if something tries to touch that memory and the memory controller is in the middle of reclocking at the time.

The first and most obvious use of video memory is by the gpu itself. Accelerated operations on radeon are carried out by sending a command packet to the command processor. This is achieved by sharing a ring buffer between the cpu and the gpu, with the gpu reading packets out of that ring buffer and performing the operations contained within them. Many of these operations will touch video memory (that being the nature of most things you want a gpu to do), and if that happens bad things occur. Like the card locking up and potentially taking your PCI bus with it.

So, obviously, we don't want that to happen. The first thing we do is take a mutex that blocks any further accelerated operations from being submitted by userspace. Then we wait until we get an interrupt from the gpu telling us that the display engine has gone idle. The problem here is that we don't have a terribly good idea of how many more operations there are to complete and we don't know how long each of those operations is going to take, but this is less bad than some of the alternatives[1]. Jerome Glisse has some ideas on how to improve this to require less waiting, but the effects should still be pretty much invisible to the average user.

So we've stopped the command processor touching ram. Everything's good, right?

Well, not really. The obvious problem is that users typically want to display something, so there's a separate chunk of chip that's repeatedly copying video memory over to your monitor. That's got to go too. Thankfully, there's a convenient bit in the crtc registers that lets us turn that off, but the pretty unsurprising downside is that your screen goes blank while that's happening. So we don't want to do that. Instead, we try to perform the reclock while there's nothing being displayed on the screen - that is, while we're in the screen region where a crt's electron gun would be scanning back from the bottom of the screen to the top. It turns out that rather a lot of display assumptions depend on this happening even if there's no crt, no electron gun and no thick sheet of glass with a decent approximation of vacuum behind it, so we get to do this even if we're displaying to an LVDS. And we have about 400-500 microseconds to do it - an almost generous amount of time.

So we ask the hardware to generate an interrupt when we enter vblank and then we reclock. Except the hardware has an irritating habit of lying - sometimes we get the interrupt a line or two before vblank, sometimes we get it after we've already gone out the other side. Vexing, and not entirely solved yet - so sometimes you'll still get a single blank frame during reclock. But there are plans, and they'll probably even work.

At this point the acceleration hardware isn't touching the memory and the scanout hardware isn't touching the memory. Except it still crashes under some workloads. This one took me longer to track down, but the answer turned out to be pretty straightforward. Not all operations are accelerated. When they're not accelerated they have to be done in software. That means that the CPU has to write to the video memory itself. I'm sure you can see where this is going. This was fixed without too much trouble once I'd finished picking through the driver to work out every location where objects might be mapped into the CPU's address space, at which point it's a simple matter of unmapping them and blocking the fault handler from remapping them until the reclock is finished. Linux, thankfully, has lots of synchronisation primitives. And now everything works.

Except when it doesn't. This took a final period of head scratching, followed by the discovery that ttm (the memory allocator used by radeon) has a background thread that would occasionally fire to clean up objects. And by clean up objects, I mean change their mapping - which means updating their status in the gart, which means touching video memory. So, let's block that. And that tripped me off to the fact that even if it couldn't submit new commands, the CPU could still create or destroy objects - with the same consequences.

So, once all of these are blocked, video memory is quiescent and we can do what we want. And we do, at least once I'd sorted out the bits where I was taking locks in the wrong order and deadlocking. Depending on the powerplay tables available on your card we'll chose different rates and so your power savings will vary heavily depending on the values that your vendor provided, but the card I'm testing on sees a handy 30W drop at idle. Right now we're only changing clocks and not dropping voltage so there's potentially a little more to come.

While getting this stable was pretty miserable, the documented entry points for clock changing made a lot of this easier than it would otherwise have been. It's also probably worth noting that Intel's clock configuration registers are entirely missing from any public documentation and the dirver Intel submitted to make them work in their latest chips appeared to have been deliberately obfuscated, so thanks to AMD for making all of this possible.

[1] It's possible to insert various command packets that either indicate when they've passed or stall until a register value gets updated, but these either cause awkward problems with the cp locking or mean that the gui idle functionality never goes idle, so they're not ideal either.

Michal Čihař: Let the fun begin

We've yesterday announced students accepted this year to Google Summer of Code. The number of our projects is increasing every year and the number of applications grows even more. So it was hard to choose this year knowing that we can mentor only limited amount of students (we would happily do two or three more projects, but we simply lack manpower to mentor them). So the students are chosen and I hope they will produce good results and all phpMyAdmin users will benefit from their contributions. If you want to follow the projects more closely, Planet phpMyAdmin is the right place, first student blog posts are already appearing there.

Diego Escalante Urrelo: Summer lessons

Over the last few months I ve been working for Igalia, as an intern, fixing regressions in Epiphany, which extends to WebKitGTK+ sometimes. Surely a dream job: working on my favourite projects, on a great company, surrounded by great teammates and friends. Igalia: Free Software Engineering I m happy about it, really really happy. I love this job, totally, completely!. I ve had the chance to learn a lot. Here I d like to share some things I have learned so far, I look forward to post again with more ideas, but meanwhile here you have two. Different timezones are hard The time when I find most of the Igalia crew online is between 2am and 12pm. Of course this doesn t mean you can t find them past 12pm, but it s already 7pm or 8pm in Europe then. I m on UTC-5 and Spain is on UTC+1 or (now) UTC+2. You probably agree that asking anyone to wake up at 7am in summer is unrealistic. Luckily, Igalia doesn t make me pass a turing test everyday at a fixed time. This rocks.

valpo
Valpara so, Chile I love it when people understand that a happy hacker working at midnight is better than an unhappy hacker working on a set in stone schedule. Kudos to Igalia for that. Your code should explain and defend itself My written expression teacher says Your text should be good enough to explain and defend itself . This applies to code too. I confirmed this at the expense of Xan s patience. It s a common situation: when the maintainer reviews your patch you are not around to explain it, or present the rationale you put into the change. The solution? well, simple, your patch and commit log should explain by themselves.

Felix, chillin
Don t just post a patch and do this.
Review it yourself. I saw, after realising how much ping-pong Xan and I had to play to get a patch in, that my patches lacked a harder review by myself before being posted. You have to be your first reviewer. Be a severe judge of your patch, ask yourself if you would accept such a patch, if you would like a commit message like that, if that variable name is really good, if someone could quickly grasp what s it all about, etc. Get into the flippy flops of the maintainer, don t assume everything is obvious to everyone.

Dirk Eddelbuettel: R Project and Google Summer of Code: Welcome to our students!

A few hours ago, I sent the following to both the R development list and the informal R / GSoC list:
Date: Mon, 26 Apr 2010 15:27:29 -0500
To: R Development List <r-devel>
CC: gsoc-r <gsoc-r>
Subject: R and the Google Summer of Code 2010 -- Please welcome our new students!
From: Dirk Eddelbuettel <edd>
Earlier today Google finalised student / mentor pairings and allocations for
the Google Summer of Code 2010 (GSoC 2010).  The R Project is happy to
announce that the following students have been accepted:
   Colin Rundel, "rgeos - an R wrapper for GEOS", mentored by Roger Bivand of
      the Norges Handelshoyskole, Norway
   Ian Fellows, "A GUI for Graphics using ggplot2 and Deducer", mentored by
      Hadley Wickham of Rice University, USA
   Chidambaram Annamalai, "rdx - Automatic Differentiation in R", mentored by
      John Nash of University of Ottawa, Canada
   Yasuhisa Yoshida, "NoSQL interface for R", mentored by Dirk Eddelbuettel,
      Chicago, USA
   Felix Schoenbrodt, "Social Relations Analyses in R", mentored by Stefan
      Schmukle, Universitaet Muenster, Germany
   Details about all proposals are on the R Wiki page for the GSoC 2010 at
   http://rwiki.sciviews.org/doku.php?id=developers:projects:gsoc2010
The R Project is honoured to have received its highest number of student
allocations yet, and looks forward to an exciting Summer of Code.  Please
join me in welcoming our new students.
At this time, I would also like to thank all the other students who have
applied for working with R in this Summer of Code. With a limited number of
available slots, not all proposals can be accepted -- but I hope that those
not lucky enough to have been granted a slot will continue to work with R and
towards making contributions within the R world. 
I would also like to express my thanks to all other mentors who provided for
a record number of proposals.  Without mentors and their project ideas we
would not have a Summer of Code -- so hopefully we will see you again next
year. 
Regards,  
Dirk (acting as R/GSoC 2010 admin)

26 April 2010

Jan Hauke Rahm: Talking about Free Software

When I first started using Free Software yes, that s quite some time ago I found it the right thing to do. It felt like I was first time ever really controlling what my computer does. Took just a bit of time to notice I really never knew anything about it. Well, almost at least. Then when I started contributing to Free Software this obviously changed. One of the freedoms is to learn from the code you re running. And I did it by reading, contributing, doing nonsense with it, reimplementing it you know the deal. Starting to work on Debian was something that just evolved out of it. Doing so gave me the opportunity to give back. I never achieved that actually, though. At least I don t consider my contributions even remotely sufficient to make up for the great deal of software that I use sometimes without noticing it. Still there s lots of stuff that I don t have a clue of and I rely on other developers to do their (volunteer) job. Otherwise my knowledge wouldn t help much. I could never continue kernel development if those kernel hackers all stepped back for instance. I guess, it s getting more and more difficult the closer the piece of code is to hardware. Another thought I had was about Free Software in general. It s more than just the code (although it might be the most important part), it s also a concept that needs to be communicated. Firefox on Windows shows how Free Software can have a big market share in an otherwise closed source environment. Yet I m not sure all users know about it being open source and what that means (hear them talk about Freeware instead). And who is dealing with this conceptual stuff? I decided to do something about all that. Now I don t really have money but a bit I thought I could share. And I could set yet another sign by publicly talking about it, or casually mentioning that I consider Free Software and Linux a good thing. So you will see a mail signature now below all my mails. I always wondered what one might gain with such and now I found a reason to include one. It reads:
Debian Developer                                 www.debian.org
Member of the Linux Foundation                    www.linux.com
Fellow of the Free Software Foundation Europe      www.fsfe.org

Obey Arthur Liu: Welcome to our 2010 Debian Google Summer of Code students!

I d like to extend a warm welcome to our selected students for the 2010 Debian Google Summer of Code! They should pop up on Debian Planet soon and you re welcome to come talk to them on #debian-soc on irc.debian.org Aptitude Qt by Piotr Galiszewski, mentored by Sune Vuorela Qt GUI for aptitude. Currently, KDE users need to use Aptitude via the console interface, or install the newly developed GTK frontend, which does not fit well into KDE desktop. Making Qt frontend to Aptitude would solve this problem and bring an advanced and fully Debian-compliant graphical package manager to KDE. Content-aware Config Files Upgrading by Krzysztof Tyszecki, mentored by Dominique Dumont When a package deliver configuration files, the problem of merging user data with new configuration instructions will arise during package upgrades on users systems. Sometimes merging can be done with 3 way merge, but this process does not insure that the resulting file is correct or even legal. This project intends to create standards, tools an heuristics to make the scary config file conflict resolution debconf prompt a thing of the past. Debbugs Bug Reporting and Manipulation API by David Wendt Jr., mentored by Bastian Venthur Currently debbugs supports a SOAP interface for querying Debian s Bug Tracking System. Unfortunately this operation is read-only. This project would create an API for debbugs which supports sending and manipulating bug reports, without having to resort to email. This project does not intend to replace email as mean to manipulate the BTS but rather to enhance the BTS to allow other means of bug creation and manipulation. Debian High Performance Computing on Clouds by Dominique Belhachemi, mentored by Steffen Moeller The project paves a way to combine the demands in high performance computing with the dynamics of compute clouds with Debian. Combining the Eucalyptus cloud computing infrastructure with the TORQUE resource manager and preparing the components for dynamically added and removed instances provides the user with a attractive high performance computing environment. Such a system allows users to share resources with large compute centers with minimal changes in their workflow and scripts. Debian-Installer on Neo FreeRunner and Handheld Devices by Thibaut Girka, mentored by Gaudenz Steinlin This project aims to improve the installation experience of Debian on handheld devices by replacing ad-hoc install scripts by a full-blown and adapted Debian-Installer. The Neo FreeRunner is used as it is the most convenient and open device from a development standpoint, but other devices will also be explored. Hurd port and de-Linux-ization of Debian-Installer by J r mie Koenig, mentored by Samuel Thibault The primary means of distributing the Hurd is through Debian GNU/Hurd. However, the installation CDs presently use an ancient, non-native installer. The goal of this project is to port the missing parts of Debian-Installer to Hurd. To achieve this, all problematic Linux-specific code in Debian-Installer will be replaced by less or non-kernel dependent code, paving the way for better support of other non-Linux ports of Debian. Multi-Arch support in APT by David Kalnischkies, mentored by Michael Vogt Hardware like 64bit processors are perfectly able to execute 32bit opcode but until now this potentiality is disregard as the infrastructure tools like dpkg and APT are not able to install and/or solve dependencies across multiple architectures. The project therefore focuses on enabling APT to work out good solutions in a MultiArch aware environments without the need of hacky and partly working biarch packages currently in use. Package Repository Analysis and Migration Automation by Ricardo O Donell, mentored by Neil Williams Emdebian uses a filter to select packages from the main Debian repositories that are considered useful to embedded devices, excluding the majority of packages. The results of processing the filter are automated but maintaining the filter list is manual. This project seeks to automate certain elements of the filtering process to cope with specific conditions. This project will also generalize to more elaborate and intelligent algorithms to improve the transitions of the main Debian archives. Smart Upload Server for FTP Master by Petr Jasek, mentored by Joerg Jaspert Making packages upload smarter, more interactive and painless for uploaders by switching from anonymous FTP and Cron jobs to a robust protocol and modern package checking and processing daemon. This daemon would test early and report early, saving developers time. More details coming soon on http://wiki.debian.org/gsoc Congratulations everyone and have a fruitful summer!

25 April 2010

Vincent Sanders: XCB programming is hard

I have been looking at writing a program with XCB today. I seem to pick projects that are an exercise in undocumented frustration.

All I wanted to do was have an x window open and be able to plot an arbitrary changeable image to it (no I am not writing yet another image viewer!). The XCB documentation seems to be a cross between "good luck with that" and some rather erratic doxygen output.

The utility libraries are rather skimpy on examples and its tiresome worrying that when you feed any of the xcb function names into Google (web or codesearch) there are very, very few hits beyond the freedesktop API docs.

My favourite has to be the xcb_image_create documentation, go on I challenge anyone to follow that logic without going and reading the sources! The answer (afaict) is that if you pass a pointer in base that pointer will be freed by xcb_image_destroy otherwise the data pointer will be used, unless the bytes value is too small in which case memory will be allocated with malloc and the passed pointer ignored.
Anyhow, I have succeeded and the result is below, mainly so I do not loose it :-)

/* XCB application drawing an updating bitmap in a window
*
* Inspired by the xcb black rectangle in a window example
*
* Copyright 2010 V. R. Sanders, released under the MIT licence
*/

/* compile with:
* gcc -Wall -lxcb-icccm -lxcb -lxcb-image -o disp disp.c
*/

#include <string.h>

#include <xcb/xcb.h>
#include <xcb/xcb_image.h>
#include <xcb/xcb_atom.h>
#include <xcb/xcb_icccm.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>

static xcb_format_t *
find_format (xcb_connection_t * c, uint8_t depth, uint8_t bpp)

const xcb_setup_t *setup = xcb_get_setup(c);
xcb_format_t *fmt = xcb_setup_pixmap_formats(setup);
xcb_format_t *fmtend = fmt + xcb_setup_pixmap_formats_length(setup);
for(; fmt != fmtend; ++fmt)
if((fmt->depth == depth) && (fmt->bits_per_pixel == bpp))
/* printf("fmt %p has pad %d depth %d, bpp %d\n",
fmt,fmt->scanline_pad, depth,bpp); */
return fmt;

return 0;


void
fillimage(unsigned char *p, int width, int height)

int i, j;
for(i=0; i < width; i++)

for(j=0; j < height; j++)

if((i < 256)&&(j < 256))

*p++=rand()%256; // blue
*p++=rand()%256; // green
*p++=rand()%256; // red
else
*p++=i%256; // blue
*p++=j%256; // green
if(i < 256)
*p++=i%256; // red
else if(j < 256)
*p++=j%256; // red
else
*p++=(256-j)%256; // red

p++; /* unused byte */




xcb_image_t *
CreateTrueColorImage(xcb_connection_t *c,
int width,
int height)

const xcb_setup_t *setup = xcb_get_setup(c);
unsigned char *image32=(unsigned char *)malloc(width*height*4);
xcb_format_t *fmt = find_format(c, 24, 32);
if (fmt == NULL)
return NULL;

fillimage(image32, width, height);

return xcb_image_create(width,
height,
XCB_IMAGE_FORMAT_Z_PIXMAP,
fmt->scanline_pad,
fmt->depth,
fmt->bits_per_pixel,
0,
setup->image_byte_order,
XCB_IMAGE_ORDER_LSB_FIRST,
image32,
width*height*4,
image32);


int
main (int argc, char **argv)

xcb_connection_t *c;
xcb_screen_t *s;
xcb_window_t w;
xcb_pixmap_t pmap;
xcb_gcontext_t gc;
xcb_generic_event_t *e;
uint32_t mask;
uint32_t values[2];
int done=0;
xcb_image_t *image;
uint8_t *image32;
xcb_expose_event_t *ee;
char *title="Hello World!";
xcb_size_hints_t *hints;

/* open connection with the server */
c = xcb_connect (NULL, NULL);

if (!c)
printf ("Cannot open display\n");
exit (1);


s = xcb_setup_roots_iterator (xcb_get_setup (c)).data;

/* printf("root depth %d\n",s->root_depth); */

/* create image */
image = CreateTrueColorImage(c, 640, 480);
if (image == NULL)
printf ("Cannot create iamge\n");
xcb_disconnect(c);
return 1;

image32 = image->data;

/* create window */
mask = XCB_CW_BACK_PIXEL XCB_CW_EVENT_MASK;
values[0] = s->white_pixel;
values[1] = XCB_EVENT_MASK_EXPOSURE
XCB_EVENT_MASK_KEY_PRESS
XCB_EVENT_MASK_BUTTON_PRESS;

w = xcb_generate_id (c);
xcb_create_window (c, XCB_COPY_FROM_PARENT, w, s->root,
10, 10, image->width, image->height, 1,
XCB_WINDOW_CLASS_INPUT_OUTPUT,
s->root_visual,
mask, values);

/* set title on window */
xcb_set_wm_name(c, w, STRING, strlen(title), title);

/* set size hits on window */
hints = xcb_alloc_size_hints();
xcb_size_hints_set_max_size(hints, image->width,image->height);
xcb_size_hints_set_min_size(hints, image->width,image->height);
xcb_set_wm_size_hints(c, w, WM_NORMAL_HINTS, hints);

/* create backing pixmap */
pmap = xcb_generate_id(c);
xcb_create_pixmap(c, 24, pmap, w, image->width, image->height);

/* create pixmap plot gc */
mask = XCB_GC_FOREGROUND XCB_GC_BACKGROUND;
values[0] = s->black_pixel;
values[1] = 0xffffff;

gc = xcb_generate_id (c);
xcb_create_gc (c, gc, pmap, mask, values);

/* put the image into the pixmap */
xcb_image_put(c, pmap, gc, image, 0, 0, 0);

/* show the window */
xcb_map_window (c, w);
xcb_flush (c);

/* event loop */
while (!done && (e = xcb_wait_for_event (c)))
switch (e->response_type)
case XCB_EXPOSE:
ee=(xcb_expose_event_t *)e;
/* printf ("expose %d,%d - %d,%d\n",
ee->x,ee->y,ee->width,ee->height); */
xcb_copy_area(c, pmap, w, gc,
ee->x,
ee->y,
ee->x,
ee->y,
ee->width,
ee->height);
xcb_flush (c);
image32+=16;
break;

case XCB_KEY_PRESS:
/* exit on keypress */
done = 1;
break;

case XCB_BUTTON_PRESS:
fillimage(image->data, image->width, image->height);
memset(image->data, 0, image32 - image->data);
xcb_image_put(c, pmap, gc, image, 0, 0, 0);
xcb_copy_area(c, pmap, w, gc, 0,0,0,0,image->width,image->height);
xcb_flush (c);
break;

free (e);


/* free pixmap */
xcb_free_pixmap(c, pmap);

/* close connection to server */
xcb_disconnect (c);

return 0;




24 April 2010

Neil Williams: pdebuild-cross

A new package - a set of hooks and wrappers to extend pbuilder and pdebuild for cross-building support - is heading to Debian experimental soon. Support depends on replacing debootstrap with multistrap when creating the cross building chroot and includes an svn helper to use alongside svn-buildpackage to cross-build directly from SVN.

There are limitations - apt-cross being the main one. Once these rough edges are smoothed over, pdebuild-cross could possibly migrate into pbuilder.

The reason to ditch debootstrap is so that the cross-building chroot can collect packages from multiple repositories (in this case, the emdebian toolchain repository). The package scripts support no options directly but will pass options down to the underlying tools. Other configuration happens in /etc/pdebuild-cross/pdebuild-cross.rc which is an extended pbuilder.rc file. I see this as a better option than bulking out the scripts with option support - especially at this stage of development.

Initially, pdebuild-cross will be a binary package built from the emdebian-crush source package (which itself replaces the emdebian-tools source package) and pdebuild-cross will depend on the version of multistrap also in Debian experimental.

Coming to Debian NEW soon....

SVN: http://www.emdebian.org/svn/browser/current/host/trunk/emdebian-tools/branches/2.2.0

Russell Coker: Is the NBN a Good Idea

Since writing my post about whether the National Broadband Network can ever break even [1] I ve had a number of people try to convince me of it s merit. He is my summary and rebuttal of some of the arguments for the NBN: The FUD Claims are made that Australia may fall behind the rest of the world, children may be disadvantaged in their education, or many other bad things may unless a faster net access is provided to everyone. That is FUD. If we are going to spend $43,000,000,000 then we should have some evidence that it will do some good. One thing to note is that rural areas will not get anything near the 100Mb/s speeds that FTTH will deliver to people in the cities. So if faster net access is actually essential then it would probably make sense to start by delivering ADSL2+ speeds to rural areas something that is not planned to be part of the NBN. Some people claim that having slow net access (ADSL2+) is going to preclude unknown future uses of the Internet. However the NBN is only going to be 100MB/s so any such claim is essentially that unknown things will happen in the future to make 24Mb/s too slow, but those unknown things won t make 100Mb/s too slow . I started using computer networks when a 1200/75 modem was considered fast. Over about the last 25 years typical net access speeds have increased from 1200b/s to about 12Mb/s, that s a factor of 10,000 speed improvement! Now the people who believe that we only need to multiply the current speed by a factor of 4 to address all future needs could be correct, but it seems unlikely and doesn t seem like a good idea for a $43,000,000,000 bet. If they were talking about 1Gb/s net access then things would be different in this regard. Technological Development Some people compare the NBN to the Apollo Program and suggest that the scientific research involved in implementing a FTTH network might result in useful developments in other areas. The Wikipedia page about Fiber to the premises by country indicates that Hong Kong had 1Gb/s available in 2006. It seems that a service which is rolled out 4 years later and 10 times slower than in Hong Kong is not going to involve a huge amount of research. Certainly nothing like the Apollo Program. It would allow multiple HDTV channels to be viewed at the same time According to the Australian Bureau of Statistics in 2001 the average Australian household had 2.6 people (3.0 in NT) [2]. The incidence of every member of a household wanting to watch a different live TV channel at the same time is probably quite low, it seems that young people nowadays spend as much time watching Youtube as they do watching TV. Based on the Wikipedia bit-rate page it seems that an ADSL2+ link could stream two HDTV channels. I could use fiber speeds today There are some people who claim that they need faster speeds for their data transfers right now. The problem is that the latency of data transfer is a bottleneck in transfer rates. In some quick tests between servers that have reasonably fast connections to the Internet I was only able to get one international transfer in excess of ADSL2+ speed (24Mb/s or 3MB/s). I was able to transfer data between London and Germany at a speed of 11MB/s which was possibly limited by the 100baseT Ethernet connection in the data center in Germany. Now if the people who pay for that German server were to pay more then they would get a higher speed. So anyone who downloads anything from my web site in it s current configuration would get a significant performance boost by using an NBN connection if they live in Europe! But if they live in Australia then they will probably get a fraction of that speed (a quick test indicates that my ADSL2+ connection gives me about 500KB/s from my web server. I need to have at least three transfers going at the same time to get more than 600KB/s when downloading files from other countries, and it s rare to have a single download run at a speed higher than 100KB/s. It would be handy if I could download at higher speeds from my ISP s mirror (which seems to be the only way someone in Australia can even use full ADSL2+ speeds). But it s certainly not worth the $5,000 per household installation cost of the NBN to get that. I m sure that there are some people who really do have usage patterns that could take advantage of fiber net access, one possibility would be downloading large files from a local source such as a major web site that uses a CDN. It seems likely to me that the majority of people who fit this category are using major porn services. Fiber would be good for Video-Conferencing To be better than ADSL for video-conferencing a NBN connection would need a faster upload speed. Internode is already selling NBN connections for the trial areas [3]. The cheapest fiber plans that they offer are at the 50/2 speed, that s 50Mb/s download and 2Mb/s upload in theory ADSL2+ should have a higher upload speed. In my tests the best result I ve got from sending files over my ADSL2+ link to another country is about 110KB/s (880Kb/s). The fact that the theoretical speed of a fiber connection is better than the measured speed of an ADSL2+ connection in this regard doesn t mean much, let s not assume that a fiber connection will get it s theoretical maximum speed. Not that you necessarily need higher speeds for video-conferencing, Youtube is one of many services that uses a lot less bandwidth than the upload speed of an ADSL2+ connection. Also video-calls which are supported on most 3G mobile phones use even less bandwidth again. I want an NBN connection to run a server at home The fastest connection for uploading that Internode offers to home users is the Home High plan at 100/8 speed, that is maybe a bit more than twice as fast for uploading as ADSL2+. They also offer a SOHO plan that supports 16Mb/s upload speed (at extra expense) and suggest that customers who want higher speeds contact their business sales department. But they include both sending and receiving data in the bandwidth quota for the fiber connections. Transmitting data at 16Mb/s isn t that great for a server. The cheapest virtual server plan on offer from Linode [4] includes 200GB of data transfer per month and has significantly higher transmission speeds. You could get a Linode virtual server plus an Internode ADSL2+ connection for about the same price as an Internode fiber connection to the home. There are two down-sides to virtual servers, one is that they are limited in the amount of RAM that they have (I can easily afford to buy 8G of RAM for a home system but renting an 8G virtual server is going to be expensive) and the other is that the storage is limited. Shared storage on virtual servers can be slow and is limited in size. If you need to run a server with a few terabytes of data storage (which is cheap on commodity SATA disks but expensive on server-grade disks) and you don t need to transfer much of it then a home server on the NBN might do well. Otherwise it s probably not going to work well for server use. The NBN will avoid people leaving their PC on to do downloads and save electricity To save electricity you would have to have a significant incidence of situations where a download can complete fast enough over the NBN to allow the user to turn their PC off before going to bed but be slow enough over ADSL to require that it be left on overnight. That would probably only apply to downloads from a CDN or from a local ISP mirror. From the Internode mirror I can download a test file at a speed of 850KB/s (I guess this means that my ADSL connection is not delivering full speed I suspect poor quality wiring in my home and would try to fix it if the current speed was too slow). In 5 minutes spend brushing my teeth I could download 250M of data, in the same time if I had a 100/4 connection on the NBN I might be able to download almost 3G of data. So in the unlikely event that I wanted to download a CD or DVD image and turn off my PC immediately before going to bed then the NBN would be a good thing. But then of course I would want to burn the CD or DVD image to disc and that would take long enough that I would leave it on overnight My ADSL connection gives really low speeds or I m out of range for ADSL Some people who live only a short distance from an exchange are unable to get ADSL2+. Some people live a long way from exchanges and are outside ADSL range. The ideal solution to these problems is not to provide fiber access to the majority of the population, it is to provide ADSL to everyone who is near an exchange and maybe provide fiber access to some people who are a long way from exchanges. I find it rather ironic that some people in the country are essentially saying because net access in the country is so slow we need fiber in the cities . The NBN is not going to give fiber to rural areas, satellite is one of the options that will be used. Really High Speeds I have been informed that to get the distances needed for the NBN they have to use Single Mode Fiber, this permits scaling up to higher speeds at some later time by changing the hardware at the end points. So we could end up with a Hong Kong speed network at some future time with the same fibers. This is a good thing. But I don t think that we need to get fiber installed right now so that we can use 100Mb, we could wait until there is enough need to get the faster transmission rates from the start. At the moment it s just a waste of money.

22 April 2010

Matt Brubeck: Headless Web Workers: Does the web need background apps?

At my last job, I created several web applications designed to replace built-in apps on mobile phones. While modern browsers and HTML5 made this incredibly easy in many ways, we still ended up writing native (i.e. non-web) code for most of our applications. There were a few different areas where the browser alone didn't meet our needs, but one that I found suprisingly common was background processing. Consider the following mobile applications: Ideally, each of these apps will perform some actions even when the user does not have it open. (Background processing is not strictly necessary for the e-reader, but it would be useful to ensure the library is up-to-date even when opened in a place with no network connection.) You can't do this with a web app. Web Workers don't solve the problem, because they run only while the web page is open. What we need are headless web workers.1 The API Headless workers could use almost the same API as Web Workers. Instead of responding to messages from a web page, they would listen to events from the host system (browser or OS). These events might include time intervals, power-on/resume, changes in network connection, geographic locations, or "push" notifications from a remote server. The event-driven architecture of JavaScript in the browser allows the host system a high level of discretion over resource consumption. There's no special code needed to suspend processes and later restore their state, because JavaScript workers are naturally inactive between events. The host can provide limits on CPU or memory usage per event, with a separate message to notify processes whose handlers were aborted. And it can limit the number of concurrent processes by choosing when to dispatch events to listeners. Some listeners could even be disabled completely at times (like if the device is busy or the battery is low), and notified later of the events they missed. This is almost a return to the old days of cooperative multitasking. Mobile computing is definitely driving everyone towards higher-level process control in the OS, and different assumptions for applications. It's not surprising that my whole proposal resembles Android and iPhone 4.0 multitasking in several ways, since I've been doing development on Android for the last 18 months and encountering many of the same issues. The UI Headless workers do need some way to interact with the user. They could display standard system notifications (via Growl on the Mac, libnotify on Ubuntu, the status bar in Android, etc.) using W3C Web Notifications, which already have an experimental implementation in Chrome. Users also needs to know which sites have background tasks installed. Headless workers could be represented by icons in a standard location (perhaps a toolbar in desktop browsers, or the home screen on a mobile device). The icons could display ambient status; clicking one would reveal a menu with options to configure or remove it. Questions This proposal might be hard to standardize, especially where it's tied to specific OS capabilities. For now I'm just curious: would it be useful? You can write a native app or a browser extension to solve this problem today. But would it be worthwhile to have a standard, cross-platform way to do it? Has anyone else run into problems that this approach could solve?
  1. Because all web standards should have names that sound like Harry Potter creatures.

21 April 2010

Russell Coker: Upgrading a SE Linux system to Debian/Testing (Squeeze)

Upgrade Requirements Debian/Squeeze (the next release of Debian) will be released some time later this year. Many people are already upgrading test servers, and development systems and workstations that are used to develop code that will be deployed next year. Also there are some significant new features in Squeeze that compel some people to upgrade production systems now (such as a newer version of KVM and Ext4 support). I ve started working on an upgrade plan for SE Linux. The first thing you want when upgrading between releases is a way of supporting booting a new kernel independently of the other parts of the upgrade. Either supporting the old user-space with the new kernel or the new kernel with the old user-space. It s not that uncommon for a new kernel to have a problem when under load so it s best to be able to back out of a kernel upgrade temporarily while trying to find the cause of the problem. For workstations and laptops it s not uncommon to have a kernel upgrade not immediately work with some old hardware, this can usually be worked around without much effort, but it s good to be able to keep systems running while waiting for a response to a support request. Running a Testing/Unstable kernel with Lenny Policy deb http://www.coker.com.au lenny selinux In Lenny the version of selinux-policy-default is 2:0.0.20080702-6. In the above APT repository I have version 2:0.0.20080702-18 which is needed if you want to run a 2.6.32 kernel. The main problem with the older policy is that the devtmpfs filesystem that is used by the kernel for /dev in the early stages of booting [1] is not known and therefore unlabeled so most access to /dev is denied and booting fails. So before upgrading to testing or unstable it s a really good idea to install the selinux-policy-default package from my Lenny repository and then run selinux-policy-upgrade to apply the new changes (by default upgrading the selinux-policy-default package doesn t change the policy that is running we consider the running policy to be configuration files that are not changed unless the user requests it). There are also some other kernel changes which require policy changes such as a change to the way that access controls are applied to programs that trigger module load requests. Upgrading to the Testing/Unstable Policy While some details of the policy are not yet finalised and there are some significant bugs remaining (in terms of usability not security) the policy in Unstable is usable. There is no need to rush an upgrade of the policy, so at this stage the policy in Unstable and Testing is more for testers than for serious production use. But when you upgrade one thing you need to keep in mind is that we don t to support upgrading the SE Linux policy between different major versions of Debian while in multi-user mode. The minimum requirement is that after the new policy package is installed you run the following commands and then reboot afterwards: setenforce 0
selinux-policy-upgrade
touch /.autorelabel If achieving your security goals requires running SE Linux in enforcing mode all the time then you need to do this in single-user mode. The changes to names of domains and labeling of files that are entry-points for domains is significant enough that it s not practical to try and prove that all intermediate states of partial labeling are safe and that there are suitable aliases for all domains. Given that you need to reboot to install a new kernel anyway the reboot for upgrading the SE Linux policy shouldn t be that much of an inconvenience. The relabel process on the first boot will take some time though. Running a Lenny kernel with Testing/Unstable Policy In the original design SE Linux didn t check open as a separate operation, only read/write etc. The reason for this is that the goal for SE Linux was to control information flows. The open() system call doesn t transfer any data so there was no need to restrict access to it as a separate operation (but if you couldn t read or write a file then an attempt to open it would fail). Recent versions of the SE Linux policy have added support for controlling file open, the reason for this is to allow a program in domain A to open a file and then let a program in domain B inherit the file handle and continue using the file even if it is not normally permitted to open the file this matches the Unix semantics where a privileged process can allow an unprivileged child to inherit file handles or use Unix domain sockets to pass file handles to another process with different privileges. SELinux: WARNING: inside open_file_mask_to_av with unknown mode:c1b6 Unfortunately when support was added for this a bug was discovered in the kernel, this post to the SE Linux mailing list has the conclusion to a discussion about it [2]. The symptom of this problem is messages such as the above appearing in your kernel message log. I am not planning to build a kernel package for Lenny with a fix for this bug. The command dmesg -n 1 will prevent such messages from going to the system console which is something you want to do if you plan to login at the console as they can occur often.

20 April 2010

Matt Zimmerman: Lucid ruminations

A few months ago, I wrote about changes in our development process for Ubuntu 10.04 LTS in order to meet our goals for this long-term release. So, how has it turned out? Well, the development teams are still very busy preparing for the upcoming release, so there hasn t been too much time for retrospection yet. Here are some of my initial thoughts, though. A more complete set of retrospectives on Lucid should give us some good ideas for how to improve further in Maverick and beyond. Update: Fixed broken link.

15 April 2010

MJ Ray: Debian Project Leader Election Campaign Round-up

I ve been AWOL for most of the debian project leader election campaigns this year, but I still want to vote, so I ve been dredging the emails on the last day. Maybe someone still has to vote and reads this, or maybe it ll help someone to interpret the result. What I ve done this year is to make a quick tally of platform points and campaign answers that I liked and disliked for each candidate, then total them up to get an order. If anyone gets a negative (more dislikes than likes), then I ll rank them below NOTA. Platforms: Margarita Manterola, Stefano Zacchiroli, Wouter Verhelst, Charles Plessy. Some of the questions: Make of that what you will. It surprised me which way the scores came out, but there was one answer with such a massive problem (in my opinion) that I ranked its giver equal with the next candidate down anyway. If you re a debian developer (DD), have you voted yet?

11 April 2010

Andrew Pollock: [life] I (apparently) have obstructive sleep apnea

A while ago now, Sarah commented that one night when she's woken up in the middle of the night, that she'd observed me stop breathing for a while in my sleep. She wanted me to go see the sleep doctors that she'd seen (she was diagnosed with obstructive sleep apnea a couple of years ago). So I eventually packed myself off to the Stanford Sleep Disorders Clinic, which is at Stanford's shiny new outpatient centre in Redwood City. They took a look at me, and declared I had a crowded mouth, and a narrow airway, and asked me to do a sleep study. I did the sleep study in the middle of January. That was "interesting". I had a ton of wires glued to me, and needless to say, I didn't sleep particularly well, but apparently I slept enough for them to be able to diagnose me. A couple of weeks later, I got a letter with the results. I had a Respiratory Disturbance Index of 16.6. The letter from the clinic defined RDI as including "events of 10 seconds or more with cessation of airflow or discernable reduction in airflow associated with arousal or oxygen desaturation of 3% or more". I had one obstructive sleep apnea event, and 119 obstructive hypopnea events during the sleep study. At one point my O2 saturation dropped to 89%, but it was very briefly. The overall average was 96%. A month ago I did another sleep study, this time while I was hooked up to a CPAP machine, and they fiddled with the pressure. I was fairly exhausted that night, and slept like a log. Based on the results of the second sleep study, I've been prescribed an auto PAP machine with a pressure ranging between 13 to 15 cm of water pressure (whatever that means). I got the machine, a ResMed S8 AutoSet II a week ago last Friday. It's slightly more advanced than the CPAP machine that Sarah uses, in that it'll adjust its pressure within the range it is set to, as it deems necessary throughout the night. It also backs off the pressure when you exhale, like Sarah's does. I'm not particularly thrilled to be sleeping with this thing on my face. The first night I used it, I slept fine. Something like from 10:30pm until 6am. Subsequently, I keep waking up at precisely 3:30am for some reason, and it's hard to get back to sleep with it on, so I take it off. A few nights, some sort of mask leakage alarm has gone off at some random point in the night. I haven't been in any state to try and diagnose what's going on when that happens, so I just take off the mask and turn the whole thing off. I was curious as to whether I'd notice some sort of life-changing difference between using it and not using it, like Sarah does, but so far, I can't say I've noticed any discernable difference. I also haven't gone a night without using it yet. Getting a properly fitting mask was a bit of a challenge. I spent ages at the durable medical equipment supplier trying on different sorts of masks. The one I've currently got still leaves a bit of a red mark across the bridge of my nose. I can take it back in the first 30 days, and get refitted, so I might still be doing that.

Martin-&#201;ric Racine: xf86-video-geode: EXA Composite support broken on X server 1.7

Just a quick note to everyone who recently upgraded their X server to 1.7 on Geode hardware to report that changes in X server since 1.6 have exposed and escalated bugs in our Geode driver that make EXA Compositing fail, resulting in broken image support on GTK2+ applications. As a temporary work-around, add this line to /etc/X11/xorg.conf in the Devices section:
Option "EXANoComposite" "true"
On my Lucid host, this gives me the following minimalistic xorg.conf:
Section "Device"
 Identifier "Geode Video"
 Driver "geode"
 Option "EXANoComposite" "true"
EndSection
We're currently working to produce new upstream packages that default to EXANoComposite as a temporary work-around, so that the driver at least produces useful results by default, and push this to distributions ASAP, especially keeping in mind pending Squeeze and Lucid releases. Meanwhile, AMD recently assigned new engineers to contribute to the driver's development and to eventually pick up maintenance, so we should soon have a permanent solution to these recent breakages in acceleration support for both GX2 and LX chipsets.

6 April 2010

Jonathan Yu: Google Summer of Code 2010

Last year, I had a great time participating in the Google Summer of Code with the Debian project. I had a neat project with some rather interesting implications for helping developers to package and maintain their work. It s still a work-in-progress, of course, as many projects in open source are, but I was able to accomplish quite a bit and am proud of my work. I learned quite a bit about coding in C, working with Debian and met some very intelligent people. My student peers were also very intelligent and great to learn from. I enjoyed meeting them virtually and discussing our various projects on the IRC channel as the summer progressed and the Summer of Code kicked into full swing. The Debian project in particular also helps arrange travel grants for students to attend the Debian Conference (this year, DebConf10 is being held in New York City!). DebConf provides a great venue to learn from other developers (both in official talks but also unofficial hacking sessions). As the social aspect is particularly important to Debian, DebConf helps people meet those with whom they are working with the most, thereby creating lifelong friendships and making open source fun. I have had several interviews for internships, and the bit of my work experience most asked about is my time doing the Google Summer of Code. I really enjoyed seeing a project go from the proposal stage, setting a reasonable timeline with my mentor, exploring the state of the art, and most importantly, developing the software. I think this is the sort of indispensible industry-type experience we often lack in our undergrad education. We might have an honours thesis or presentation, but much of the work in the Google Summer of Code actually gets used in the field. Developing software for people rather than for marks is significant in a number of ways, but most importantly it means there are real stakeholders that must be considered at all stages. Proposing brilliant new ideas is important, however, without highlighting the benefits they can have for various users, the reality is that it simply will not gain traction. Learning how to write proposals effectively is an important skill and working with my prospective mentor (at the time he later mentored my project once it was accepted) to develop mine was tremendously useful for my future endeavours. The way I see the Google Summer of Code is, in many ways, similar to an academic grant (and the stipend is about the same as well). It provides a modest salary (this year it s US$5000) but more importantly, personal contact with a mentor. Mentors are typically veterans in software development or the Debian project and act in the same role as supervisors for post-graduate work: they help monitor your progress and propose new ideas to keep you on track. The Debian Project is looking for more students and proposals. We have a list of ideas as well as application instructions available on our Wiki. As I will be going on internship starting May, I have offered to be a mentor this year. I look forward to seeing your submissions (some really interesting ones have already begun to filter in as the deadline approaches).
Filed under: Computer Science, Peer Relationships, Software Engineering Tagged: Computer Science, Debian, Google Summer of Code, Support

David Welton: Upcoming Conferences: "Better Software" and "Whymca"

One of the things I missed when we lived in Innsbruck was interacting with other "computer people". Back here in Italy, I have more friends and connections, and of course speak the language fluently. I sent in a few proposals for talks at several conferences this year, and they were both accepted. I'll be speaking at the Better Software conference in Florence in early May, about the economics of software and open source businesses. Later on in May, I'll be at Whymca, a conference dedicated to mobile development, to give a presentation about Hecl. That one's in Milan, which isn't as nice as Florence, but it should be a very interesting conference! I'm looking forward to both of them a lot, as there are a series of interesting talks that I'm looking forward to seeing, and a number of sharp programmers who I haven't seen in a while who I am looking forward to catching up with. I think I have some discount codes or something along those lines to hand out, for Better Software, so if you're interested, write me an email.

3 April 2010

Russ Allbery: Tie::ShadowHash 1.00

This is a Perl module that lets you stack multiple hash-like data sources together and treat them like a merged hash. Queries on the hash are satisfied by the first hash in the stack that has the desired key and iteration through the hash iterates through each in turn. You can store or delete elements from the hash and those changes are reflected in later uses, but the underlying data sources are not changed; all modifications are kept as local overrides. There's also some convenience support for treating text files as hashes. I wrote this module eons ago and have always been fairly proud of it, but so far as I knew, no one ever used it. It was going to be used in a new version of News::Gateway, but as long-time followers of my development work know, that module is kind of cursed. So I'd mostly forgotten about it. Imagine my surprise when it turned out that Wiki::Toolkit started using it, and then someone wanted to package it for Debian! I found out about that because of a discussion on debian-perl about how to name the package, since I always used to abbreviate my CPAN distribution names and it was in CPAN as ShadowHash. This release dusts the module off, converts the VCS from CVS to Git, updates the coding style and documentation, fixes the test suite to use Test::More, and fixes a few long-standing bugs that I noticed when reading through the code with fresh eyes. It also now has a proper 1.00 version number, since I don't expect to have to make backward-incompatible changes to it. You can get the latest version from the Tie::ShadowHash distribution page.

Joerg Jaspert: Fun with broken hardware

As you might have read, one of Debians slightly important machines had some hardware trouble. Before I get into more details of this story, let me say thank you to all the people who worked on this issue: - The DSA (Debian System Administrators) team, - the local admins at Brown University Computer Science department. I intentionally leave out the HP support here, I think their work in this issue was pretty suboptimal. While they do follow a certain schema, it took a bit long and the stunt with the technician not finding the replacement board and later on leaving without further notice wasn t all that great. (Update: Just to clarify this: In general we are happy with HP and the hardware and relationship and all that stuff. And usually the support works pretty well, like for all the times we had to replace broken disks.) Now, the story. This all started by us noticing something in ries kernel output telling about some trouble with the ram:
ries kernel: [168074.392016] Northbridge Error, node 0, core: -1
ries kernel: [168074.392186] K8 ECC error.
This was first noticed somewhere at the beginning of January. A DSA member then used the usual HP tools to check whats up, none of it reported an error. There wasn t much of an option left besides waiting what happens in the future. No support would do anything here. Until March we did not have any trouble. But at the 14th of March this changed, as the HP tools now found a serious error, reporting Status: DIMM is degraded for the third DIMM module. A support ticket was opened at this point and a replacement DIMM got sent out. Unfortunately the well known Mr. Murphy did look at this issue and so the wrong thing was sent out to the wrong place. Countdown continued and 10 days later the host decided to randomly lockup. And as HP servers have an Automatic Server Reset function in such cases, the whole machine rebooted. Not only once. Guess how much a filesystem likes such things when the machine is active. And for added fun a second DIMM got reported as broken, we now had DIMM #3 and #4 gone off. A certain ping-pong with HP support later we asked the local admins to take out the broken DIMMs and then also to change the order in which the remaining DIMMs are arranged. Unfortunately this did not help, the errors stayed. At this time we already turned off the services running on the machine and it mostly waited in the HP Insight Diagnostics test from HPs SmartStart CD. (And it always locked itself at 97% here). The support asked us to try another round of work on the machine, which our DSA team and the local admins did. But as it already was the 30th of March and nothing helped, our DSA got a little more direct in asking for a working solution. Like replacing the board and sending a technician to look and fix the machine. This was scheduled for the 1st of April, but turned out to not work as hoped. For some reason the technician was not able to find the board that was sent as replacement (no fault of our local admins!), so nothing happened the 1st of April except us getting a little upset. DSA communicated this to HP support, asking for escalation and finally getting this fixed. Our local admins could at last greet a HP technician on the 2nd of April then and he even had the replacement board with him, so he did his work there. He also confirmed that the two DIMMs mentioned above are faulty. For some reason he did not replace them, but left them outside of the machine. He also left the place without telling the Brown CS staff, so it wasn t the best performance he made. Fortunately the machine was in a working state again, only missing 2 of its DIMMs, 4Gb RAM. This has to be replaced by HP in the next days. When the Brown CS staff found out the technician left without notice they enabled our DSA to access the ilo of the machine again, so they could get the machine back up. Of course the first run was the HP SmartStart stuff again, running a full test. It reported no error, pheeew. There had been a few more steps in getting the settings of the ilo correct again and running a full filesystem check on it, but somewhere in the night the machine was back up and ready for us FTPMasters and Release team to get the services back. As the machine was back up in the middle of the night where I was sleeping, I did start my work somewhere this morning. At 10:25 CEST this started with a md5sum check of all files we have in our pool/ structure, ie. the Debian archive. We had 655k files to check, so it naturally took a while. Luckily we only found 32 broken files. Even better a set of them is already fixed, the rest of them is easily fixable with a set of binnmus the buildd people helpfully schedule, so not too much work involved in fixing this issue. Could have been worse. (Actually, when the first reboots happened, before the long downtime, we also had some broken logfiles, unreadable changes files and such things. But those I already fixed back then). More or less in parallel to the md5sum check we also started the queue daemons again, to let our upload queues from the other hosts forward their files to the central queue on ftp-master, tested the triggers to the new buildd peoples machine and the release team did a britney run. This took the first 5 hours (and a bit) of the process. As a second step, and we are still in this, we started the processing of the incoming queue. Turned out we have to process 10gigabytes of new uploads, in a total of 1285 uploads. This process is still running at the time I write this blog post, having processed about 1100 uploads. A good 9 til 10 hours run, but this was expected. I actually thought it will take even longer. When this is done I will go and have a manual dinstall run. That is, I will run every action a dinstall is usually doing by hand. And verify each step. Usually dinstall takes some 2 hours, manually it can take a bit more. Somewhere during this run a mirror push will also be done, a fairly huge one this time, with all those updates. When this all works I will turn on FTPMaster cronjobs after this dinstall, or at latest tomorrow morning, as usual, and service is back as usual. As far as I know the release team plans to turn on their cronjobs tomorrow morning too, so this will all be nice then. Apropos release: Please note what I wrote in my latest d-d-a mail:
As a sidenote from me: If you have a package that might be involved in
whatever transition to testing, or simply might get involved, it could
really help if you wait a day or two with the next upload, so the
release team can sort through all before you give them new trouble. If
in doubt, try mailing debian-release@lists.debian.org and ask. :)
So, thats it, enough annoyance with all those status bits here and there. I go leave you alone now. :)

2 April 2010

Thomas Koch: Design document for a patch management system on a DVCS

Dear friends of Debian,

this is my first post to Planet Debian. - The planet with the most geeky registration procedure in the known universe!

I proposed an alternative to topgit some days ago on the vcs-pkg.org list. Martin asked (and encouraged) me to give a better explanation of the idea, which I'll hereby try. Sorry for not giving any drawings, but I'm totally incapable of anything graphical.

Hopefully, I'll manage to come to the Debian Miniconf in Berlin. Then we could discuss the idea further and maybe even start implementing it. (Somebody would need to help me with my first steps in Perl then...)

The following text is available on github. Please help me expand it!

Design document for a patch management system on a DVCS
Requirements The system to implement manages patchsets. A patchset is a set of patches with a tree-ish dependency graph between the patches. There's one distinct root of this dependency graph. Patches are managed as branches with each branch representing a patch. Modification of a patch is done by a commit to the respective branch. A branch representing a patch as part of a patchset is called patchbranch. The patch of a patchbranch is created as the diff between the root of the patchbranch and the head. The most important management methods are:
  • Export a patchset in different formats
    • quilt
    • a merged commit of all patches
    • a line of commits with each commit representing one patch
  • Update a patchset against an updated root.
  • Copy a patchset
  • Delete a patchset from direct visibility while preserving all history about it
  • Hide and unhide a patchset from direct visibility
Additional requirements:
  • The system should be implementable on top of GIT, Mercurial and eventually Bazaar.
  • The system must easily cope with multiple different and independent patchsets.
  • All information about a patchset must be encoded in one distinct branch. Publishing this one branch must be sufficient to allow somebody else to recreate the patchset with all of its patchbranches.
  • The system should not rely on the presence of hooks.
  • The system should not require the addition of management files in patch branches (like .topmsg and .topdeps in topgit)
  • The system must be easy to understand for a regular user of the underlying DVCS.
  • The implementation may allow a patchset to depend on another patchset(s).

implementation
patchset meta branch A patchset meta branch holds all informations about one patchset. First, it holds references to the top commits of all patch branches in the form of parent references of commits. Thus pushing the patchset meta branch automatically also pushes all commits of all patch branches. Secondly, the patchset meta branch contains meta informations about the patchset. These meta informations are:
  • The names of all patch branches together with the most recent commit identifier of a particular patch branch. Let's save this information in a file called branches.
  • A message for each patch branch that explains the patch. These messages can be saved in the file tree as msg/$ PATCH-BRANCH-NAME
  • References to the dependencies of the patch (other patches of the same patchset or the root of the patchset). This is also encoded in the file branches.
Since the patchset meta branch holds all this informations, it is possible, to delete all patch branches and recreate them from this informations. Although the commits of the patchset meta branches hold references to the patch branches, its file tree does not need to contain any files from the referenced patches. This may confuse the underlying DVCS, but the patch meta branch is not ment to be directly inspected.

The branches file A branches file for a fictive patchset could look like:
# patch branches without an explicit dependency depend on the root of the
# patchset tree
# A Root can be given as either a fix commit (seen here), a branch or a tag.
# A fixed commit or tag is useful to maintain a patchset against an older
# upstream version
ROOT: 6a8589de32d490806ab86432a3181370e65953ca
# A tag as a dependency
#ROOT: upstream/0.1.2
# A branch as a dependency
#ROOT: upstream
# A regular patch with it's name and last commit
BRANCH: debian/use-debian-jars-in-build-xml 4bab542c261ff1a1ae87151c3536f19ef02d7937
# two other regular patches
BRANCH: upstream-jira/HDFS-1234 a8e4af13106582ca1bfbbcaeb0537f73faf46d87
BRANCH: upstream-jira/MAP-REDUCE-007 e3426bcbcb2537478f851edcf6eb04b34f6c7106
# This patch depends on the above two patches
# The sha1 below the dependency patches references a merge commit of the two
# dependencies
BRANCH: upstream-jira/HDFS-008 517851aa829d77e09bc5e59985fed1b0aa339cc6
DEPENDENCIES:
  upstream-jira/HDFS-1234
  upstream-jira/MAP-REDUCE-007
    cc294f2e4773c4ff71efb83648a0e16835fca841
# A patch branch that belongs to the patch branch, but won't get exported (yet)
BRANCH: upstream-jira/HDFS-9999 74257905azgsa4689bc5e59985fed1b0aa339cc6
BRANCH-FLAGS: noexport



Next.

Previous.