Yesterday I watched Les invasions barbares, a film by Canadian director Denys Arcand. I came to find it because one of my favourite cinemas in this oh-so-small city was premiering L’ ge des t n bres, the third part of a trilogy started by Le d clin de l’empire am ricain, and continued by Les invasions barbares above. I loved Les invasions barbares. I think it’s a very honest film, but for me not only it portrayed a reality and set of characters that I found credible, I also felt empowered by watching attitudes towards life so compatible, if not similar, to my own. Reminds me a bit of some of the feelings I had when watching Juno. It is also one film more to add to my (smallish) collection of films I’ve watched in French (with subtitles), though I’m intending to fix that: I really enjoy French, as well as enjoyed all Canadian or French I’ve watched in the past. We’ll see how it goes.

New developer accounts have been created moments ago, “finally”. Congratulations to all new DDs, with a special note for Aur lien G R ME (ag) and Cyril BRULEBOIS (KiBi). All of this made possible by Sam, our best DPL to date. Also, don’t listen to the fucktards going around telling “OMG DEBIAN IS DYING!!11!1!”. They’re just that, fucktards.

YAY! I'm now a Debian Developer. I started celebrating by uploading an NMU for the oldest patch of a RC bug I had sitting in the BTS. Thanks to Ari for advocating my membership, to Ari, Clint, Free, Piotr for sponsoring my uploads, and TomV. for managing my application. Also a special thanks to sam for making the last minute DAM appointments, which probably sped up my account creation, thus, finally 'making debian sexier' :).

A few months ago, I had shown a chart with per-5k-segment paces of my prior marathons runs (or at least those for which organisers had 5k splits). Since then, I ran two more: New York in November and London last Sunday. I have updated the charts to show both of these: A few things stand out:

• New York (in orange with small squares) clearly was my steadiest run, with very little variation and that nice burst at the end where the pace dropped below 7min/mile for the finish;
• London (in yellow with open circles) was allright, but with a noticeable drop around kilometers 30, 35 and 40 -- but then the second-best finish.
• I really cannot pace myself at the start: The two most recent races had the fastest start, including the hot and humid Chicago 2007 marathon (in light blue with circles)

In case anybody is interested (C'est bien toi, Christian, non?), the R script is available and I will be taking questions by email as R may not be obvious at first if you haven't used it.

It's time for me to stop complaining about the commercialisation of SourceForge and vote with my code. I'm leaving for brighter, bluer pastures at Alioth.

Query Object Framework is first (libqof1). The others will follow. I'm leaving because of unusable mailing list archives, impossibly slow web pages and wiki and the general amount of work involved in the SF interface. I'm gaining sane mailing lists, a friendly and usable ikiwiki and a simpler (although very similar) interface.

I'm also migrating most of those packages to subversion. With so many other changes, it's not exactly a problem to add cvs2svn into the mix.

Now, of course, this does not mean that these upstream packages suddenly become Debian-only or Debian-native. It does mean that I have a chance of encouraging new developers to join the team by providing the kinds of services and support that people need when starting off with a new project. (I suppose the one thing I don't yet have is an IRC channel. Not sure if it is worthwhile but I'll see what happens.)

QOF will form the basis of the support available for each of the other projects - all will share the same low-volume QOF-devel mailing list and QOF Wiki as provided by Alioth. I'll also migrate QuickList because I'm looking at restarting development on a stalled project allied to QOF and which could provide missing functionality in QuickList. I did look at Gaby in this mix but a botched Gtk1->Gtk2 transition that ended up with two incompatible forks running different parts of the original codebase put me off completely (and neither of the forks actually worked). (Some packages deserve to be dead upstream.)

I'm looking for new blood, fresh faces and a new desire to take these projects forwards. The driving force is data freedom - creating data-driven applications that are completely agnostic to where and how the data happens to be stored. It is user data, it should be as free as the source code that wrote it. Who cares what format it is in? Let's get it into a format that some other program can use and make use of the existing data instead of duplicating it all the time. (The observant will note that my old haunt of GnuCash is stubbornly ignorant of the merits of a decent import/export implementation that actually covers more than 50% of the data in the application. No wonder I took the chance to sponsor homebank.)

DWI Data With Interaction (aka DUI -- Data Under the Interface) is what can provide the missing interfaces. It'll take a while to get it back up to date and available in Debian but it can provide the link between data and interfaces. GNU/Linux users should be able to design an interface purely on the basis of the data that it needs to handle and not care one jot about how that data comes from one free software app and goes on to the next free software app. QOF is gaining a connection to libgda which means that data can go to and fro between a whole range of databases and file formats. Homebank and pilot-qof can gain new data inputs (and possibly outputs) by utilising the existing import/export support in new ways via DWI/QOF (as libqof2). That can give a complete interface for all PIM and financial data for a typical SOHO user and all based on SQL-type queries so that every stage of the process can be tweaked and optimised according to user requirements at runtime.

(Where does Emdebian come into this? Well, many people like me need to record and process PIM/financial data on the move so Emdebian means that those users can have the same data processing on their handhelds as on the desktop. Smaller apps to capture the data quickly and efficiently whilst on the move; larger apps to refine, optimise and filter the data on the desktop entirely under user control but with a simple, scriptable, interface based on runtime queries so that the user can do exactly the same conversions tomorrow with new data but none of the configuration. Easy!)

Is this groupware ? No - it is personal, ultimately flexible and entirely customised for particular users by the users themselves. It's about small niche markets, about independent business people with bespoke and highly specialised requirements. No one is going to spend time writing their reports in Scheme or Haskell, because they are the only people who need that particular report. Instead, that individual user needs to be empowered through a simple and intuitive interface to take a piece of data from one source, correlate it with a static piece of data from individual configuration sources, calculate a result according to a third piece of data from a completely separate source and combine them all into a new set of objects that can be used in other queries. It's about preparing invoices from Palm data, it's about handling customer accounts for a one-man company or handling a stocks and shares portfolio for a partnership. Ultimately customised, ultimately flexible and completely format-agnostic.

These are the kind of applications that hold people back from GNU/Linux migrations - bespoke applications that are developed for just a few small companies or just a few situations. Free software finds it hard to provide those apps because free software depends (currently) on someone else being motivated to write a bespoke program and that person rarely understands the niche. This is a chance to put the development into the hands of those with the motivation to do the work, by making the development process more WYSIWYG.

The question is: Can we come up with a better name for DWI, dear lazyweb?

"DWI is a fairly simple environment for quickly creating data-driven applications, that is, graphical applications that manipulate and show info from a database. This environment differs from others in that it is focused on native GTK/Gnome support through the Glade GUI designer, and thus allows you to build user interfaces as elegant as you can make them in Glade."
http://dwi.sourceforge.net/

The big appeal is that DWI is database-agnostic - it only cares about the data structure, not the file format. Ally that with QOF which provides data-centric SQL-based queries and conversions between objects (including combining elements from one object into one or more other objects without having to hardcode the connection between the objects at compile time) and we have an interesting and powerful combination of engines, frontends and backends that are all data-centric.

Please, if this sparks your interest, join the QOF-devel mailing list, have a play with the ikiwiki at http://qof.alioth.debian.org/ and have a go with the code. This whole area is begging for development but it is just so far from being usable by the target audience at this stage. There is a lot of programming to be done by current developers before the principles can be let loose on the masses.


This is about working between existing projects and between distributions - no longer dedicated to a single application, platform or file format, no more tunnel vision. Work on the edges of existing applications (and indeed projects), remove the sharp edges of applications that should work together but can't. The majority of the code in these libraries and applications is already stable and mature. Like so many parts of free software, it is time, IMHO, to put aside our egotistical devotion to the one-true-executable and come up with an actual solution.

Bah! I'm beginning to sound like a PR exec instead of a dedicated geek. What does it take to motivate people and get things done around here?

Hell, yeah! Are you?

Anthony Towns writes about using an improved version of jigdo to download CD/DVD images [1]. His improvement is basically to pipeline operation for better performance. Jigdo (the Jigsaw download) is a tool to download a set of files and then use them to create a CD or DVD image [2]. The idea is that most web sites that have CD or DVD images also have a collection of files which comprise the DVD image available. This removes the need to store the data twice (wasting disk space on mirrors and in some situations breaking web caching). I have never used jigdo, and for all Debian installations in recent times (the last few years at least) I download a small net-inst CD image and then have it download the other files from the net. I have Squid set up to cache large objects so this doesn’t waste too much of my precious network bandwidth (which is limited and expensive in Australia). Now I’m thinking about what the optimum method for doing installs might be. One thing that would be good would be to support multiple repositories, the packages have unique file names and checksums so it should be possible to check one repository and then check another if it’s not working. I don’t mean multiple “deb” lines in the APT configuration. What I would like to do is to have an NFS file server or web server with an archive of downloaded packages and have APT check there first before downloading a file. So APT could get a list of packages from the net and then get the actual files locally if they are available. The next thing that would be good is the ability to create a CD or DVD image dynamically and to store all temporary files. So I could download files from the repository and create a DVD image with just the packages that I need. Every time I create a DVD image my sub-set of the Debian archives would increase and the number of files actually downloaded in the creation process would be reduced. The effect would be to incrementally create a local mirror of the Debian repository. Then I would like to see a two-stage DVD install process. I would like to boot from a CD or DVD and start the install and then have it give a list of files needed (which could be stored on a USB device or floppy) to create further CDs or DVDs for installation. One situation where this could have been beneficial was when I was doing an emergency install of CentOS. I did the first part of the install (selecting packages etc) to determine which CDs were needed. It turned out that almost all CDs were needed even though some of the CDs had only a few files that I was installing. If the installer could have written a list of packages to a USB device then I could have downloaded just those packages and got the install working a lot sooner. It seems to me that it’s fairly common to do one test install and then do some dozens of other installs with the same settings. So the ability to create a DVD of exactly the needed files for the other dozens of installs would be a great benefit. Now this is just random commentary, unfortunately don’t have time to do any coding on this. But it seems obvious that something has to be done to improve the situation for developers and IT staff who need some degree of mirroring of the Debian package pool but who can’t do a full mirror. Back in 1996 I was able to mirror Debian over a 28K8 modem link and fit it on what was a reasonable hard drive by the standards of the day (incredibly tiny by today’s standards). Now I can’t practically mirror Debian over an Australian cable broadband connection and even by the standards of about 4 years ago (the age of most of my hard disks) the space requirements are significant. I hope this post helps inspire some interest in developing these features. As delays in joining Debian [3] is the topic of the day it should be noted that work on preparing DVD images can easily be done by people who are not DDs. Such software should work from Debian archives without requiring any changes to them, and thus nothing special is needed from the Debian project to start work.

• [1] http://azure.humbug.org.au/~aj/blog/2008/04/15#2008-04-15-jigdodl


• [2] http://atterer.net/jigdo/


• [3] http://etbe.coker.com.au/2008/04/17/motivation-and-perspective/

Share This

A bug was reported on linux-parisc@ last week, that “ls -l” was whinging about “Operation not supported” once per file. Easy, I thought, I’ll just break out strace, figure out which syscall it is, and Bob’s your uncle, we’ll have it fixed. Not so, unfortunately, as strace was bloody broken. upeek: ptrace(PTRACE_PEEKUSER,2606,4294967292,0): Input/output error Again, no big deal, some kernel change must have broken ptrace(2) on me. git logs didn’t shed any light on it, so I pulled an old strace binary out of an etch chroot and tried that, which failed as well. Wonderful. But… curious… it still works inside the chroot. At this point I was a little bit beyond annoyed, but the fact that it worked in the chroot was a big clue. Anyway, long story short, strace decides to assume that you can have up to 32 possible syscall arguments (wtf?#1) which is all well and good and generally harmless. However, it also decides to yank MAX_ARGS of them out with ptrace(PEEK_USER, …) if strace is unaware of the syscall prototype. Again, this would be mostly harmless, on every other architecture. On parisc though, argument registers count down from %r26 through %r20, which is the syscall number. Instrumenting the PEEK_USER path through sys_ptrace in the kernel made it pretty obvious when a consecutive request was suddenly 0xfffffffc (which is 0-4, especially suspicious given the previous values of 4, 8, 16…) that I needed to find a loop in ptrace. Searching through all the HPPA specific syscall code found it quickly,
for (i = 0; i < tcp->u_nargs; i++)
    if (upeek(pid, PT_GR26-4*i, &tcp->u_arg[i]) < 0)
        return -1;

where “i” is 0 .. u_nargs, which is set to either the known number of syscall args, or MAX_ARGS (which, remember, was 32.) There, the culprit found, and with a simple one line fix, strace works again. In case you’re curious, the reason things worked inside the chroot, is because the syscall being strace’d was dependant on libselinux being loaded.
syscall_298(0x400e0fbc, 0x58, 0xfb444310, 0, 0x409878ac, 0x409878ac) = -1 (errno 2)
Which is statfs64. The moral of the story here? strace is out of date and doesn’t yet support recently added syscalls. Also, the code quality is pretty atrocious. Oh, and the problems with ls? getxattr was returning -EOPNOTSUP because the filesystem doesn’t support
xattrs. Or, more correctly, because that config option was disabled. This seems to be causing laughs on the buildds too. Doesn’t anyone check errno anymore? Now back to our regularly scheduled hacking. –
wtf?#1: Generally speaking, more than 5 uint32_t syscall args is frowned upon, since that makes the poor register-starved i386 pass args on the stack.

It's stories like the following that make you realise just how puny your talents are... (Reposted from a place not to be named, by the legendary Al Viro)

Give them what I got yesterday. As in, box that got
        a) Linux kernel running.
        b) init and bash running.
        c) serial and floppy - built as modules. And not loaded.
        d) no sash.
        e) no ethernet.
        f) bloody large number-crunching that Should Not Be Aborted(tm).
        g) libc and ld-linux.so - unlinked (self-LART by owner).
Now, I could tell the guy to piss off, but... WTF? He had decent beer and
was properly scared. Oh, well... So we have no exec() for anything
dynamically
linked. And we have no chance to access any external stuff - insmod is
linked
dynamically, so no insmod floppy for you. Shutting the system down was not
an
option due to (f) (aside of dealing with fsck later - umount(8) is
dynamically
linked too). Now, I knew that both /lib/libc.so-2.1.2 and
/lib/ld-linux.so-2.1.2 were still alive - mmaped by init, for one. And
/proc/1/maps would even contain their inumbers. So the plan of attack was
to create a file in root and then cannibalize the entry (change inumber in
the directory). Alas - not enough. In-core inode got zero i_nlink and
if I would just create a link by hands it would not become positive. I.e.
still remove-upon-close. But. But if we will manage to call link() on the
hand-made link we will get i_nlink raised to 1 - iget() will find the same
in-core inode, so we are OK. We'll have to revert the phony link to avoid
PO'd fsck, but that's not a problem...
        So there we go: assuming that we got static ln
echo >foo
ln foo bar
flip inumber in foo entry to point to libc
ln foo /lib/libc.so-2.1.2
flip inumber.......................... bar
repeat for ld-linux.so
rm foo bar
begin recovering other damage (self-LART was a bit larger).
        But... we don't have this flip stuff and we don't have (aaarrgh)
static
ln. Oh, shit... OK, but all we really need is a couple of syscalls, right?
There we go:
        eax=__NR_link;
        ebx=s1;
        ecx=s2;
        int 0x80;
        eax=__NR_exit;
        ebx=0;
        int 0x80;
s1:     "foo"
s2:     "bar"
The next step was getting the syscall numbers. grep? We don't need no
stinkin'
grep.
# while read i; do case $i in *__NR_link*) echo $i;; esac;
done </usr/include/asm/unistd.h
#define __NR_link 9
# while read i; do case $i in *__NR_exit*) echo $i;; esac;
done </usr/include/asm/unistd.h
#define __NR_exit 1
#define __NR__exit __NR_exit
Now, scratching the head and recalling intel code...
start:  b8 09 00 00 00
        bb (address of s1)
        b9 (address of s2)
        cd 80
        b8 01 00 00 00
        bb 00 00 00 00
        cd 80
s1:     66 6f 6f 00
s2:     62 61 72 00
Fine, but... a.out support is compiled... you guessed it, as module and not
currently loaded. So we are in for crufting up an ELF binary. OK, we don't
actually need 100%-correct ELF, just something that will pass for the
exec(). Now, we don't have shell scripts, but . will work. So the next
step was rolling more(1) in shell and reading through the relevant code
(surprisingly small - binfmt_elf.c and two headers). After much swearing
the following abortion was created:
7f 45 4c 46 00 00 00 00 00 00 00 00 00 00 00 00
02 00 03 00 01 00 00 00 start______ 34 00 00 00
00 00 00 00 00 00 00 00 34 00 20 00 01 00 28 00
00 00 00 00 01 00 00 00 00 00 00 00 base_______
base_______ size_______ size_______ 05 00 00 00
00 10 00 00
start:
b8 09 00 00 00 bb start+1d___ b9 start+21___ cd
80 b8 01 00 00 00 bb 00 00 00 00 cd 80 66 6f 6f
00 62 61 72 00
OK, set base to something page-aligned, start=base+54, size=79. There we go:
7f 45 4c 46 00 00 00 00 00 00 00 00 00 00 00 00
02 00 03 00 01 00 00 00 54 00 00 80 34 00 00 00
00 00 00 00 00 00 00 00 34 00 20 00 01 00 28 00
00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 80
00 00 00 80 79 00 00 00 79 00 00 00 05 00 00 00
00 10 00 00 b8 09 00 00 00 bb 71 00 00 80 b9 75
00 00 80 cd 80 b8 01 00 00 00 bb 00 00 00 00 cd
80 66 6f 6f 00 62 61 72 00
well, while read l; do for i in  echo $l ; do echo -ne "\\$i"; done; done
made for oct2bin, overwriting /usr/bin/emacs with the output of that gave
us static equivalent of ln foo bar. And it worked. The rest was essentially
the same - the only tricky part was to find the location of directory entry.
Which was done with (lseek, read byte, exit(said_byte)) and shell wrapper
around that. So we had a way to read a block and dump it on the console.
The rest was obvious - start from relevant block in inode table and walk
through the references... Once we got that, it was an easy ride - trick
with inumber flipping brought libc and dynamic linker back and after that
we had 99% of system back into the working state. Amazing how little you
actually need to bring the system back to life...

Hot damn that's some deep hackery...

Updates of the moment:

* I can put a t-shirt on now! About 3 days ago, I got brave and decided to give it a try. So, yay, no more zip up sweaters!

* Had my 6-week checkup yesterday, and everything is looking good. The incision is still a bit red and since my skin is so sensitive, the surgeon doesn't want me in the hottub for another 2 weeks. *pout*

* I started physiotherapy today. I can only give it a mixed review. The therapist was absolutely wonderful, it turns out that I got the clinic director. However, she doesn't actually have time to see me 3 days a week. Her schedule is so booked up that I'd be lucky to see her once a week. On top of that, the receptionists and assistants were not terribly professional (tending not to articulate, getting flustered, sort of tossing sheets of paper at me expecting me to know what to do with them), and the place was not clean. It was also in one big room, so the therapy is done in front of everyone else. Blech. So I have an appointment with a different clinic on Friday. Hopefully this one will go better.

* I don't think I've mentioned Leif recently. He's walking, can open the doors in the house now, and can clearly say "All Done". He also climbed onto my lap the other day and cuddled with me. So cute!

* Lastly, this is the end of the contest of "Guess the cost!" And the final total for the bill was.... $199,849.24. Our running total is of all things combined at UCSF is now somewhere around $280k, so I'm guessing the final total for everything will be around $300k when we're all said and done. Crazy. The closest guess was by runasand at $192,000, whoever you are. Linked-in says the winner is in Norway. So finding that vegan restaurant might be interesting. I still remember Debconf 3 in Norway when the caterer told me "In Norway, we have a special way of preparing vegetables... We feed them to the animals." And the meat-eaters begging the vegetarians who were going elsewhere for food for their meals so that they had something other than slabs of meat to eat.

I think because of this, I'm going to have to award the prize to mmmmtino in the meantime, who came in second place at $185k. That way I can pay it out hopefully within the next year. I'll still keep the other as a raincheque in case I happen to wind up in Norway sometime again. =)

Further on the OMG-This-medical-system-is-barbaric, It looks like the insurance company negotiates the hospital down in price, because they're in network. I've only seen the payout information for that one day overnight stay, but the insurance company will only be paying 10% of the cost. So if I didn't have insurance, I'd have to pay $67k. But the insurance company gets to pay around $7k. Do they charge so much because they don't make enough money when someone is insured? How does that all work?

16:46 < epmf> tp-glib is made of magic

—#telepathy, 2008-04-15

Rob pointed out today that I'd been promising to blog about telepathy-glib for several months and still haven't done so. I think there's too much to cover in one blog post, so I'll start with the oldest part - implementing a service (typically a Telepathy connection manager).

While telepathy-glib is (obviously) intended for Telepathy implementors, I think the ideas we've been implementing are likely to be useful for any D-Bus API, particularly flexible/complex APIs where an object implements many interfaces.

So I am nearly done with teaching this year, which is a relief, not because I don’t like it but because after a full year of teaching, one naturally wants a break. But before it is even over, you have to start thinking about your fall courses, mostly so you can order books in time and because developing a syllabus also requires more than a few days or weeks of work. Next fall, I am re-teaching a first year course Introduction to Human Communication and Culture as well as a new course on computer hacking. While I have an old version of the syllabus, I am going to spend the next few weeks revising, reshuffling, and updating. If there is anything you think should be included in this type of course drop me a line. Below is the new description of the course.

The Culture and Politics of Computer Hacking. This course takes as its object computer hackers to interrogate not only the ethics and practices of hacking, but to examine more broadly how hackers and hacking have transformed the politics of computing and the Internet more generally. We will examine how hacker values are realized and constituted by different legal, technical, and ethical activities of computer hacking for example, free software production, cyberactivism and hactivism, cryptography, and the pranksih games of hacker underground. We will pay close attention to how ethical principles are variably represented and thought of by hackers, journalists, and academics and will use the example of hacking to address various topics on law, order, and politics on the Internet such as: free speech and censorship, computer gaming, privacy and security, and intellectual property. This will allow us to critically 1) problematize thinking on computer hackers as a socio-cultural group guided by a singular ethic and set of practices 2) examine the multiple ways hackers draw on and reconfigure dominant ideas of property, freedom, and privacy through their diverse moral codes and technical activities 3) broaden our understanding of politics of the Internet by evaluating the various political effects and ramifications of hacking.

Another zero-day announcement of a Software in the Public Interest online meeting has been posted. The announcement mentions "one motion has been raised" but the agenda doesn't include any motions, so I'm not sure what. Also, the agenda lists "Debian logo licence" as up for discussion but I thought last month's meeting resolved that. I watch SPI fairly closely and I'm confused. How about the rest of you? SPI isn't alone in this: many software organisations seem to suffer from similar problems. For example, I ranted in passing about Free Software Manchester yesterday, who just posted the notes from their own zero-day meeting and I've suggested possible ways of reforming debian's currently-stalled New Maintainer process more than once. Are free software users particularly bad at the basics of running an interest society (like welcoming and expiring members, calling meetings, publishing routine communications, and so on), have I been spoiled by cooperatives with their friendly Member Services departments or secretariats, or what? Is this why so many free software orgs seem to include self-perpetuating leadership groups? Is this a serious problem if, as reported, Software Development is a Team Sport [etbe]? Are there fully-working free software mass participation groups out there? I feel a lot of these problems are caused by attempting to order our inherently entropy-filled world completely and insisting everything follows petty rules, such as refusing to answer a question because the "wrong" member asked it. The world will not become less random just because hackers try to impose arbitrary rules. Sometimes it's good to put down minimum standards (because calling zero-day meetings is a mostly-avoidable way of excluding some members) but it will always be a poor alternative to trying to do the best you can for others. How do we get past this? My pro-cooperation-and-better-business platform for SPI board went pretty badly and I've had some anti-cooperative flames back from someone starting another free software group this month, so I don't think I can fix these existing organisations any time soon. About 1 in 6 people in the UK are members of a cooperative, so even if that is reflected among hackers (and I think it's lower), all of them would not be enough to reform much. One of the most common memes in free software is "show us the code" and the few other free software cooperatives I've seen have mostly failed, with a few surviving but hitting a size limit. As a result, I'm currently negotiating the start of a new free software cooperative. So far, I'm really happy with how that's going. Many cooperators learn at the feet of large consumer cooperatives like the Cooperative Group who run training courses for new members about putting cooperative values and principles into practice, which we smaller groups couldn't run ourselves. As a result, most cooperators already know how to work well together. Should large software societies like SPI try commissioning similar courses?

Uwe Hermann has described how to resize a root filesystem after booting from a live-cd or recovery disk [1]. He makes some good points about resizing an LVM PV (which I hadn’t even realised was possible). The following paragraph is outdated, see the update at the end:
Incidentally it should be noted that if your root filesystem is an LVM logical volume then it can’t be resized without booting from a different device because the way LVM appears to work is that the LV in question is locked, then files under /etc/lvm/ are written, and then the LV is unlocked. If the LV in question contains /etc/lvm then you deadlock your root filesystem and need to press reset. Of course if your root filesystem is on an LV which has been encrypted via cryptsetup (the LV is encrypted not the PV) then a live resize of the root filesystem can work as locking the LV merely means that write-backs from the encryption layer don’t get committed. I’m not sure if this means that data written to an encrypted device is less stable (testing this is on my todo list). If your root filesystem is on a partition of a hard drive (such as /dev/hda2) then it is possible to extend it without booting from different media. There is nothing stopping you from running fdisk and deleting the partition of your root filesystem and recreating it. When you exit fdisk it will call an ioctl() to re-read the partition table, the kernel code counts the number of open file handles related to the device and if the number is greater than 1 (fdisk has one open handle) then it refuses to re-read the table. So you can use fdisk to change the root partition and then reboot to have the change be noticed. After that ext2online can be used to take advantage of the extra space (if the filesystem has a recent enough version of the ext3 disk format). One thing he didn’t mention is that if you do need to boot from another device to manipulate your root filesystem (which should be quite rare if you are bold and know the tricks) then you can always use your swap space. To do this you simply run swapoff and then run mkfs on the device that had been used for swap (incidentally there is nothing really special about the swap space, but it does tend to often be used for such recovery operations simply because it has no data that persists across a reboot). The minimum set of files that need to be copied to a temporary filesystem is usually /bin, /sbin, /dev, /lib (excluding all directories under /lib/modules apart from the one related to the kernel you are using), and /etc. Also you should make directories /proc, /sys, and /selinux (if the machine in question runs SE Linux). The aim is not to copy enough files for the machine to run in a regular manner, merely enough to allow manipulating all filesystems and logical volumes. Often for such recovery I boot with init=/bin/bash as a kernel parameter to skip the regular init and just start with a shell. Note that when you use init=/bin/bash you end up with a shell that has no job control and ^C is not enabled, if you want to run a command that might not terminate of it’s own accord then the command “openvt /bin/bash” can be used to start another session with reasonable terminal settings. I recommend that anyone who wants to work as a sys-admin experiment with such procedures on a test machine. There are lots of interesting things that you can learn and interesting ways that you can break your system when performing such operations. Update: Wouter points out that the LVM bug of deadlocking the root filesystem has been fixed and that you can also use ext2online to resize the mounted filesystem [2].

• [1] http://www.hermann-uwe.de/blog/resizing-a-dm-crypt-lvm-ext3-partition


• [2] http://www.grep.be/blog/en/computer/cluebat/actually_you_can

Share This

I'm writing this blog entry as a reaction of Lucas Nussbaums one. More or less one year ago I also was a New Maintainer who completed the process after 3 months (Started in February and ended in May). My Intention was to become as fast as possible a Developer and I was very motivated. (I helped Lucas with filling the build twice in a row bugs in this time). The FD approves me quite fast... (I'll guess I was only waiting for two weeks?) but the next to steps (Approval of DAM and creation of the account) took very long. After waiting for 3 months I was quite demotivated and doing only necessary things in order to update my packages... it was very frustrating. But now I'm finally a DD, since 6 months. For those who doesn't want to calculate: I was waiting for DAM about 4 months. When I saw that I was again highly motivated and started again to work on Debian a bit more, because I thought that the account creation can't took long...this was an error: I was approved by DAM in the middle of October... and got my account in Dezember... again waiting for a simple action. I can really understand New Maintainer who are directly MIA after being a DD. Waiting a half year for account creation is much too long. In my eyes there is only one possible solution for this issue: Since DAM and the Account Manager have also many other tasks which consumes there time, I would suggest to give them assistants...every manager needs at least one assistant ;-). This would speed up the process and people who are doing actually a great job doesn't get frustrated by waiting, but can actively work on Debian. I hope that the new DPL will find a solution for this really big issue during his time.

Every so often, I come across some utility that need. I think it must have been written before, but I can't find it.

Today I needed a tool to take a set of files and split them up into directories in a size that will fit on DVDs. I wanted a tool that could either produce the minimum number of DVDs, or keep the files in order. I couldn't find one. So I wrote datapacker.

datapacker is a tool to group files by size. It is perhaps most often used to fit a set of files onto the minimum number of CDs or DVDs.

datapacker is designed to group files such that they fill fixed-size containers (called "bins") using the minimum number of containers. This is useful, for instance, if you want to archive a number of files to CD or DVD, and want to organize them such that you use the minimum possible number of CDs or DVDs.

In many cases, datapacker executes almost instantaneously. Of particular note, the hardlink action can be used to effectively copy data into bins without having to actually copy the data at all.

datapacker is a tool in the traditional Unix style; it can be used in pipes and call other tools.

I have, of course, uploaded it to sid. But while it sits in NEW, you can download the source tarball (with debian/ directory) from the project homepage at http://software.complete.org/datapacker. I've also got an HTML version of the manpage online, so you can see all the cool features of datapacker. It works nicely with find, xargs, mkisofs, and any other Unixy pipe-friendly program.

Those of you that know me will not be surprised that I wrote datapacker in Haskell. For this project, I added a bin-packing module and support for parsing inputs like 1.5g to MissingH. So everyone else that needs to do that sort of thing can now use library functions for it.

Update... I should have mentioned the really cool thing about this. After datapacker compiled and ran, I had only one mistake that was not caught by the Haskell compiler: I said < where I should have said <= one place. This is one of the very nice things about Haskell: the language lends itself to compilers that can catch so much. It's not that I'm a perfect programmer, just that my compiler is pretty crafty.

Sunday was the 100th anniversary of the modern marathon: It was during the London 2008 Olympics that the marathon was run for the first time for the 26 miles 385 yard, or 42.185 km, distance. And it so happens that I was there to run the 2008 edition of the London Marathon. Pretty nice weather at the start and finish: sunny, not too hot, occassional clouds. But being London, we still managed to get drenched for about 30 minutes. Overall, the conditions were good -- or else Martin Lel would have had a hard time for a new course record. The course is pretty nice, right from the start in Greenwhich all the way to the spectacular finish in Westminster. Crowd support was good too, if only a little uneven. But the second half of the race, and particularly the last miles in the back from Canary Whard through the City to Westminster past Parliament and Buckingham Palace were awesome. Lots of people, lots of noise. Oh, and I even saw the leaders in a group of five, including Lel and Hall as the course had parallel 'out' and 'back' tracks around miles 13 for me and 22 for them. I had planned to take it easy and not try to run too hard, and aimed for a time of around 3:25 and then finished at 3:24:41. A little less 'even' than I had hoped, but still a very satisfying result. And for once the legs aren't all that shot afterwards :) And the whole weekend was nice as I got to stay with friends in the southwest of London. Staying up late on both ends of the trip suppressed the jet lag fairly well. At least that's what I keep telling myself.

We are at T-48h of fisl9.0, and everything starts to take form. I spent the whole afternoon getting our icecast infrastructure online. We ve got 6 servers, all with different bandwidth and not all with the same architecture The debian boxes were prevalent (4 of them 1 amd64 and 3 i386), all of them etch. We ll be able to follow the statistics at tvstats website (not all the servers are enlisted yet). In this process I found an evil bug in debian s icecast package, I will have to debug to understand it better: when relaying from a master server (which is our set-up), if the source stops feeding the master, the relay segfaults. The strace shows it just segfaults while pooling I got no explanation for it and I ll have to dissect it if I want to build a proper bug report. In the meantime, I installed the sarge-backports version, and it worked so, the bug report will have to wait until after fisl :-) I took some pictures to show how s everything being put together. Take a look: And some of the infrastructure crew: See you all there!

This morning, I just wasted two hours of my life trying to deal with a bill with AT&T. I am a work-at-home employee, and my company has a contract with AT&T so that when I deal 1-700-xxx-xxxx, I can reach the internal corporate phone network. In addition, long distance calls on my home office line are billed to the company at the pre-negotiated corporate rates. I also had a (long-dormant) AT&T long distance account, dating from before I started working at this company. Starting at the beginning of the year, that account grew a $18 monthly fee. When I tried to make it go away, the AT&T consumer side of the house said that according to Verizon, that was because I had my long distance service through AT&T. Which was true, in a sense — AT&T was providing my service, but through a corporate account. But the consumer side of AT&T didn’t understand that, and were in my opinion, willfully ignorant. Several phone calls later, I got the 1-800 number for the AT&T corporate side of the house, and those folks said they couldn’t look at consumer/personal AT&T accounts, and implied it was my fault that I had both accounts on the line. (This took over an hour while the support person tried multiple things, all of which was not helpful at all.) I finally googled for AT&T CEO’s office, and found a number on the consumerist.com web site that claimed to be the AT&T CEO’s office. It had long since been directed to a help center, but after I told my tale, I was quickly transferred to an executive customer support person, who was able to fix the problem in ten minutes. The only questions remaining are: 1) Why can’t the different parts of AT&T talk to one another?
2) Will this problem really be solved, or will I see another bill in a month or two and have to spend more time dealing with this mess all over again?
3) When will VOIP services put AT&T long distance out of its misery? (Given the absolute frustration of this morning, this can’t happen soon enough.)
4) Will AT&T compensate me for two hours of frustration and of my life that I will never get back. (Not bloody likely.) One thing is for certain, it will be a long, long, LONG time before I will ever voluntarily choose AT&T to provide service for just about anything. Even an iPhone wouldn’t be enough inducement….