I believe I'm not the only one who thinks that use case oriented Grub2 documentation is hard to find, and a lot of the documentation is obsolete or wrong. My main cause for writing this blog post is a currently unanswered question regarding 2.00, but meanwhile it seems months have passed and still most 1.99 documentation is wrong as well, which might be interesting to some.

The aim is to prevent grub entries from being edited, while not restricting actual booting. This protection is meant for computers not having any confidential stuff, but just wanting to do some light weight security with the assumption that the computer isn't physically opened.

Common setup

You will obviously want to disable any automatically generated root access giving entries, by for example uncommenting GRUB_DISABLE_RECOVERY="true" in /etc/default/grub on Debian or Ubuntu. Also you would disable allowing any external boot devices to be used in BIOS/EFI/coreboot, which you would also have protected with a password. And that often means you need to also disable USB legacy support, since some BIOSes tend to offer all USB devices as bootable without password otherwise (note that I guess that could also cause problems accessing setup on desktop computers if your only keyboard is USB).

1.99

So to first fix the false instructions in various places - no, setting the superuser in 00_header as instructed is not enough. It might be, but does not apply if eg. old kernels are put into submenu (Ubuntu bug 718670, Fedora bug 836259). The protection from editing does not apply there. And if you remove all but one kernel so that there is no submenu, a submenu will be automatically created when there is a new kernel installed via security updates. I didn't need the submenu feature anyway, so I used to comment out the following lines in /etc/grub.d/10_linux:

#if [ "$list" ] && ! $in_submenu; then
#echo "submenu \"Previous Linux versions\" "
#in_submenu=:
#fi

...

#if $in_submenu; then
# echo " "
#fi

I hope that was useful. I can imagine it causing a couple of family battles if the commonly instructed setup was the only protection used and there's for example a case of two computer savvy siblings that are eager to get to each others' computers...

2.00 & The Question

The problem with 2.00 is that the superusers setup yields a non-bootable system, ie. password is required for booting. But Google wasn't smiling at me today! Terrible. Can you help me (and others) with 2.00? The aim would be to have a 1.99-like setup where superuser password protects all entries from editing, but booting is fine without any passwords.

Update: Thanks, problem solved, see comments! Find the following line in /etc/grub.d/10_linux:

echo "menuentry '$(echo "$os" grub_quote)' $ CLASS \$menuentry_id_option 'gnulinux-simple-$boot_device_id' " sed "s/^/$submenu_indentation/"

And add --unrestricted there. Don't mix the line with the another menuentry line two lines earlier. The submenu problem doesn't exist anymore in 2.00.

FD passing for DRI.Next Using the DMA-BUF interfaces to pass DRI objects between the client and server, as discussed in my previous blog posting on DRI-Next, requires that we successfully pass file descriptors over the X protocol socket. Rumor has it that this has been tried and found to be difficult, and so I decided to do a bit of experimentation to see how this could be made to work within the existing X implementation. (All of the examples shown here are licensed under the GPL, version 2 and are available from git://keithp.com/git/fdpassing) Basics of FD passing The kernel internals that support FD passing are actually quite simple POSIX already require that two processes be able to share the same underlying reference to a file because of the semantics of the fork(2) call. Adding some ability to share arbitrary file descriptors between two processes then is far more about how you ask the kernel than the actual file descriptor sharing operation. In Linux, file descriptors can be passed through local network sockets. The sender constructs a mystic-looking sendmsg(2) call, placing the file descriptor in the control field of that operation. The kernel pulls the file descriptor out of the control field, allocates a file descriptor in the target process which references the same file object and then sticks the file descriptor in a queue for the receiving process to fetch. The receiver then constructs a matching call to recvmsg that provides a place for the kernel to stick the new file descriptor. A helper API for testing I first write a stand-alone program that created a socketpair, forked and then passed an fd from the parent to the child. Once that was working, I decided that some short helper functions would make further testing a whole lot easier. Here s a function that writes some data and an optional file descriptor:

ssize_t
sock_fd_write(int sock, void *buf, ssize_t buflen, int fd)
 
    ssize_t     size;
    struct msghdr   msg;
    struct iovec    iov;
    union  
        struct cmsghdr  cmsghdr;
        char        control[CMSG_SPACE(sizeof (int))];
      cmsgu;
    struct cmsghdr  *cmsg;
    iov.iov_base = buf;
    iov.iov_len = buflen;
    msg.msg_name = NULL;
    msg.msg_namelen = 0;
    msg.msg_iov = &iov;
    msg.msg_iovlen = 1;
    if (fd != -1)  
        msg.msg_control = cmsgu.control;
        msg.msg_controllen = sizeof(cmsgu.control);
        cmsg = CMSG_FIRSTHDR(&msg);
        cmsg->cmsg_len = CMSG_LEN(sizeof (int));
        cmsg->cmsg_level = SOL_SOCKET;
        cmsg->cmsg_type = SCM_RIGHTS;
        printf ("passing fd %d\n", fd);
        *((int *) CMSG_DATA(cmsg)) = fd;
      else  
        msg.msg_control = NULL;
        msg.msg_controllen = 0;
        printf ("not passing fd\n");
     
    size = sendmsg(sock, &msg, 0);
    if (size < 0)
        perror ("sendmsg");
    return size;
 

And here s the matching receiver function:

ssize_t
sock_fd_read(int sock, void *buf, ssize_t bufsize, int *fd)
 
    ssize_t     size;
    if (fd)  
        struct msghdr   msg;
        struct iovec    iov;
        union  
            struct cmsghdr  cmsghdr;
            char        control[CMSG_SPACE(sizeof (int))];
          cmsgu;
        struct cmsghdr  *cmsg;
        iov.iov_base = buf;
        iov.iov_len = bufsize;
        msg.msg_name = NULL;
        msg.msg_namelen = 0;
        msg.msg_iov = &iov;
        msg.msg_iovlen = 1;
        msg.msg_control = cmsgu.control;
        msg.msg_controllen = sizeof(cmsgu.control);
        size = recvmsg (sock, &msg, 0);
        if (size < 0)  
            perror ("recvmsg");
            exit(1);
         
        cmsg = CMSG_FIRSTHDR(&msg);
        if (cmsg && cmsg->cmsg_len == CMSG_LEN(sizeof(int)))  
            if (cmsg->cmsg_level != SOL_SOCKET)  
                fprintf (stderr, "invalid cmsg_level %d\n",
                     cmsg->cmsg_level);
                exit(1);
             
            if (cmsg->cmsg_type != SCM_RIGHTS)  
                fprintf (stderr, "invalid cmsg_type %d\n",
                     cmsg->cmsg_type);
                exit(1);
             
            *fd = *((int *) CMSG_DATA(cmsg));
            printf ("received fd %d\n", *fd);
          else
            *fd = -1;
      else  
        size = read (sock, buf, bufsize);
        if (size < 0)  
            perror("read");
            exit(1);
         
     
    return size;
 

With these two functions, I rewrote the simple example as follows:

void
child(int sock)
 
    int fd;
    char    buf[16];
    ssize_t size;
    sleep(1);
    for (;;)  
        size = sock_fd_read(sock, buf, sizeof(buf), &fd);
        if (size <= 0)
            break;
        printf ("read %d\n", size);
        if (fd != -1)  
            write(fd, "hello, world\n", 13);
            close(fd);
         
     
 
void
parent(int sock)
 
    ssize_t size;
    int i;
    int fd;
    fd = 1;
    size = sock_fd_write(sock, "1", 1, 1);
    printf ("wrote %d\n", size);
 
int
main(int argc, char **argv)
 
    int sv[2];
    int pid;
    if (socketpair(AF_LOCAL, SOCK_STREAM, 0, sv) < 0)  
        perror("socketpair");
        exit(1);
     
    switch ((pid = fork()))  
    case 0:
        close(sv[0]);
        child(sv[1]);
        break;
    case -1:
        perror("fork");
        exit(1);
    default:
        close(sv[1]);
        parent(sv[0]);
        break;
     
    return 0;
 

Experimenting with multiple writes I wanted to know what would happen if multiple writes were made, some with file descriptors and some without. So I changed the simple example parent function to look like:

void
parent(int sock)
 
    ssize_t size;
    int i;
    int fd;
    fd = 1;
    size = sock_fd_write(sock, "1", 1, -1);
    printf ("wrote %d without fd\n", size);
    size = sock_fd_write(sock, "1", 1, 1);
    printf ("wrote %d with fd\n", size);
    size = sock_fd_write(sock, "1", 1, -1);
    printf ("wrote %d without fd\n", size);
 

When run, this demonstrates that the reader gets two bytes in the first read along with a file descriptor followed by one byte in a second read, without a file descriptor. This demonstrates that a file descriptor message forms a barrier within the socket; multiple messages will be merged together, but not past a message containing a file descriptor. Reading without accepting a file descriptor What happens when the reader isn t expecting a file descriptor? Does it just get lost? Does the reader not get the message until it asks for the file descriptor? What about the boundary issue described above? Here s my test case:

void
child(int sock)
 
    int fd;
    char    buf[16];
    ssize_t size;
    sleep(1);
    size = sock_fd_read(sock, buf, sizeof(buf), NULL);
    if (size <= 0)
        return;
    printf ("read %d\n", size);
    size = sock_fd_read(sock, buf, sizeof(buf), &fd);
    if (size <= 0)
        return;
    printf ("read %d\n", size);
    if (fd != -1)  
        write(fd, "hello, world\n", 13);
        close(fd);
     
 
void
parent(int sock)
 
    ssize_t size;
    int i;
    int fd;
    fd = 1;
    size = sock_fd_write(sock, "1", 1, 1);
    printf ("wrote %d without fd\n", size);
    size = sock_fd_write(sock, "1", 1, 2);
    printf ("wrote %d with fd\n", size);
 

This shows that the first passed file descriptor is picked up by the first sockfdread call, but the file descriptor is closed. The second file descriptor passed is picked up by the second sockfdread call. Zero-length writes Can a file descriptor be passed without sending any data?

void
parent(int sock)
 
    ssize_t size;
    int i;
    int fd;
    fd = 1;
    size = sock_fd_write(sock, "1", 1, -1);
    printf ("wrote %d without fd\n", size);
    size = sock_fd_write(sock, NULL, 0, 1);
    printf ("wrote %d with fd\n", size);
    size = sock_fd_write(sock, "1", 1, -1);
    printf ("wrote %d without fd\n", size);
 

And the answer is clearly no the file descriptor is not passed when no data are included in the write. A summary of results

1. read and recvmsg don t merge data across a file descriptor message boundary.
2. failing to accept an fd in the receiver results in the fd being closed by the kernel.
3. a file descriptor must be accompanied by some data.

Make X pass file descriptors I d like to get X to pass a file descriptor without completely rewriting the internals of both the library and the X server. Ideally, without making any changes to the existing code paths for regular request processing at all. On the sending side, this seems pretty straightforward we just need to get the X connection file descriptor and call sendmsg directly, passing the desired file descriptor along. In XCB, this could be done by using the xcbtakesocket interface to temporarily hijack the protocol as Xlib does. It s the receiving side where things are messier. Because a bare read will discard any delivered file descriptor, we must make sure to use recvmsg whenever we want to actually capture the file descriptor. Kludge X server fd receiving Because a passed fd creates a barrier in the bytestream, when the X server reads requests from a client, the read will stop sending data after the message with the file descriptor is consumed. Of course, this process consumes the passed file descriptor, and if that call isn t made with recvmsg set up to receive it, the fd will be lost. As a simple kludge, if we pass a meaningless fd with the X request and then the real fd with a following XNoOperation request, the existing request reading code will get the request, discard the meaningless fd and then stop reading at that point due to the barrier. Once into the request processing code, recvmsg can be called to get the real file descriptor and the associated XNoOperation request. I wrote a test for this that demonstrates how this works:

static void
child(int sock)
 
    uint8_t xreq[1024];
    uint8_t xnop[4];
    uint8_t req;
    int i, reqlen;
    ssize_t size, fdsize;
    int fd = -1, *fdp;
    int j;
    sleep (1);
    for (j = 0;; j++)  
        size = sock_fd_read(sock, xreq, sizeof (xreq), NULL);
        printf ("got %d\n", size);
        if (size == 0)
            break;
        i = 0;
        while (i < size)  
            req = xreq[i];
            reqlen = xreq[i+1];
            i += reqlen;
            switch (req)  
            case 0:
                break;
            case 1:
                if (i != size)  
                    fprintf (stderr, "Got fd req, but not at end of input %d < %d\n",
                         i, size);
                 
                fdsize = sock_fd_read(sock, xnop, sizeof (xnop), &fd);
                if (fd == -1)  
                    fprintf (stderr, "no fd received\n");
                  else  
                    FILE    *f = fdopen (fd, "w");
                    fprintf(f, "hello %d\n", j);
                    fflush(f);
                    fclose(f);
                    close(fd);
                    fd = -1;
                 
                break;
            case 2:
                fprintf (stderr, "Unexpected FD passing req\n");
                break;
             
         
     
 
int
tmp_file(int j)  
    char    name[64];
    sprintf (name, "tmp-file-%d", j);
    return creat(name, 0666);
 
static void
parent(int sock)
 
    uint8_t xreq[32];
    uint8_t xnop[4];
    int i, j;
    int fd;
    for (j = 0; j < 4; j++)  
        /* Write a bunch of regular requests */
        for (i = 0; i < 8; i++)  
            xreq[0] = 0;
            xreq[1] = sizeof (xreq);
            sock_fd_write(sock, xreq, sizeof (xreq), -1);
         
        /* Write our 'pass an fd' request with a 'useless' FD to block the receiver */
        xreq[0] = 1;
        xreq[1] = sizeof(xreq);
        sock_fd_write(sock, xreq, sizeof (xreq), 1);
        /* Pass an fd */
        xnop[0] = 2;
        xnop[1] = sizeof (xnop);
        fd = tmp_file(j);
        sock_fd_write(sock, xnop, sizeof (xnop), fd);
        close(fd);
     
 

Fixing XCB to receive file descriptors Multiple threads may be trying to get replies and events back from the X server at the same time, which means the kludge of having the real fd follow the message will likely lead to the wrong thread getting the file descriptor. Instead, I suspect the best plan will be to fix XCB to internally capture passed file descriptors and save them with the associated reply. Because the file descriptor message will form a barrier in the read stream, xcb can associate any received file descriptor with the last reply in the read data. The X server would then send the reply with an explicit sendmsg call to pass both reply and file descriptor together. Next steps The next thing to do is code up a simple fd passing extension and try to get it working, passing descriptors back and forth to the X server. Once that works, design of the rest of the DRM-Next extension should be pretty straightforward.

Is Asperger Syndrome a Disability? Some people tell me that I m disabled. Usually it s an unstated implication such as referring to Asperger Syndrome as a disability with the assumption that I ll agree. One time I had someone assume that I had never had a paid job because they knew I m an Aspie, maybe I should boast more about my career successes. One interesting take on this is represented by Maco s bost about Disablism/Ablism where she says Vocab note: A person has an impairment. Society s treatment of that impairment is what disables the person [1]. The same concept is presented by BRAINHE in their Social Model of Disability document [2]. The Wikipedia page on Ableism says The ableist worldview holds that disability is an error, a mistake, or a failing, rather than a simple consequence of human diversity, akin to race, ethnicity, sexual orientation or gender [3]. This is fairly close to the position that Neurodiversity [4] advocates take on Autism. Jaarsma P and Welin S wrote an interesting paper titled Autism as a Natural Human Variation: Reflections on the Claims of the Neurodiversity Movement [5] which considers these issues in depth and comes to the conclusion that High Functioning Autism (for which Asperger Syndrome is generally regarded as a synonym) is a difference while Low Functioning Autism is a disability. I think that generally we should accept the opinion of the person in question. Someone who is unable to communicate or is too young to make an informed decision could have their disability status determined by carers. But anyone who is capable of making an informed decision and communicating it should have their opinion respected. I am not going to argue with any of the people who claim that they are disabled due to an Autism Spectrum Disorder. But I don t think that I am disabled and I think that people shouldn t argue with me about this. Over Pathologising Lynne Soraya wrote an interesting article for Psychology Today about one aspect of the supposed over-diagnosis [6]. She responds to Paul Steinberg, a psychiatrist who made a number of claims about Asperger Syndrome which lack evidence. Paul s main idea seems to be that anyone who has social problems but who seems to be successful regardless shouldn t have an Autism Spectrum diagnosis and he claims that such people should be regarded as having a social disability instead. His main idea seems to be that having a diagnosis is a bad thing, but his idea of having a social disability diagnosis instead doesn t seem so great. In many other discussions I ve seen people claim that a large number of diagnosis of anything is a problem. Their idea seems to be that the vast majority of the population shouldn t have a diagnosis for anything and that whenever a significant number of people are diagnosed with a psychological condition (and 1% of the population seems to be a significant number) then it s a problem. I don t think that having a large portion of the population diagnosed is necessarily a problem, I think that it would be OK if the majority of the population was diagnosed with something. The issue is not whether people are diagnosed but what happens after the diagnosis. When a child is diagnosed their parents can help them deal with whatever the issues are this may or may not require further involvement with psychologists or special schools. For the milder cases (of Autism, ADHD, and other conditions) merely knowing what areas will cause difficulty and teaching kids how to deal with them will be enough to solve many problems. When someone is diagnosed as a child but doesn t have obvious symptoms as an adult that is more likely to be an indication that they were taught good coping mechanisms and protected from bad situations as a child not that the diagnosis was wrong. There are some serious issues with special schools and psychiatric drugs, but diagnosis doesn t necessarily imply mistreatment and avoiding a diagnosis is not the correct way to avoid such mistreatment. When someone is diagnosed as an adult they have to learn to deal with it. The general lack of psychologists (waiting times as long as 6 months are common) and the fact that most psychologists won t do any good for someone on the Autism Spectrum is a real problem. But merely knowing the source of your problems is a major step towards alleviating or solving them. One of the arguments that is commonly used against so-called over-diagnosis is that adults don t show apparent symptoms. The issue here is that with some effort and planning adults on the spectrum can act like NTs. Acting like an NT doesn t imply being an NT, it usually requires a lot of ongoing effort that could be applied to other things if society didn t expect us to act like NTs all the time. Conclusion I wish people would stop telling me that I m either disabled or too high functioning to be on the Autism Spectrum. I will never think like an NT and I don t want to, so I ll always be an Aspie. By most objective measures I m at least as successful as the general population in all things that require social skills, so unless something like always losing at Poker is considered a disability I don t think that it s reasonable to consider me to be disabled. It would be nice if I could lock the people who claim that Autism is always a disability in a room with the people who think it s over-diagnosed and let them debate it, no matter which side lost the debate the result would be good! Update: I removed a broken link to a Youtube video, I published this post from a 3G connection and didn t test that the Youtube link still worked. For some reason the author had marked it private since the last time I visited it.

• [1] http://ubuntulinuxtipstricks.blogspot.com.au/2011/05/blogging-against-disablism-day-asl.html
• [2] http://www.brainhe.com/TheSocialModelofDisabilityText.html
• [3] http://en.wikipedia.org/wiki/Ableism
• [4] http://en.wikipedia.org/wiki/Neurodiversity
• [5] http://www.imh.liu.se/avd_halsa_samhalle/filarkiv1/1.264263/JaarsmaWelin2011Autismasanaturalvariation.pdf
• [6] http://www.psychologytoday.com/blog/aspergers-diary/201202/is-aspergers-overdiagnosed

Related posts:

1. Autism vs Asperger Syndrome Diagnostic Changes for Autism Spectrum Disorders Currently Asperger Syndrome (AS)...
2. Is Asperger Syndrome a Good Thing? A meme that keeps going around is that Asperger Syndrome...
3. Autism Awareness and the Free Software Community It s Autism Awareness Month April is Autism Awareness month, there...

Day 9 is a bit of a strange one. I had a rare three evenings spare, but I have been feeling very ill so I didn't achieve much, relatively, despite the increase. I update game-data-packager with experimental support for Hexen 2, which paved the way for uploading the first version of uhexen2, which has been accepted into experimental contrib. I also sponsored a new version of vavoom, both by Gustavo Panizzo. Picking up the pre-freeze musicbrainz work, I contacted the musicbrainz devel mailing list to see if they had any definitive plans to get rid of the old web services API, which a lot of packages in soon-to-be-stable depend upon. I also got in touch with linuxppc-devel regarding suspend to RAM on a PowerPC G4 Mac Mini. This is presently not supported due to lack of support for waking up the graphics chip; however I'm running the mac headless. Finally I want to try and put more time into stuff which is relevant for the next release. The best interface to relevant RC bugs, in my opinion, is the Ultimate Debian Database (UDD) 'bugs' query. I considered hacking a copy of that to display stuff only for a given maintainer, and then track what RC bugs were owned by the Debian Games Team. I didn't get that far, but I did submit an inaugural patch and fix a wiki page.

Well this weeks log comes bit late as my week got extended till today ;-). I had working day at office on Saturday and Sunday due to 2nd Quarter end and instead today is declared as compensatory holiday and tomorrow any how is Gandhi Jayanti. So here it goes. Debian Related Most of my work this week was done for Debian-IN team

• Integrated the KGB bot for Debian-IN team repository. Thanks to gregoa Tincho and dam for extending helping hand in getting it configured
• Updated the setup-repository to install a proper post-receive hook when creating new repositories.
• Copied the set-git-hook from pkg-ruby-extras team and modified it slightly to install post-receive hook for KGB integration for already existing repository. With this script I fixed few of the packages to use KGB which I own. Waiting for Kartik to fix remaining repositories as he is the DD for our team :-)
• Configured my personal instance of KGB which now stands in Debian-IN's channel (#debian-in). But bot is named RAW (Research and Analysis Wing) equivalaent of KGB and CIA in India :-P

Well most of this weeks work involved KGB ;-) Personal Works

• Attended the PyconIndia-2012 on Saturday, Got a NFC card :-) which I can't use as my phone is not NFC capable. Only handful of talks were nice rest were all newbie like talk which even lacked proper presentation skills hence didn't attend on second day. Looks like quality of conference in India is decreasing.
• Finally completed answering all questions on Philosophy and Procedures - 1 part of NM process :-). Phew it took me more time than I expected. Waiting for my AM's comment. Hopefully I will get next bunch of questions soon :-)

Misc Played Cricket after very long time (Guess after 10 long years). Well it was not official match or anything it was just another Gully Cricket. After the match it came to mind that I need regular physical workout :-P. Well that's it Cya till next week :-)

I was working on writing tests for gnome-settings-daemon a week or so ago, and finally got blocked on being unable to set up upower/ConsoleKit/etc. the way I need them. Also, doing so needs root privileges, I don t want my test suite to actually suspend my machine, and using the real service is generally not suitable for test suites that are supposed to run during make check , in jhbuild, and the like these do not have the polkit privileges to do all that, and may not even have a system D-Bus running in the first place. So I wrote a little test_upower.py helper, then realized that I need another one for systemd/ConsoleKit (for the system idle property), also looked at the mock polkit in udisks and finally sat down for two days to generalize this and do this properly. The result is python-dbusmock, I just released the first tarball. With this you can easily create mock objects on D-Bus from any programming language with a D-Bus binding, or even from the shell. The mock objects look like the real API (or at least the parts that you actually need), but they do not actually do anything (or only some action that you specify yourself). You can configure their state, behaviour and responses as you like in your test, without making any assumptions about the real system status. When using a local system/session bus, you can do unit or integration testing without needing root privileges or disturbing a running system. The Python API offers some convenience functions like start_session_bus() and start_system_bus() for this, in a DBusTestCase class (subclass of the standard unittest.TestCase ). Surprisingly I found very little precedence here. There is a Perl module, but that s not particuarly helpful for test suites in C/Vala/Python. And there is Phil s excellent Bendy Bus, but this has a different goal: If you want to thoroughly test a particular D-BUS service, such as ensuring that it does the right thing, doesn t crash on bad input, etc., then Bendy Bus is for you (and python-dbusmock isn t). However, it is too much overhead and rather inconvenient if you want to test a client-side program and just need a few system services around it which you want to set up in different states for each test. You can use python-dbusmock with any programming language, as you can run the mocker as a normal program. The actual setup of the mock (adding objects, methods, properties, etc.) all happen via D-Bus methods on the org.freedesktop.DBus.Mock interface. You just don t have the convenience D-Bus launch API. The simplest possible example is to create a mock upower with a single Suspend() method, which you can set up like this from Python:

import dbus
import dbusmock
class TestMyProgram(dbusmock.DBusTestCase):
[...]
    def setUp(self):
        self.p_mock = self.spawn_server('org.freedesktop.UPower',
                                        '/org/freedesktop/UPower',
                                        'org.freedesktop.UPower',
                                        system_bus=True,
                                        stdout=subprocess.PIPE)
        # Get a proxy for the UPower object's Mock interface
        self.dbus_upower_mock = dbus.Interface(self.dbus_con.get_object(
            'org.freedesktop.UPower', '/org/freedesktop/UPower'),
            'org.freedesktop.DBus.Mock')
        self.dbus_upower_mock.AddMethod('', 'Suspend', '', '', '')
[...]
    def test_suspend_on_idle(self):
        # run your program in a way that should trigger one suspend call
        # now check the log that we got one Suspend() call
        self.assertRegex(self.p_mock.stdout.readline(), b'^[0-9.]+ Suspend$')

This doesn t depend on Python, you can just as well run the mocker like this:

python3 -m dbusmock org.freedesktop.UPower /org/freedesktop/UPower org.freedesktop.UPower

and then set up the mocks through D-Bus like

gdbus call --system -d org.freedesktop.UPower -o /org/freedesktop/UPower \
      -m org.freedesktop.DBus.Mock.AddMethod '' Suspend '' '' ''

If you use it with Python, you get access to the dbusmock.DBusTestCase class which provides some convenience functions to set up and tear down local private session and system buses. If you use it from another language, you have to call dbus-launch yourself. Please see the README for some more details, pointers to documentation and examples. Update: You can now install this via pip from PyPI or from the daily builds PPA. Update 2: Adjusted blog entry for version 0.0.3 API, to avoid spreading now false information too far.

Dear Facebook friend, I like you and I want to chat with you, just not on Facebook! My Facebook account is there to reach you where you are, but I do not hang out there. Please join me at Free and Open spaces instead - via email, IRC or Jabber

Good old email If you cannot be bothered with change, then simply email me! My email address is dr@jones.dk like always

Simple webchat If you are in a hurry and do not use Jabber yet, then try use my Simple Groupchat. Use guest login if you are not a user at jones.dk.

Proper Jabber Jabber is similar to email: You need 1) an account somewhere, 2) some client application setup to use that account, and 3) the address of those you want to connect.

1. Register an account at e.g. Indonesian http://jabin.org/ or some other server.
2. Install a Jabber client program :
   • on Windows try install Gajim from http://gajim.org/.
   • on smartphone try app from https://imo.im/.
   • on Ubuntu try use the Me menu in top right corner.
   • on Debian try install Pidgin: aptitude install pidgin.
3. Invite me as your friend/buddy: my Jabber ID is jonas@jones.dk
4. Tell your friends to do the same!

Geeky IRC Many computer geeks hang out on IRC. I find it ugly and clumsy to use, but most of my techie friends are there, so I recommend going there to discuss technical matters. On OFTC my nickname is jonas, and I often hang out at the chatroom #debian-devel (plus a few others).

Why not Facebook? Facebook is a closed system: all activities happen at one central place. Email, Jabber and IRC are open protocols, exchanging between many Free systems (and some non-free ones as well).

Imagine having an overprotecting mother, who allow you only to meet your friends at one spot in town - where she can look after you. You maybe have no secrets from your mom, but probably still appreciate the option of being alone - to have some privacy.

Facebook is that one spot in town. Jabber is "let's go meet somewhere". Regards, Jonas

Linux 3.6, which was released earlier today, saw continuing improvements in the factoring out of common code in the API lots of new features making drivers smaller and easier to write!

• Initial framework support for GPIO controlled enable signals, saving a bunch of code in drivers.
• Move fixed regulator enable time and voltage mapping table specifications to data.
• Used some of the recent framework enhancements to make voltage change notifications more useful, passing the voltage in as an argument to the notification.
• New drivers for Maxim MAX77686, TI LP872x and LP8788, Samsung S2MPS11, Wolfson Arizona microphone supplies and LDOs

Linux 3.6 was a very quiet release for regmap:

• Support for buses and devices specifying their endianness, providing some performance improvements for MMIO devices.
• Support for wake IRQs in regmap-irq.

Another release of Rcpp has just appeared on CRAN and was just uploaded to Debian. It addresses yet another issue we had on OS X and should hopefully put the build issues to rest. Three new (vectorized) sugar functions were added, along with some new regression tests and more. The complete NEWS entry for 0.9.14 is below; more details are in the ChangeLog file in the package and on the Rcpp Changelog page.

Changes in Rcpp version 0.9.14 (2012-09-30)

• Added new Rcpp sugar functions trunc(), round() and signif(), as well as unit tests for them
• Be more conservative about where we support clang++ and the inclusion of exception_defines.h and prevent this from being attempted on OS X where it failed for clang 3.1
• Corrected a typo in Module.h which now again permits use of finalizers
• Small correction for (unexported) bib() function (which provides a path to the bibtex file that ships with Rcpp)
• Converted NEWS to NEWS.Rd

Thanks to CRANberries, you can also look at a diff to the previous release 0.9.13. As always, even fuller details are on the Rcpp Changelog page and the Rcpp page which also leads to the downloads, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page

Problem #2: How to add a wireless module to a d-i image? Slightly longer version: As reported in #686605, a whole wireless module family is missing from the linux kernel udebs, meaning they re missing from the d-i images. Please note the hypothesis here is that the regular linux kernel images contain the relevant modules, only the udebs don t. Recipe #2: Craft a local udeb!

1. The following assumes one is interested in adding a wireless module, but that can be adapted to other module families. The relevant udeb base name is nic-wireless-modules.
2. When it comes to linux kernel modules, picking the right ones is important, so one has to consider the ABI. Right now, the current ABI in testing is 3.2.0-3-amd64. The complete udeb name is therefore nic-wireless-modules-3.2.0-3-amd64-di.
3. If you re running a kernel with the same ABI, all good, relevant modules should be available under /lib/modules/$(uname -r), which is what will be assumed below. Otherwise, one would have to download a proper linux-image-$ABI package, and extract it into a temporary directory to grab the wanted module(s).
4. Determine the relevant module and its dependencies. As an example, I ll pick a Realtek module: rtl8192cu.ko. If it s loaded, one can use the following to detect its dependencies:

   $ lsmod grep rtl8192cu
   rtl8192cu              78863  0
   rtlwifi                81350  1 rtl8192cu
   rtl8192c_common        52602  1 rtl8192cu
   mac80211              192768  4 iwlwifi,rtl8192c_common,rtlwifi,rtl8192cu
   cfg80211              137140  3 mac80211,iwlwifi,rtlwifi
   usbcore               128498  7 ehci_hcd,usbhid,uvcvideo,rtlwifi,rtl8192cu
   

   But it isn t even needed to load the module to get those dependencies:

   $ /sbin/modinfo /lib/modules/3.2.0-3-amd64/kernel/drivers/net/wireless/rtlwifi/rtl8192cu/rtl8192cu.ko grep ^depends
   depends:        rtlwifi,mac80211,rtl8192c-common,usbcore
   

   Compared to what the d-i image contains, only rtlwifi and rtl8192c-common are missing.
5. Fetch the udeb using http://packages.debian.org/ or browsing your favourite mirror if you know the paths by heart.
6. Extract the udeb into a temporary data/ directory, along with its meta data:

   dpkg -x nic-wireless-modules-3.2.0-3-amd64-di_3.2.23-1_amd64.udeb data
   dpkg --control nic-wireless-modules-3.2.0-3-amd64-di_3.2.23-1_amd64.udeb data/DEBIAN
   

7. Copy the 3 modules under the relevant subdirectory under data/.
8. Update the version in data/DEBIAN/control, for example by adding the +local1 suffix.
9. Build a new binary package, using fakeroot to avoid owner/group noise in the diff:

   fakeroot dpkg -b data nic-wireless-modules-3.2.0-3-amd64-di_3.2.23-1+local1_amd64.udeb
   

10. Use debdiff to check the results:

    $ debdiff nic-wireless-modules-3.2.0-3-amd64-di_3.2.23-1*_amd64.udeb
    Files in second .deb but not in first
    -------------------------------------
    -rw-r--r--  root/root   /lib/modules/3.2.0-3-amd64/kernel/drivers/net/wireless/rtlwifi/rtl8192cu/rtl8192c-common.ko
    -rw-r--r--  root/root   /lib/modules/3.2.0-3-amd64/kernel/drivers/net/wireless/rtlwifi/rtl8192cu/rtl8192cu.ko
    -rw-r--r--  root/root   /lib/modules/3.2.0-3-amd64/kernel/drivers/net/wireless/rtlwifi/rtlwifi.ko
    Control files: lines which differ (wdiff format)
    ------------------------------------------------
    Version: [-3.2.23-1-]  +3.2.23-1+local1+ 
    

11. Put the resulting udeb under localudebs/ in your d-i build tree, and rebuild your image as usual. Almost-tada!

Why almost ? Because a firmware is needed, so one can go back to the 7th step, extract the proper file (lib/firmware/rtlwifi/rtl8192cufw.bin) from the non-free firmware-realtek package, and add it under data/lib/firmware/realtek/. Rebuild the udeb, rebuild the d-i image. Then: tada!

Foreword: This is the first blog post of a mini-series, trying to document some recipes about Debian Installer hacking. VirtualBox will probably be mentioned on a regular fashion, but most of those recipes should be easily adaptable to qemu/kvm. Problem #1: How to share some d-i logs without using the network? Slightly longer version: Investigating some issues on a wired-less virtual machine (Network Adapter 1 Disable Network Adapter), it s hard to get log files out of the virtual machine. Recipe #1: Use serial ports! Serial Ports Port 1 Enable Serial Port, then pick Raw File as Port Mode, and specify a file on the host system to gather logs. There s nothing to do on d-i side, the serial console appears automatically as /dev/ttyS0. From a console, one can extract the needed bits:

cp /var/log/syslog /dev/ttyS0
dmesg > /dev/ttyS0

good to see that there are again Bug Squashing Parties going on, & that quite a few people send patches to the Bug Tracking System &/or make Non-Maintainer Uploads. here are my contributions for this week:

• #636158 maradns: "maradns: Debian default config is not robust to user change + upgrade"
  send a patch to the BTS
• #687604 libkdtree++-dev: "libkdtree++-dev: kdtree headers are unusable on gcc 4.7.1"
  sponsor NMU from Sebastian Ramacher, upload to DELAYED/2
• #687825 src:icinga: "FTBFS on kfreebsd-* due to missing build dependency"
  add info to bug report, fixed by maintainer the day after
• #687879 readpst: "readpst: copyright file missing after upgrade (policy 12.5)"
  replace directory with symlink in postinst, upload to DELAYED/2
• #688112 gummi: "gummi leaks file descriptors leading to a crash if the fd limit is reached"
  add patch from Justus Winter, upload to DELAYED/2
• #688194 prewikka: "prewikka: modifies conffiles (policy 10.7.3): /etc/prewikka/prewikka.conf"
  don't ship configuration file in /etc, upload to DELAYED/2
• #688249 nuauth: "nuauth: postinst deletes files from /usr/share/nuauth/"
  copy template configuration file instead of moving it, upload to DELAYED/2
• #688615 nufw: "nufw: postinst deletes /usr/share/nufw/nufw.conf breaking package reconfiguration"
  copy template configuration files instead of moving them, upload to DELAYED/2

CIA.vc has unfortunately disappeared, and is unlikely to return any time soon. I personally have decided to switch to KGB, but other alternatives such as FBI and irker exist. To switch, you first need to find or set up a kgb bot. If this is a Debian associated FOSS project, feel free to contact me or join #kgb-devel on irc.oftc.net and ask for someone to allow your project to talk to their bot. Once you've found a bot, we need to set up the client. [I'll talk about bot set up at the end.]

kgb-client configuration Install the kgb-client and kgb-client-git packages. Currently, kgb only supports subversion and git, but I suspect this will be rectified shortly. For git repositories, add a post-receive hook like the following:

#!/bin/sh
tee hooks/reflog   kgb-client --conf /path/to/kgbclient.conf --repository git --git-reflog -

For subversion repositories, add a post-commit hook like the following:

#!/bin/sh
kgb-client --conf /path/to/kgbclient.conf --repository svn "$1" "$2"

Then update the configuration file /path/to/kgbclient.conf:

---
repo-id: my-repository
servers:
 - uri: http://servername:9999/
   password: verysecret
# optional link to a website where the commits are;
# needs newish kgb-client and server
web-link: http://example.com/?p=my-repository;a=commitdiff;h=$ commit 

Then, send the bot owner the password, repo-id, channel, and network you'd like the changes to be reported to.

Configuring kgb-bot The bots just listen to soap requests and if the password matches, sends the commit to the appropriate IRC channel. To set one up, install kgb-bot. Then, enable the bot (set BOT_ENBALED=1 in /etc/default/kgb-bot), and configure the bot's configuration file /etc/kgb-bot/kgb.conf:

---
soap:
  server_addr: 0.0.0.0
  server_port: 9999
  service_name: KGB
queue_limit: 150
log_file: "/var/log/kgb-bot.log"
repositories:
  # just a name to identify it
  my-repository:
    # needs to be the same on the client
    password: verysecret
networks:
  oftc:
    nick: KGB-you
    ircname: KGB bot
    username: kgb
    password: ~
    nickserv_password: yournickservpassword
    server: irc.oftc.net
    port: 6667
  freenode:
    nick: KGB-you
    ircname: KGB bot
    username: kgb
    password: ~
    nickserv_password: yournickservpassword
    server: irc.freenode.net
    port: 6667
channels:
  - name: '#your-channel'
    network: oftc
    repos:
     - your-repo
  - name: '#commits'
    network: freenode
    repos:
     - your-repo

Then start the bot (/etc/init.d/kgb-bot start), and watch as it joins channels and reports your changes! You'll probably actually want to register whatever nick you are using on the networks, etc... but you can figure that out yourself!

Thoughts about DRI.Next On the way to the X Developer s Conference in Nuremberg, Eric and I chatted about how the DRI2 extension wasn t really doing what we wanted. We came up with some fairly rough ideas and even held an informal presentation about it. We didn t have slides that day, having come up with the content for the presentation in the hours just before the conference started. This article is my attempt to capture both that discussion and further conversations held over roast pork dinners that week. A brief overview of DRI2 Here s a list of the three things that DRI2 currently offers. Application authentication. The current kernel DRM authentication mechanism restricts access to the GPU to applications connected to the DRM master. DRI2 implements this by having the application request the DRM cookie from the X server which can then be passed to the kernel to gain access to the device. This is fairly important because once given access to the GPU, an application can access any flink d global buffers in the system. Given that the application sends screen data to the X server using flink d buffers, that means all screen data is visible to any GPU-accessing application. This bypasses any GPU hardware access controls. Allocating buffers. DRI2 defines a set of attachment points for buffers which can be associated with an X drawable. An application needing a specific set of buffers for a particular rendering operation makes a request of the X server which allocates the buffers and passes back their flink names. The server automatically allocates new buffers when window sizes change, sending an event to the application so that it knows to request the new buffers at some point in the future. Presenting data to the user. The original DRI2 protocol defined only the DRI2CopyRegion request which copied data between the allocated buffers. SwapBuffers was implemented by simply copy data from the back buffer to the front buffer. This didn t provide any explicit control over frame synchronization, so a new request, DRI2SwapBuffers, was added to expose controls for that. This new request only deals with the front and back buffers, and either copies from back to front or exchanges those two buffers. Along with DRI2SwapBuffers, there are new requests that wait for various frame counters and expose those to GL applications through the OMLsynccontrol extension What s wrong with DRI2? DRI2 fixed a lot of the problems present with the original DRI extension, and made reliable 3D graphics on the Linux desktop possible. However, in the four years since it was designed, we ve learned a lot, and the graphics environment has become more complex. Here s a short list of some DRI2 issues that we d like to see fixed.

• InvalidateBuffers events. When the X window size changes, the buffers created by the X server for rendering must change size to match. The problem is that the client is presumably drawing to the old buffers when the new ones are allocated. Delivering an event to the client is supposed to make it possible for the client to keep up, but the reality is that the event is delivered at some random time to some random thread within the application. This leads to general confusion within the application, and often results in a damaged frame on the screen. Fortunately, applications tend to draw their contents often, so the damaged frame only appears briefly.
• No information about new back buffer contents. When a buffer swap happens and the client learns about the new back buffer, the back buffer contents are always undefined. For most applications, this isn t a big deal as they re going to draw the whole window. However, compositing managers really want to reduce rendering by only repainting small damaged areas of the window. Knowing what previous frame contents are present in the back buffer allows the compositing manager to repaint just the affected area.
• Un-purgable stale buffers. Between the X server finishing with a buffer and the client picking it up for a future frame, we don t need to save the buffer contents and should mark the buffer as purgable. With the current DRI2 protocols, this can t be done, which leaves all of those buffers hanging around in memory.
• Driver-specific buffers. The DRI2 buffer handles are device specific, and so we can t use buffers from other devices on the screen. External video encoders/cameras/encoders can t be used with the DRI2 extension.
• GEM flink has lots of issues. The flink names are global, allowing anyone with access to the device to access the flink data contents. There is also no reference to the underlying object, so the X server and client must carefully hold references to GEM objects during various operations.

Proposed changes for DRI.Next Given the three basic DRI2 operations (authentication, allocation, presentation), how can those be improved? Eliminate DRI/DRM magic-cookie based authentication Kristian H gsberg, Martin Peres, Timoth e Ravier & Daniel Vetter gave a talk on DRM2 authentication at XDC this year that outlined the problems with the current DRM access control model and proposed some fairly simple solutions, including using separate device nodes one for access to the GPU execution environment and a separate, more tightly controlled one, for access to the display engine. Combining that with the elimination of flink for communicating data between applications and there isn t a need for the current magic-cookie based authentication mechanism; simple file permissions should suffice to control access to the GPU. Of course, this ignores the whole memory protection issue when running on a GPU that doesn t provide access control, but we already have that problem today, and this doesn t change that, other than to eliminate the global uncontrolled flink namespace. Allocate all buffers in the application DRI2 does buffer allocation in the X server. This ensures that that multiple (presumably cooperating) applications drawing to the same window will see the same buffers, as is required by the GLX extension. We suspected that this wasn t all that necessary, and it turns out to have been broken several years ago. This is the traditional way in X to phase out undesirable code, and provides an excellent opportunity to revisit the original design. Doing buffer allocations within the client has several benefits:

• No longer need DRI2 additions to manage new GL buffers. Adding HiZ to the intel driver required new DRI2 code in the X server, even though X wasn t doing anything with those buffers at all.
• Eliminate some X round trips currently required for GL buffer allocation.
• Knowing what s in each buffer. Because the client allocates each buffer, it can track the contents of them.
• Size tracking is trivial. The application sends the GL the of the viewport, and the union of all viewports should be the same as the size of the window (or there will be undefined contents on the screen). The driver can use the viewport information to size the buffers and ensure that every frame on the screen is complete.

Present buffers through DMA-buf The new DMA-buf infrastructure provides a cross-driver/cross-process mechanism for sharing blobs of data. DMA-buf provides a way to take a chunk of memory used by one driver and pass it to another. It also allows applications to create file descriptors that reference these objects. For our purposes, it s the file descriptor which is immediately useful. This provides a reliable and secure way to pass a reference from an underlying graphics buffer from the client to the X server by sending the file descriptor over the local X socket. An additional benefit is that we get automatic integration of data from other devices in the system, like video decoders or non-primary GPUs. The Prime support added in DRI version 2.8 hacks around this by sticking a driver identifier in the driverType value. Once the buffer is available to the X server, we can create a request much like the current DRI2SwapBuffers request, except instead of implicitly naming the back and front buffers, we can pass an arbitrary buffer and have those contents copied or swapped to the drawable. We also need a way to copy a region into the drawable. I don t know if that needs the same level of swap control, but it seems like it would be nice. Perhaps the new SwapBuffers request could take a region and offset as well, copying data when swapping isn t possible. Managing buffer allocations One trivial way to use this new buffer allocation mechanism would be to have applications allocate a buffer, pass it to the X server and then simply drop their reference to it. The X server would keep a reference until the buffer was no longer in use, at which point the buffer memory would be reclaimed. However, this would eliminate a key optimization in current drivers the ability to re-use buffers instead of freeing and allocating new ones. Re-using buffers takes advantage of the work necessary to setup the buffer, including constructing page tables, allocating GPU memory space and flushing caches. Notifying the application of idle buffers Once the X server is finished using a buffer, it needs to notify the application so that the buffer can be re-used. We could send these notifications in X events, but that ends up in the twisty mess of X client event handling which has already caused so much pain with Invalidate events. The obvious alternative is to send them back in a reply. That nicely controls where the data are delivered, but causes the application to block waiting for the X server to send the reply. Fortunately, applications already want to block when swapping buffers so that they get throttled to the swap buffers rate. That is currently done by having them wait for the DRI2SwapBuffers reply. This provides a nice place to stick the idle buffer data. We can simply list buffers which have become idle since the last SwapBuffers reply was delivered. Releasing buffer memory Applications which update only infrequently end up with a back buffer allocated after their last frame which can t be freed by the system. The fix for this is to mark the buffer purgable, but that can only be done after all users of the buffer are finished with it. With this new buffer management model, the application effectively passes ownership of its buffers to the X server, and the X server knows when all use of the buffer are finished. It could mark buffers as purgable at that point. When the buffer was sent back in the SwapBuffers reply, the application would be able to ask the kernel to mark it un-purgable again. A new extension? Or just a new DRI2 version? If we eliminate the authentication model and replace the buffer allocation and presentation interfaces, what of the existing DRI2 protocol remains useful? The only remaining bits are the other synchronization requests: DRI2GetMSC, DRI2WaitMSC, DRI2WaitSBC and DRI2SwapInterval. Given this, does it make more sense to leave DRI2 as it is and plan on deprecating, and eventually eliminating, it? Doing so would place a support burden on existing applications, as they d need to have code to use the right extension for the common requests. They ll already need to support two separate buffer management versions though, so perhaps this burden isn t that onerous?

I was invited as a panelist for the Laboratorio Repensar el derecho de autor y el derecho de copia en la era digital:
di logo sobre artes, regulaciones y disponibilidad de la cultura at the beautiful Foro del Dinosaurio in the Museo del Chopo, located very centrally in Mexico City. The list of speakers is quite interesting, and makes me very interested and happy to be there. The laboratory will be next week, Wednesday through Friday. I am scheduled to be part of the 17:00 table, Knowledge availability and regulation in Internet, coordinated by Pedro Mendiz bal (Creative Commons Per ), and together with Juan Vouts s (Biblotecologic Research Institute, UNAM), Armida Aponte (Creative Commons M xico). The other topics that will be covered are:

• Rights, technologies and commons
• The culture and its industries in the digital age: What are the interests at stake?
• Intelectual, cultural and scientific works: Open access or availability?
• New business models around copyright-protected works
• nowledge availability and regulation in Internet
• Visual arts and copyright in the digital media
• Open governments and citizenship: Information, data and intelectual works

Sadly, it does not seem they have planned for remote people to follow along. I will ask and update here if there is any way for people outside Mexico City to tune in For people able to attend, it's free entrance (and certificates will be given to people pre-registered, if you are interested, call 5535-2288 ext. 123) For further details on the participants, go to the laboratory's web page. Update: The talks will be streamed! http://www.chopo.unam.mx/chopoenvivo.html, via UStream.

Some days ago I moved to Aberdeen, Scotland, and I m still experiencing some issues with my Internet connection, so I m not yet able to come back to work on packaging stuff as usual. So please, have some patience, I ll be right back soon

So, yay! Title says it all! On Saturday September 22, Regina and I got married in my parents' house, in Cuernavaca, Morelos. We had a very very nice little party with our family and a small group of friends Of course, due to the nature of our life, we could not forego inviting our family and friends in Argentina, as well as those in other parts of the world, so we set up a simple video stream so that our friends could follow along And they did, with much greater success than what I expected! So, besides those people present with us in Cuernavaca, we had people tuning in (at least to the degree I could get from the log files) from Argentina (Buenos Aires, Paran , Formosa), Austria, Canada, France, Germany, Hungary, Italy, Macedonia, Mexico, Poland, Spain, Sweden and the United States. This next Saturday (September 29) we will have a second party, to which our friends in Mexico are invited, at home. And for the people from far away,, the stream will be available again Expect at least one interesting surprise :) PS- Visit also our wedding page, with some photos, video, and general information (Warning! Part of it is outdated by now)

A couple days ago, I woke up to this exciting series of text messages from a unfamiliar phone number. Because I've not received a reply in the last couple days, because it was a Seattle phone number but I haven't lived in Seattle for years, and because I don't know of anyone in Seattle who was about to give birth, I'm pretty confident that this was indeed a case of misdirected text messages! But whoever you are: Congratulations! I know it was a mistake, but that really made my day!

I ve just read Diego s response to an ill-informed NYT article about data-center power efficiency [1]. This makes me wonder, how much server use does each person have? Google Almost everyone uses Google, most of us use it a lot. The main Google product is also probably the most demanding, their search engine. In a typical day I probably do about 50 to 100 Google searches, that sounds like a lot, but half of them would probably be for one topic that is difficult to find. I don t think that I do that many Google searches because I generally know what I m doing and when I find what I need I spend a lot of time reading it. I m sure that many people do a lot more. Each Google search takes a few seconds to complete (or maybe more if it s an image search and I m on a slow link), but I think it s safe to assume that more than a few seconds of CPU time are involved. How much work would each Google search take if performed on a single system? Presumably Google uses the RAM of many systems as cache which gives a result more similar to a NUMA system than one regular server working for a longer time so there is no way of asking how long it would take to do a Google search with a single server. But I m sure that Google has some ratio of servers to the rate of requests coming in, it s almost certainly a great secret, but we can make some guesses. If the main Google user base comprises people who collectively do an average of 100 searches per day then we can probably guess at the amount of server use required for each search based on the number of servers Google would run. I think it s safe to assume that Google doesn t plan to buy one server for every person on the planet and that they want to have users significantly outnumbering servers. So even for core users they should be aiming to have each user only take a fraction of the resources that one server adds to the pool. So 100 searches probably each take more than 1 second of server use. But they almost certainly take a lot less than 864 seconds (the server use if Google had one server for every 100 daily requests which would imply one server for each of the heavier users). Maybe it takes 10 seconds of server use (CPU, disk, or network whichever is the bottleneck) to complete one search request. That would mean that if the Google network was at 50% utilisation on average then they would have 86400*.5/10/100 == 43 users per server for the core user base who average 100 daily requests. If there are 80M core users that would be about 2M servers, and then maybe something like another 4M servers for the rest of the world. So I could be using 1000 seconds of server time per day on Google searches. I also have a Gmail account which probably uses a few seconds for storing email and giving it to Fetchmail, and I have a bunch of Android devices which use Google calendars, play store, etc. The total Google server use on my behalf for everything other than search is probably a rounding error. But I could be out by an order of magnitude, if it only took 1 second of server use for a Google search then I would be at 100 server seconds per day and Google would only need one server for every 430 users like me. Google also serves lots of adverts on web sites that I visit, I presume that serving the adverts doesn t take much resources by Google standards. But accounting for it, paying the people who host content, and detecting fraud probably takes some significant resources. Other Big Services There are many people who spend hours per day using services such as Facebook. No matter how I try to estimate the server requirements it s probably going to be fairly wrong. But I ll make a guess at a minute of server time per hour. So someone who averages 3 hours of social networking per day (which probably isn t that uncommon) would be using 180 seconds of server time. Personal Servers The server that hosts my blog is reasonably powerful and has two other people as core users. So that could count as 33% of a fairly powerful server in my name. But if we are counting server use per USER then most of the resources of my blog server would be divided among the readers. My blog has about 10,000 people casually reading it through Planet syndication, that could mean that each person who casually reads my blog has 1/30,000 of a server allocated to them for that. Another way of considering it is that 10% of a server (8640 seconds) is covered by me maintaining my blog and writing posts, 20% is for users who visit my blog directly, and 3% is for the users who just see a Planet feed. That would mean that a Planet reader gets 1/330,000 of a server (250ms per day) and someone who reads directly gets 1/50,000 of a server (1.72s per day) as I have about 10,000 people visiting my blog directly in a month. My mail server which is also shared by a dozen or so people (maybe that counts as 5% of a server for me or 4320 seconds per day). Then there s the server I use for SE Linux development (including my Play Machine) and a server I use as a DNS secondary and a shell server for various testing and proxying. Other People s Servers If every reader of a Planet instance like Planet Debian and Planet Linux Australia counts as 1/330,000 of a server for their usage of my blog, then how would that count for my own use of blogs? I tend to read blogs written by the type of people who like to run things themselves, so there would be a lot of fairly under-utilised servers that run blogs. Through Planet Debian and Planet Linux Australia I could be reading 100 or more blogs which are run in the same manner as mine, and in a typical day I probably directly visit a dozen blogs that are run in such a manner. This could give me 50 seconds of server time for blog reading. Home Servers I have a file server at home which is also a desktop system for my wife. In terms of buying and running systems that doesn t count as an extra server as she needs to have a desktop system anyway and using bigger disks doesn t make much difference to the power use (7W is the difference between a RAID-1 server and a single disk desktop system). I also have a PC running as an Internet gateway and firewall. Running servers at home isn t making that much of an impact on my computer power use as there is only one dedicated 24*7 server and that is reasonably low power. But having two desktop systems on 24*7 is a significant factor. Where Power is Used/Wasted No matter how things are counted or what numbers we make up it seems clear that having a desktop system running 24*7 is the biggest use of power that will be assigned to one person. Making PCs more energy efficient through better hardware design and better OS support for suspending would be the best way of saving energy. Nothing that can be done at the server side can compare. Running a server that is only really used by three people is a significant waste by the standards of the NYT article. Of course the thing is that Hetzner is really cheap (and I m not contributing any money) so there isn t a great incentive to be more efficient in this regard. Even if I allocate some portion of the server use to blog readers then there s still a significant portion that has to be assigned to me for my choice to not use a managed service. Running a mail server for a small number of users and running a DNS server and a SE Linux development server are all ways of wasting more power. But the vast majority of the population don t have the skills to run their own server directly, so this sort of use doesn t affect the average power use for the population. Nothing else really matters. No matter what Google does in terms of power use it just doesn t matter when compared to all the desktop systems running 24*7. Small companies may be less efficient, but that will be due to issues of how to share servers among more people and the fact that below a certain limit you can t save money by using less resources particularly if you pay people to develop software. Conclusion I blame Intel for most of the power waste. Android phones and tablets can do some amazing things, which is hardly surprising as by almost every measure they are more powerful than the desktop systems we were all using 10 years ago and by many measures they beat desktop systems from 5 years ago. The same technology should be available in affordable desktop systems. I d like to have a desktop system running Debian based on a multi-core ARM CPU that can drive a monitor at better than FullHD resolution and which uses so little power that it is passively cooled almost all the time. A 64bit ARM system with 8G of RAM a GPU that can decode video (with full Linux driver support) and a fast SSD should compete well enough with typical desktop systems on performance while being quiet, reliable, and energy efficient. Finally please note that most of this post relies on just making stuff up. I don t think that this is wrong given the NYT article that started this. I also think that my estimates are good enough to draw some sensible conclusions.

• [1] http://blog.diegodoval.com/2012/09/23/a-lot-of-lead-bullets-a-response-to-the-new-york-times-article-on-data-center-efficiency/

Related posts:

1. Name Server IP and a Dead Server About 24 hours ago I rebooted the system that runs...
2. Google Custom Search Engine I ve just been experimenting with Google Custom Search [1]. Below...
3. Google Server Design Cnet has an article on the design of the Google...