Search Results: "stone"

28 October 2011

Lars Wirzenius: On the future of Linux distributions

Executive summary Debian should consider: Introduction I've taken a job with Codethink, as part of a team to develop a new embedded Linux system, called Baserock. Baserock is not a distribution as such, but deals with many of the same issues. I thought it might be interesting to write out my new thoughts related to this, since they might be useful for proper distributions to think about too. I'll hasten to add that many of these thoughts are not originally mine. I shamelessly adopt any idea that I think is good. The core ideas for Baserock come from Rob Taylor, Daniel Silverstone, and Paul Sherwood, my colleagues and bosses at Codethink. At the recent GNOME Summit in Montreal, I was greatly influenced by Colin Walters. This is also not an advertisment for Baserock, but since my of my current thinking come from that project, I'll discuss things in the context of Baserock. Finally, I'm writing this to express my personal opinion and thoughts. I'm not speaking for anyone else, least of all Codethink. On the package abstraction Baserock abandons the idea of packages for all individual programs. In the early 1990s, when Linux was new, and the number of packages in a distribution could be counted on two hands in binary, packages were a great idea. It was feasible to know at least something about every piece of software, and pick the exact set of software to install on a machine. It is still feasible to do so, but only in quite restricted circumstances. For example, picking the packages to install a DNS server, or an NFS server, or a mail server, by hand, without using meta packages or tasks (in Debian terminology), is still quite possible. On embedded devices, there's also usually only a handful of programs installed, and the people doing the install can be expected to understand all of them and decide which ones to install on each specific device. However, those are the exceptions, and they're getting rarer. For most people, manually picking software to install is much too tedious. In Debian, we've realised this a great many years ago, and developed meta packages (whose only purpose is to depend on other packages) and tasks (solves the same problem, but differently). These make it possible for a user to say "I want to have the GNOME desktop environment", and not have to worry about finding every piece that belongs in GNOME, and install that separately. For much of the past decade, computers have had sufficient hard disk space that it is no longer necessary to be quite so picky about what to install. A new cheap laptop will now typically come with at least 250 gigabytes of disk space. An expensive one, with an SSD drive, will have at least 128 gigabytes. A fairly complete desktop install uses less than ten gigabytes, so there's rarely a need to pick and choose between the various components. From a usability point of view, choosing from a list of a dozen or two options is much easier than from a list of thirty five thousand (the number of Debian packages as I write this). This is one reason why Baserock won't be using the traditional package abstraction. Instead, we'll collect programs into larger collections, called strata, which form some kind of logical or functional whole. So there'll be one stratum for the core userland software: init, shell, libc, perhaps a bit more. There'll be another for a development environment, one for a desktop environment, etc. Another, equally important reason, to move beyond packages is the problems caused by the combinatorial explosion of packages and versions. Colin Walters talks about this very well. When every system has a fairly unique set of packages, and versions of them, it becomes much harder to ensure that software works well for everyone, that upgrades work well, and that any problems get solved. When the number of possible package combinations is small, getting the interactions between various packages right is easier, QA has a much easier time to test all upgrade paths, and manual test coverage improves a lot when everyone is testing the same versions. Even debugging gets easier, when everyone can easily run the same versions. Grouping software into bigger collections does reduce flexibility of what gets installed. In some cases this is important: very constrainted embedded devices, for example, still need to be very picky about what software gets installed. However, even for them, the price of flash storage is low enough that it might not matter too much, anymore. The benefit of a simpler overall system may well outweigh the benefit of fine-grained software choice. Everything in git In Baserock, we'll be building everything from source in git. It will not be possible to build anything, unless the source is committed. This will allow us to track, for each binary blob we produce, the precise sources that were used to bulid it. We will also try to achieve something a bit more ambitious: anything that affects any bit in the final system image can be traced to files committed to git. This means tracking also all configuration settings for the build, and the whole build environment, in git. This is important for us so that we can reproduce an image used in the field. When a customer is deploying a specific image, and needs it to be changed, we want to be able to make the change with the minimal changes compared to the previous version of the image. This requires that we can re-create the original image, from source, bit for bit, so that when we make the actual change, only the changes we need to make affect the image. We will make it easy to branch and merge not just individual projects, but the whole system. This will make it easy to do large changes to the system, such as transitioning to a new version of GNOME, or the toolchain. Currently, in Debian, such large changes need to be serialised, so that they do not affect each other. It is easy, for example, for a GNOME transition to be broken by a toolchain transition. Branching and merging has long been considered the best available solution for concurrent development within a project. With Baserock, we want to have that for the whole system. Our build servers will be able to build images for each branch, without requiring massive hardware investment: any software that is shared between branches only gets built once. Launchpad PPAs and similar solutions provide many of the benefits of branching and merging on the system level. However, they're much more work than "git checkout -b gnome-3.4-transition". I believe that the git based approach will make concurrent development much more efficient. Ask me in a year if I was right. Git, git, and only git There are a lot of version control systems in active use. For the sake of simplicity, we'll use only git. When an upstream project uses something else, we'll import their code into git. Luckily, there are tools for that. The import and updates to it will be fully automatic, of course. Git is not my favorite version control system, but it's clearly the winner. Everything else will eventually fade away into obscurity. Or that's what we think. If it turns out that we're wrong about that, we'll switch to something else. However, we do not intend to have to deal with more than one at a time. Life's too short to use all possible tools at the same time. Tracking upstreams closely We will track upstream version control repositories, and we will have an automatic mechanism of building our binaries directly from git. This will, we hope, make it easy to follow closely upstream development, so that when, say, GNOME developers make commits, we want to be able to generate a new system image which includes those changes the same day, if not within minutes, rather than waiting days or weeks or months. This kind of closeness is greatly enhanced by having everything in version control. When upstream commits changes to their version control system, we'll mirror them automatically, and this then triggers a new system image build. When upstream makes changes that do not work, we can easily create a branch from any earlier commit, and build images off that branch. This will, we hope, also make it simpler to make changes, and give them back to upstream. Whenever we change anything, it'll be done in a branch, and we'll have system images available to test the change. So not only will upstream be able to easily get the change from our git repository, they'll also be easily verify, on a running system, that the change fixes the problem. Automatic testing, maybe even test driven development We will be automatically building system images from git commits for Baserock. This will potentially result in a very large number of images. We can't possibly test all of them manually, so we will implement some kind of automatic testing. The details of that are still under discussion. I hope to be able to start adding some test driven development to Baserock systems. In other words, when we are requested to make changes to the system, I want the specification to be provided as executable tests. This will probably be impossible in real life, but I can hope. I've talked about doing the same thing for Debian, but it's much harder to push through such changes in an established, large project. Solving the install, upgrade, and downgrade problems All mainstream Linux distributions are based on packages, and they all, pretty much, do installations and upgrades by unpacking packages onto a running system, and then maybe running some scripts from the packages. This works well for a completely idle system, but not so well on systems that are in active use. Colin Walters again talks about this. For installation of new software, the problem is that someone or something may invoke it before it is fully configured by the package's maintainer script. For example, a web application might unpack in such a way that the web server notices it, and a web browser may request the web app to run before it is configured to connect to the right database. Or a GUI program may unpack a .desktop file before the executable or its data files are unpacked, and a user may notice the program in their menu and start it, resulting in an error. Upgrades suffer from additonal problems. Software that gets upgraded may be running during the upgrade. Should the package manager replace the software's data files with new versions, which may be in a format that the old program does not understand? Or install new plugins that will cause the old version of the program to segfault? If the package manager does that, users may experience turbulence without having put on their seat belts. If it doesn't do that, it can't install the package, or it needs to wait, perhaps for a very long time, for a safe time to do the upgrade. These problems have usually been either ignored, or solved by using package specific hacks. For example, plugins might be stored in a directory that embeds the program's version number, ensuring that the old version won't see the new plugins. Some people would like to apply installs and upgrades only at shutdown or bootup, but that has other problems. None of the hacks solve the downgrade problem. The package managers can replace a package with an older version, and often this works well. However, in many cases, any package maintainer scripts won't be able to deal with downgrades. For example, they might convert data files to a new format or name or location upon upgrades, but won't try to undo that if the package gets downgraded. Given the combinatorial explosion of package versions, it's perhaps just as well that they don't try. For Baserock, we absolutely need to have downgrades. We need to be able to go back to a previous version of the system if an upgrade fails. Traditionally, this has been done by providing a "factory reset", where the current version of the system gets replaced with whatever version was installed in the factory. We want that, but we also want to be able to choose other versions, not just the factory one. If a device is running version X, and upgrades to X+1, but that version turns out to be a dud, we want to be able to go back to X, rather than all the way back to the factory version. The approach we'll be taking with Baserock relies on btrfs and subvolumes and snapshots. Each version of the system will be installed in a separate subvolume, which gets cloned from the previous version, using copy-on-write to conserve space. We'll make the bootloader be able to choose a version of the system to boot, and (waving hands here) add some logic to be able to automatically revert to the previous version when necessary. We expect this to work better and more reliably than the current package based one. Making choices Debian is pretty bad at making choices. Almost always, when faced with a need to choose between alternative solutions for the same problem, we choose all of them. For example, we support pretty much every init implementation, various implementations of /bin/sh, and we even have at least three entirely different kernels. Sometimes this is non-choice is a good thing. Our users may need features that only one of the kernels support, for example. And we certainly need to be able to provide both mysql and postresql, since various software we want to provide to our uses needs one and won't work with the other. At other times, the inability to choose causes trouble. Do we really need to support more than one implemenation of /bin/sh? By supporting both dash and bash for that, we double the load on testing and QA, and introduce yet another variable to deal with into any debugging situation involving shell scripts. Especially for core components of the system, it makes sense to limit the flexibility of users to pick and choose. Combinatorial explosion d j vu. Every binary choice doubles the number of possible combinations that need to be tested and supported and checked during debugging. Flexibility begets complexity, complexity begets problems. This is less of a problem at upper levels of the software stack. At the very top level, it doesn't really matter if there are many choices. If a user can freely choose between vi and Emacs, and this does not add complexity at the system level, since nothing else is affected by that choice. However, if we were to add a choice between glibc, eglibc, and uClibc for the system C library, then everything else in the system needs to be tested three times rather than once. Reducing the friction coefficient for system development Currently, a Debian developer takes upstream code, adds packaging, perhaps adds some patches (using one of several methods), builds a binary package, tests it, uploads it, and waits for the build daemons and the package archive and user-testers to report any problems. That's quite a number of steps to go through for the simple act of adding a new program to Debian, or updating it to a new version. Some of it can be automated, but there's still hoops to jump through. Friction does not prevent you from getting stuff done, but the more friction there is, the more energy you have to spend to get it done. Friction slows down the crucial hack-build-test cycle of software development, and that hurts productivity a lot. Every time a developer has to jump through any hoops, or wait for anything, he slows down. It is, of course, not just a matter of the number of steps. Debian requires a source package to be uploaded with the binary package. Many, if not most, packages in Debian are maintained using version control systems. Having to generate a source package and then wait for it to be uploaded is unnecessary work. The build daemon could get the source from version control directly. With signed commits, this is as safe as uploading a tarball. The above examples are specific to maintaining a single package. The friction that really hurts Debian is the friction of making large-scale changes, or changes that affect many packages. I've already mention the difficulty of making large transitions above. Another case is making policy changes, and then implementing them. An excellent example of that is in Debian is the policy change to use /usr/share/doc for documentation, instead of /usr/doc. This took us many years to do. We are, I think, perhaps a little better at such things now, but even so, it is something that should not take more than a few days to implement, rather than half a decade. On the future of distributions Occasionally, people say things like "distributions are not needed", or that "distributions are an unnecessary buffer between upstream developers and users". Some even claim that there should only be one distribution. I disagree. A common view of a Linux distribution is that it takes some source provided by upstream, compiles that, adds an installer, and gives all of that to the users. This view is too simplistic. The important part of developing a distribution is choosing the upstream projects and their versions wisely, and then integrating them into a whole system that works well. The integration part is particularly important. Many upstreams are not even aware of each other, nor should they need to be, even if their software may need to interact with each other. For example, not every developer of HTTP servers should need to be aware of every web application, or vice versa. (It they had to be, it'd be a combinatorial explosion that'd ruin everything, again.) Instead, someone needs to set a policy of how web apps and web servers interface, what their common interface is, and what files should be put where, for web apps to work out of the box, with minimal fuss for the users. That's part of the integration work that goes into a Linux distribution. For Debian, such decisions are recorded in the Policy Manual and its various sub-policies. Further, distributions provide quality assurance, particularly at the system level. It's not realistic to expect most upstream projects to do that. It's a whole different skillset and approach that is needed to develop a system, rather than just a single component. Distributions also provide user support, security support, longer term support than many upstreams, and port software to a much wider range of architectures and platforms than most upstreams actively care about, have access to, or even know about. In some cases, these are things that can and should be done in collaboration with upstreams; if nothing else, portability fixes should be given back to upstreams. So I do think distributions have a bright future, but the way they're working will need to change.

24 October 2011

Colin Watson: Quality in Ubuntu 12.04 LTS

As is natural for an LTS cycle, lots of people are thinking and talking about work focused on quality rather than features. With Canonical extending LTS support to five years on the desktop for 12.04, much of this is quite rightly focused on the desktop. I'm really not a desktop hacker in any way, shape, or form, though. I spent my first few years in Ubuntu working mainly on the installer - I still do, although I do some other things now too - and I used to say only half-jokingly that my job was done once X started. Of course there are plenty of bugs I can fix, but I wanted to see if I could do something with a bit more structure, so I got to thinking about projects we could work on at the foundations level that would make a big difference. Image build pipeline One difficulty we have is that quite a few of our bugs - especially installer bugs, although this goes for some other things too - are only really caught when people are doing coordinated image testing just before a milestone release. Now, it takes a while to do all the builds and then it takes a while to test them. The excellent work of the QA team has meant that testing is much quicker now than it used to be, and a certain amount of smoke-testing is automated (particularly for server images). On the other hand, the build phase has only got longer as we've added more flavours and architectures, particularly as some parts of the process are still serialised per architecture or subarchitecture so ARM builds in particular take a very long time indeed. Exact timings are a bit difficult to get for various reasons, but I think the minimum time between a developer uploading a fix and us having a full set of candidate images on all architectures including that fix is currently somewhere north of eight hours, and that's with people cutting corners and pulling strings which is a suboptimal thing to have to do around release time. This obviously makes us reluctant to respin for anything short of showstopper bugs. If we could get things down to something closer to two hours, respins would be a much less horrible proposition and so we might be able to fix a few bugs that are serious but not showstoppers, not to mention that the release team would feel less burned out. We discussed this problem at the release sprint, and came up with a laundry list of improvements; I've scheduled this for discussion at UDS in case we can think of any more. Please come along if you're interested! One thing in particular that I'm working on is refactoring Germinate, a tool which dates right back to our first meeting before Ubuntu was even called Ubuntu and whose job is to expand dependencies starting from our lists of "seed" packages; we use this, among other things, to generate Task fields in the archive and to decide which packages to copy into our images. This was acceptably quick in 2004, but now that we run it forty times (eight flavours multiplied by five architectures) at the end of every publisher run it's actually become rather a serious performance problem: cron.germinate takes about ten minutes, which is over a third of the typical publisher runtime. It parses Packages files eight times as often as it needs to, Sources files forty times as often as it needs to, and recalculates the dependency tree of the base system five times as often as it needs to. I am confident that we can significantly reduce the runtime here, and I think there's some hope that we might be able to move the publisher back to a 30-minute cycle, which would increase the velocity of Ubuntu development in general. Maintaining the development release Our release cycle always starts with syncing and merging packages from Debian unstable (or testing in the case of LTS cycles). The vast majority of packages in Ubuntu arrive this way, and generally speaking if we didn't do this we would fall behind in ways that would be difficult to recover from later. However, this does mean that we get a "big bang" of changes at the start of the cycle, and it takes a while for the archive to be usable again. Furthermore, even once we've taken care of this, we have a long-established rhythm where the first part of the cycle is mainly about feature development and the second part of the cycle is mainly about stabilisation. As a result, we've got used to the archive being fairly broken for the first few months, and we even tell people that they shouldn't expect things to work reliably until somewhere approaching beta. This makes some kind of sense from the inside. But how are you supposed to do feature development that relies on other things in the development release? In the first few years of Ubuntu, this question didn't matter very much. Nearly all the people doing serious feature development were themselves serious Ubuntu developers; they were capable of fixing problems in the development release as they went along, and while it got in their way a little bit it wasn't all that big a deal. Now, though, we have people focusing on things like Unity development, and we shouldn't assume that just because somebody is (say) an OpenGL expert or a window management expert that they should be able to recover from arbitrary failures in development release upgrades. One of the best things we could do to help the 12.04 desktop be more stable is to have the entire system be less unstable as we go along, so that developers further up the stack don't have to be distracted by things wobbling underneath them. Plus, it's just good software engineering to keep the basics working as you go along: it should always build, it should always install, it should always upgrade. Ubuntu is too big to do something like having everyone stop any time the build breaks, the way you might do in a smaller project, but we shouldn't let things slide for months either. I've been talking to Rick Spencer and the other Ubuntu engineering leads at Canonical about this. Canonical has a system of "rotations", where you can go off to another team for a while if you're in need of a change or want to branch out a bit; so I proposed that we allow our engineers to spend a month or two at a time on what I'm calling the +1 Maintenance Team, whose job is simply to keep the development release buildable, installable, and upgradeable at all times. Rick has been very receptive to this, and we're going to be running this as a trial throughout the 12.04 cycle, with probably about three people at a time. As well as being professional archive gardeners, these people will also work on developing infrastructure to help us keep better track of what we need to do. For instance, we could deploy better tools from Debian QA to help us track uninstallable packages, or we could enhance some of our many existing reports to have bug links and/or comment facilities, or we could spruce up the weather report; there are lots of things we could do to make our own lives easier. By 12.04, I would like, in no particular order: Of course, this overlaps to a certain extent with the kinds of things that the MOTU team have been doing for years, not to mention with what all developers should be doing to keep their own houses in reasonable order, and I'd like us to work together on this; we're trying to provide some extra hands here to make Ubuntu better for everyone, not take over! I would love this to be an opportunity to re-energise MOTU and bring some new people on board. I've registered a couple of blueprints (priorities, infrastructure) for discussion at UDS. These are deliberately open-ended skeleton sessions, and I'll try to make sure they're scheduled fairly early in the week, so that we have time for break-out sessions later on. If you're interested, please come along and give your feedback!

14 October 2011

John Goerzen: Greece part 2: History (and sauntering up to guys with machine guns)

Terah and I went to the Greek island Rhodes recently. This is the second in a series about it. I am one to enjoy history. There is something deeply, well, connecting, about standing in an old place. There is a timeless quality to it a feeling of being connected to so many people of the past, and yet still being connected to change, visible in things such as weathering of stones. To gaze at pottery that s 300 years old, walk past 700-year-old walls, or pass through what remains of the grand portico of an ancient temple to Athena stirs a feeling I can barely explain, of timelessness. Although Rhodes doesn t have the famous Greek sites such as the Parthenon or Delphi, I can t help but wonder why the Rhodes sites aren t better known. They were incredible and it is hard to condense all that we saw into a short blog post. I have to start with the medieval Rhodes Old Town. We got off the bus a few blocks from it one bright morning, and our first task was to find a gate across the moat. Oh yes, A GATE ACROSS THE MOAT. It s a dry moat, and that bridge off in the distance is the gate we were headed to. Outside of the outer wall is a nice quiet walking area. The moat and walls completely surround Old Town and, for the most part, date back about 500 years. The round stones you see on that picture, we were told, were likely surplus from catapults and other projectile weapons. Cross one line of walls and you come to another, with original canons still present. The Knights Hospitaller of St. John, which held Rhodes for a few centuries until the Ottomans captured it, sure knew how to build to impress. The gate we happened to use was Amboise, the Grand Master s Gate. Right there is the stunningly rebuilt landmark Palace of the Grand Master. It is absolutely impossible for any photograph to begin to do this building justice. Between its imported Greek and Roman floors, to the grand nature of everything in it, and the archaeological museum in one corner, it was a fitting start to a visit to Old Town. Here s one of the main staircases. Just near the Palace is quiet courtyard with an old door. Pass through that door and suddenly you re in the midst of the busy Old Town. And among the landmarks in Old Town, the most prominent is Ippoton, the Avenue of the Knights. Along this avenue are the buildings built by the various nationalities of knights, many of which are historical sites in their own. Taken together, it is quite clear why Rhodes is said to be one of the world s best-preserved medieval cities. Down at the other end of Ippoton is the Knights Hospital, which is now part of the archaeological museum. Step off the Avenue a few blocks and you get to some quieter narrow streets just as old, in many cases. On Sunday morning, we were able to visit Mount Filerimos. In contrast to the busy Rhodes, Filerimos had an air of quiet and still to it. It was the site of a monastery, two historic churches, and a landmark Italian cross on the mountaintop. We arrived, and begin our visit with a walk up the quiet stone path. When we got to the top, we walked past this peaceful church. As we walked past the outside, we heard the beautiful music of chant from indoors. We got to step in and listen to mass for a few minutes. In typical fashion, directly in front of the church are two much older sites: one, the ruins of a temple to Athena, and the other a 4th-century Christian bapistery. Rhodes is a popular tourist destination, and of course we saw plenty of popular sites (such as the grandmaster s palace). Filerimos had a few tourists too, but not as many. I frequently like to operate on the plan of going wherever all the tourists aren t. And so, on Filerimos, that meant seeing what was behind the monastery. It started with this peaceful tree-lined path. And the deserted, but intentionally open, gate led to the remains of a Byzantine fortress, which had been a staging area for both the Knights and the Ottomans before their campaigns to capture Rhodes. It also provided incredible views of the surrounding countryside. The first historic site we had visited on our trip was the Acropolis of Lindos, parts of which are 2300 years old. Here s a view of the mountain from the rooftop of the Kalypso, our favorite restaurant in Lindos. The columns of the temple to Athena Lindia are visible, and of course so are the walls. The road up to the acropolis is accessible only on foot or by donkey. It is apparently the only road that has ever been used to get to the acropolis. Here is the partially-restored grand portico to the temple. There s an old Christian church (4th century, if memory serves) at the Acropolis too. The Acropolis makes some pretty good use of natural defenses too. Here s a view from one level of it. There s a manmade wall up there at the very top. And, of course, the beautiful Aegean always in the background. There are lots of cats on Rhodes. Here is a kitten napping at the top of the Lindos Acropolis: Lindos itself is a beautiful town. Here s one of the quieter streets: Notice the pebble steps leading into the houses those intricate pieces of artwork are all over. This post won t be complete without the story of our visit to the Acropolis of Rhodes. We walked there from Old Town. At the Acropolis, there are the remains of a temple to Apollo, an ancient theater, and an ancient stadium where qualifying matches for the Olympics were held. As we got closer to the area, we were repeatedly passed by people dressed in uniforms of various types. And as we got there, we joined a stream of people entering the area. The ancient stadium had apparently thousands of people in it, country names were being read off over the loudspeakers, policemen wielding machine guns were standing by, and we had absolutely no idea what was going on. At this point, you can appreciate the difference between Terah and me. Terah thought that we have no idea what is happening, she was tired from the walk, and so thought we should just leave. I thought that we have no idea what is happening, which is a great reason to stay. So Terah opted to sit and read a bit under some trees while I explored. Here s a view of the stadium as it was emptying out, seen from the theater: I explored the temple and theater, and eventually we were ready to head back. We knew there was a bus back to the New Market (from where we could get a bus back to our hotel), but didn t know where the bus stop was. The obvious place to ask were the policemen, which I thought I would do. Terah thought she would just stay sitting under the trees, on the grounds that the policemen nearest us were all carrying machine guns and perhaps wouldn t like to be disturbed. This led to my cryptic tweet:
Only ONE of us is the kind of person that goes up to guys with machine guns to ask what s happening. Me to Terah today
They told me that it was the preparations for the opening ceremony for a global shooting contest, and also gave me directions to the bus stop.

Rudy Godoy: Dennis M. Ritchie aka dmr

Last weekend Dennis M. Ritchie, creator of the C Programming Language, the UNIX Operating System, The Plan 9 Operating System and many other key contributions to computing, passed away due an extended illness. I learned about it yesterday trough Rob Pike, former dmr s colleague and friend. I m writing this post to honor him and his legacy. While I ve seen people mourn his departure, I ve also noticed most technical people don t really understand why his work is important for today s computing, so I m here to give a refresh. Since I m good with visual tools (mindmaps) I ve made one to describe the extension of dmr s contribution to the world. You can get a glimpse of how this is important. The main topics are their key contributions, the sub-topics are the current technologies that were built upon dmr s work. Click on the image to see a larger resolution version.

The C Programming Language

You can name this a the mother of UNIX, UNIX-like and many programming languages that people use today. The focus on simplicity by giving just a small set of tools is where it s power resides. Most operating systems available, and in development, today are written in C. Most of the services that make the Internet work have been programmed in C. C also influenced other languages such as C++, Objective-C, even Python and Google s Go.
The UNIX operating system
Today s Internet rely in UNIX-like operating systems running crucial services such DNS, webservers, email servers, etc. Nothing of this could be possible without an Operating system that was built with the simplicity and design that it has. The vision/philosophy behind is where it s power resides. write programs that do one thing and do it well . The GNU Project, Linux, the *BSD where conceived on the idea of replicating such invention. Even Windows has some UNIX bits inside.
IPC
This is probably the less-known Dennis Ritchie s contribution. Today is crucial. Web 2.0 would not be able to be AJAXian if there was not IPC. IPC means Interprocess Communication, simply put: make two system processes exchange messages. This foundation was key for concepts such threads, RPC and others. Today s AJAX relies on the RPC concept, for instance.
So, either you are programming the next hot Web 2.0 or just writing a C program for playing with sockets remember there were giants on which shoulders you are standing now. Thank you Dennis, I have the book.
UNIX is very simple, it just needs a genius to understand its simplicity. -Dennis Ritchie

6 October 2011

Vincent Sanders: Introduction to printing in another dimension

Mankind, it would appear, owe a great deal of their evolutionary advantage to using tools. This ability appears to have been massively amplified by our creation of machine tools.

A machine tool is widely defined to be a machine where the movement of the tool (the tool path) is not directly controlled by a human. One of the first known examples is a late 15th century lathe used to cut screw threads . The Industrial revolution was intimately interconnected with the creation of new machine tools and arguably by the mid 19th century all the distinct subtractive machine tool types had been discovered.

I ought to explain the word subtractive in this context, it is a pretty simple and rather arbitrary distinction (but important for this discussion). Traditional machining removes or subtracts material to obtain a finished item akin to a sculptor revealing the statute from within a block of stone by using a chisel and hammer. The corollary to this is, unsurprisingly, the additive process where material is added to create the finished item.

The machine tools from the 19th centuary were primarily single use devices controlled by gears and link mechanisms. Although the Jacquard loom was well known, because of the physical engineering difficulties, combining the concept with a machine tool to create a programmable tool path was not fully realised until the opening of the 20th century.

In the late 1940s electrical motors and punch cards/tape made machine tools Numerically Controlled (NC) and when computers arrived in the 60s we gained Computer Numerical Control (CNC) and the opportunity to completely screw things up with software became available.

With the advent of CNC additive systems became practical and by the late 1980s these machines were being widely used used for Rapid Prototyping.

The first additive systems generally used was the simple pen plotter which added ink on top of paper and became popular in draughting offices for producing blueprints etc. Though more generally thought of as computer printing technique plotters owe their heritage to CNC machines.

Next came prototyping systems based on layered object manufacture which cut shapes in a thin flat material (paper or plastic) and glued them together. These systems were expensive compared to casting processes (use a subtractive machine to make a mould and cast the part), extremely wasteful of source material and the results can be of variable quality. Systems based on this process are still manufactured and used.

Then came the stereolithography approach which scans a focused UV laser to cure resin and build up an object. There are several commercial machines available and even some home built systems but the costs of the resin have not yet made this approach generally cost effective.

Currently the most common commercial rapid prototyping additive systems are selective sintering processes where either an electron beam or a high power laser melt a layer of powdered material on a bed, the bed is lowered, more powder added and the process repeated. This process can use many different types of material and is very flexible as the power used can be plastic or metals. The quality is very high and high resolutions are available. Unfortunately these machines are expensive and generally start around 20,000 which puts them out of most individuals reach.

If anyone is still reading here is the summary of what we have covered so far:
  • Humans have used tools since they stopped being monkeys.
  • More than a century back we figured out how to make machines control the tools.
  • Fifty years back we made computers control the tools, before this all tools were subtractive.
  • In the last twenty years we have discovered several expensive ways to make objects with additive methods.
Now we get to the promise of the title, in the last few years Fused Filament Fabrication has become a viable option for a hobbyist. This method extrudes a thermoplastic through a nozzle and constructs an object one layer at a time from the bottom up.

The RepRap project at Bath university helped kickstart development of a plethora of practical operational 3D printers that can be built or bought. These machines are relatively inexpensive (starting from 400 if you build it yourself) and the feedstock is also reasonably inexpensive.

In another post I will discuss the actual practicalities of building and running one of these devices and looking at their software.



3 October 2011

Francois Marier: Three Firefox extensions to enhance SSL security

There has been a lot of talk recently questioning the trust authorities that underpin the SSL/TLS world. After a few high-profile incidents, it is clear that there is something wrong with this structure.

While some people have suggested that DNSSEC might solve this problem, here are three Firefox add-ons that can be used today to enhance the security of HTTPS:

Unlike the Convergence approach which completely takes over certificate handling, all three of the above add-ons can be used together.

30 September 2011

Axel Beckert: Fun facts from the UDD

After spotting an upload of mira, who in turn spotted an upload of abe (the package, not an upload by me aka abe@d.o), mira (mirabilos aka tg@d.o) noticed that there are Debian packages which have same name as some Debian Developers have as login name. Of course I noticed a long time ago that there is a Debian package with my login name abe . Another well-known Debian login and former package name is amaya. But since someone else came up with that thought, too, it was time for finding the definite answer to the question which are the DD login names which also exist as Debian package names. My first try was based on the list of trusted GnuPG keys:
$ apt-cache policy $(gpg --keyring /etc/apt/trusted.gpg --list-keys 2>/dev/null   \
                     grep @debian.org   \
        	     awk -F'[<@]' ' print $2 '   \
                     sort -u) 2>/dev/null   \
                   egrep -o '^[^ :]*'
alex
tor
ed
bam
ng
But this was not satisfying as my own name didn t show up and gpg also threw quite a lot of block reading errors (which is also the reason for redirecting STDERR). mira then had the idea of using the Ultimate Debian Database to answer this question more properly:
udd=> SELECT login, name FROM carnivore_login, carnivore_names
      WHERE carnivore_login.id=carnivore_names.id AND login IN
      (SELECT package AS login FROM packages, active_dds
       WHERE packages.package=active_dds.login UNION
       SELECT source AS name FROM sources, active_dds
       WHERE sources.source=active_dds.login)
      ORDER BY login;
 login                   name
-------+---------------------------------------
 abe     Axel Beckert
 alex    Alexander List
 alex    Alexander M. List  4402020774 9332554
 and     Andrea Veri
 ash     Albert Huang
 bam     Brian May
 ed      Ed Boraas
 ed      Ed G. Boraas [RSA Compatibility Key]
 ed      Ed G. Boraas [RSA]
 eric    Eric Dorland
 gq      Alexander GQ Gerasiov
 iml     Ian Maclaine-cross
 lunar   J r my Bobbio
 mako    Benjamin Hill
 mako    Benjamin Mako Hill
 mbr     Markus Braun
 mlt     Marcela Tiznado
 nas     Neil A. Schemenauer
 nas     Neil Schemenauer
 opal    Ola Lundkvist
 opal    Ola Lundqvist
 paco    Francisco Moya
 paul    Paul Slootman
 pino    Pino Toscano
 pyro    Brian Nelson
 stone   Fredrik Steen
(26 rows)
Interestingly tor (Tor Slettnes) is missing in this list, so it s not complete either At least I m quite sure that nobody maintains a package with his own login name as package name. :-) We also have no packages ending in -guest , so there s no chance that a package name matches an Alioth guest account either

16 September 2011

John Goerzen: Mexico Part 5: Food, Restaurants, and Dueling Karaoke Guys

The fifth in a series; see also parts 1, 2, 3, and 4. One piece of advice we got in Mexico went like this: the nicer a place looks, the worse the food and prices will be. Roadside taco stands will be great, and nice-looking restaurants not so much. That seemed to be accurate. We only tried one real nice-looking restaurant and it was very good (though pricy), but it may have been sort of an exception. But perhaps the most interesting bit about eating in Mexico wasn t the food. It s the adventure. We ate one day at Guadalajara s San Juan de Dios market. In that huge labyrinth somewhere was a set of restaurants. They d tend to have a small cooking area, usually just open, and a few tables. We chose one. And at this point, I have to take brief detour and explain something. There are a lot of people in Mexico that do things for tips, and quite often without being asked. Some other examples might be washing a car s windshield at a stoplight. So anyhow, we had ordered our food, and before long, a guy wanders down the aisle and plonks down a boombox. And turns it on. And then he pulled out a microphone, which we quite soon realized was connected to the boombox. (I guess making it more of a karaoke box.) Anyhow, he started singing a song decently and seemed to be enjoying it. About 45 seconds into it, a competing boombox man plonked down a competing boombox 25 feet away, turned it on, and yes, you guessed it pulled out a microphone and started singing a different song. Worse than the first person but louder. Eventually the boombox people left and our lunchtime conversation could resume. But pretty soon a drum guy showed up. He had a bunch of drums on a strap so he could just walk around and play them. He apparently decided that an excellent place to play them would be directly behind my head. I did not entirely agree with his decision, but hey, it beat the competing karaoke guys. Eventually the drum guy left, and somehow between the time I looked down to get out money to pay our bill and the time I had it counted out, a clown had shown up and made several balloon animals for our boys. I tipped him, we paid, and then headed on. You might think from this story that this would be an annoying series of events. And honestly, if it had happened in a big mall in the USA, it would probably have been both annoying and creepy. But really I enjoyed it. The fact that dueling karaoke happened, despite sounding really awful, was pretty funny and really seeing this whole parade of people was interesting too. It made American restaurants seem a little boring. You always know what s going to happen here (and if something surprising does happen, the place probably gets a bad review on Yelp.) Interesting things sometimes happen at mealtimes in Mexico and I like it that way. I had a torta ahogada (drowned sandwich) at that restaurant. And at this point, another brief aside. I m the kind of person that can go to an average American restaurant, see items on the menu helpfully indicated as spicy, order one, and genuinely wonder if other people would find them spicy, because I either don t notice spiciness at all, or maybe notice a tiny bit if I concentrate really hard. Others, meanwhile, might take a bite and lunge for the water. Having said that, I know people that lived in Thailand for awhile and I have nowhere near their tolerance for spiciness. So, having been in Mexico a whole 24 hours or so, I decided not to follow Jonathan s wise lead in ordering a torta with the spicy sauce on the side. I figured I hadn t had anything spicy yet, so maybe this was would be nice and mild for me. Via Jonathan s translation, I ordered it with the spicy sauce. I believe the phrase I heard him use was con chile . The waitress looked at me, gave me an amused the American is ordering it con chile? Hahaha . sort of smile, and went off. Pretty soon our food arrived. (The food always seems to arrive pretty soon in Mexico, by the way.) Oliver was having a bit of a culture shock that day, and mostly refusing to eat (once hunger got the best of him later, he really enjoyed Mexican food.) But the rest of us dug in, including me. I enjoyed my torta. It was spicy, but not too bad. I took some big bites (it was, after all, a thick sandwich) and was really enjoying it. For about a minute. Slowest-acting spiciness I ve had in awhile. Then it hit me. Spiciness, and lots of it. I took a big gulp of my horchata (a creamy sweet rice drink that I found at many restaurants). That helped. A little. I really liked the torta and ate it, but it wouldn t surprise me if the waitress noticed how extremely quickly a drank my horchata Another interesting experience was in Guanajuato. It was raining as we walked towards the Guanajuato market. Their market was large and similar in concept to the Guadalajara one, though a lot smaller. The restaurants were all in a row, in a side of the building that was open to the outside. Most were on the ground level but it looked like a restaurant or two were upstairs. As we approached, all of a sudden people were yelling at us. First it was a guy on the second story, then pretty soon people at the restaurants on the first floor did so as well. They were yelling rapidly in Spanish, waving their menus around in the air. I m imagining they were naming foods they sold or reasons to eat there, but I don t know enough Spanish to know. As we walked down the long row of restaurants, the ones we left behind would quiet down in disgust and other hopeful restaurant owners would take up the yelling and waving cause. I imagine if we did some time-lapse videography and walked up and down that row, we could produce an effect not unlike the sound of a dot-matrix printer going back and forth on the page. Anyhow, we selected one of the quieter restaurants pretty much random. The others then quieted down until another person chanced to walk past at which point it would get loud again. The lunch there was good but I think I mainly will remember it for the selection process! On our way into Guanajuato, we stopped at a wonderful roadside taco place. In typical fashion, they had a large vertical pork thing (I don t know the proper word for it) from which they would carve off meat on the spot anytime someone ordered something with pastor. We found a table. And we ordered a few tacos and such. They were usually a few pesos each (working out to less than a dollar), small round things on a soft tortilla, with meat, cilantro, and onion on top. And typically delicious. They had very little in common with an American taco . We d often order a few, and if we wanted more, just order more. They were made quickly enough for that. Tacos were very similar from one restaurant to the next. My favorite flavors were pastor (pork), chorizo (sausage), and bistec (beef steak). A restaurant in Guadalajara sadly I ve forgotten its name, since we kept calling it the potato place had what I might call a Mexican version of the loaded baked potato, with a meat, queso (cheese), a delicious sauce with a flavor unlike anything I d had before, and some garnish. But really my favorite thing from that restaurant was their amazing juices. I am not much of a juice drinker normally, but in Mexico I went for them whenever they were offered. What passes as fruit juice in the USA has about as much resemblance to a real Mexican fruit juice as Taco Bell has to a real Mexican taco stand. (Very little, in case that wasn t crystal clear.) That particular restaurant offered three types of juices, which were, if I m remembering right, aguas, frescas, and jugos. I has a jugo verde (green juice) on the first visit there. It was good, but the one I can still remember was called, I think, the fresa fresca (fresh strawberry juice). And it was incredible. I m not sure how to describe it, other than real. One observation before I end. It seemed a common thread at some Mexican roadside taco stands to not have soap in their restrooms. Instead there would be a plastic cup holding I kid you not powder-form Tide laundry detergent. It was amusing anyhow. My hands left those places extremely soft and smelling like laundry. One of the last restaurants we visited on our trip was in Ajijic, near the Chapala lake. It was actually right on the lake and served seafood. This was the only restaurant with prices as high as I d be used to in the United States. I ordered a stew served in a stone bowl. It came out sizzling, and since the very thick stone bowl retains heat well, it kept sizzling the entire time I was eating. It was excellent as usual. Coming up in part 6: some thoughts on returning to the United States, our decision to visit, communication, and tips for anyone else considering a first visit to Mexico.

23 August 2011

Rudy Godoy: Compute clusters for Debian development and building final report

Compute clusters for Debian development and building final report Summary
-
The goal of this project was intended to have Eucalyptus cloud to
support ARM images so it allows Debian developers and users to be able
to use such facility for taks such package building, software
development (ie. Android) under a Debian pre-set image,
software testing many others. What was expected to have at the end is a modified version of Eucalyptus
Cloud that supports ARM images on the first place. To date this goal has
been reached but is not complete, read production-ready. Extensive test
needs to be done. Besides that we have another goal which is to get
Debian community to use this new extended tool. Project overview
-
Eucalyptus is a hybrid cloud computing platform which has a Open Source
(FLOSS) version. It s targetted to be used for PaaS (Platform as a
Service), IaaS (Infrastructure as a Service) and other distribution
models. It can be used also for cloud management. Given that it has
implemented the EC2, for computing, and the S3, for storage, API, it s
compatible with existing public clouds such Amazon EC2. Currently it
supports running i386 and amd64 images or NC (Node Controllers) in
Eucalyptus naming. Eucalyptus is a complex piece of software. It s architechture it s
modular composed by five components. Cloud Controller (CLC), Walrus (W),
Storage Controller (SC), Cluster Controller (CC) and Node Controller
(NC). The first three are written in Java and the remaining are written
in C. Our project modifications was targetted to the NC component,
altough there is a remaining task for hacking the UI to allow setting
the arch of the uploaded NC image instance. The Node Controller is in charge of setting the proper information for
Eucalyptus to be able to run virtualized images. Eucalyptus uses XEN and
KVM hyphervisor to handle internal virtualization subsystem, and
Libvirt, which is an abstraction of the existing virtualization
libraries, for program interfacing. Having that in mind we are back to our project. For our project to be
sucessful we had various tasks to do. Beginning with understanding such
complex piece of software, followed by hacking the requiered bits and
later integrate the work so it results in a useful tool. Our approach to
the project was then inline with such description. The project s history
I started my participation in GSoC approaching Steffen to discuss about
what was expected, or more accurately, what was in his mind. We
exchanged emails previous to my application and that resulted on my
application being submitted. From the beginning I should tell that I
wasn t much clear on the final product of the project, however we
refined ideas and goals during the weeks following to the official
begining of the project. What was clear for me, after all, is that for
this to see the light and gain adoption Eucalyptus code needed to be
dealt with. Our first task was review ARM image creation from scratch, using the
work done by Aurelien and Dominique in previous GSoC. I ve managed to
get an updated ARM image running under qemu-system-arm. The issues
presented in the past were almost unexistant now that the versatile
kernel is official. You can see this on my first report. After that being done and sometimes in parallel my main goal was then understand
the internals of the Eucalyptus software in order to figure out it s
feasibility, where do we need to extend and how big the task is, we
didn t knew upfront. From the beginning of the project Steffen was kind
to introduce me to the Eucalyptus developers and that have resulted in a
good outcome for Debian, IMHO, to date. Understanding Eucalyptus internals was quite a fun task to say the less. As you
can see on the first part of the report Eucalyptus is modularized and
the component I was expected to work with was the NC (Node
Controller). Given that I isolated my work focus, I started to focus on
learning the internals about that component. Node Controller as described is in charge of talking with
virtualization hyphervisors in order to arrange things to run guest
image instances. The module has basically one main component: handler.c,
in charge to talk to the appropiate hyphervisor and there are
extensions (think of OOP polymorphism but acknoledge it s plain C)
that interact with KVM or XEN. I figured that if qemu-kvm is ran in Hyphervisor mode we can manage to
run an ARM image with qemu-system-arm. Given that Eucalyptus has
interaction with KVM hyphervisor in place, this answered the question of
the project s feasibility. First green light on.
>From this point the scope reduced to interacting with the KVM handler
(node/handler_kvm.c) and extend it to support running an image that is
not amd64 or i386. NC makes use of libvirt to abstract interaction with Hyphervisors. So,
the next phase was learning about it s API and figure what s needed to
be modified next. Libvirt uses a XML file for setting Domain (in
libvirt s naming) definitions. So, Eucalyptus provides a Perl wrapper to
generate this file on runtime and allow the NC to invoke libvirt s API
to run the instance. Next task then, was adapt such script to support
ARM arch. The current script is taiolred for amd64 and i383. I worked on
that front and managed to get a script prototype, that can later be
improved and support more arch s. Generating an adequate Libvirt s XML Domain definition file for ARM can
be a heroic task. There are many things to have in mind given the
diversity of the ARM processor and vendors. I focused on the versatile
flavour given that I was going to test the image I built on the first
place and it ran fine under qemu-system-arm.
The Perl script wrapper then was adapted for such configuration and it
can be tested independently by issuing the following command: $ tools/gen_kvm_libvirt_xml arch arm The arch parameter is what I implemented and it s intended to be
called from the kvm handler on instance creation. With the extensibility
in mind I ve created a hash to associate the arch with their
corresponding emulator. our %arches = (
amd64 => /usr/bin/kvm ,
arm => /usr/bin/qemu-system-arm ,
); Added the arch parameter to the GetOptions() function and used
conditions to tell whether the user is looking for a particular arch,
arm in our case. The most important parts to be considered are the
, and entries. hvm $local_kvm There s also and that require to be tailored for arm. root=0800 As output the script generates an XML template that can be adapted to
your needs and it could be used and tested with tools like Libvirt s
virsh. So that, it can be useful independently of Eucalyptus. I managed
to get to this point before the midterm evaluation. Now that we had the XML wrapper almost done, the next task was to make
the handler call pass the arch as an argument to the script, so the
image is loaded with the proper settings and we are able to run it. Eucalyptus doesn t have a arch field for NC s instances. So, after approaching
Eucalyptus developers, with whom we had already being interacting, I
settled to my proposal of extending the ncInstance struct with an arch
field (util/data.h). The ncInstance_t struct stores the metadata for the
instance being created. It s used for storing runtime data as well as
network configuration and more. It was indeed the right place to add a
new field. I did so by creating the archId field.
Now I needed to make sure the arch information is stored and later used on the
libvirt s call. typedef struct ncInstance_t
char instanceId[CHAR_BUFFER_SIZE];
char imageId[CHAR_BUFFER_SIZE];
char imageURL[CHAR_BUFFER_SIZE];
char kernelId[CHAR_BUFFER_SIZE];
char kernelURL[CHAR_BUFFER_SIZE];
char ramdiskId[CHAR_BUFFER_SIZE];
char ramdiskURL[CHAR_BUFFER_SIZE];
char reservationId[CHAR_BUFFER_SIZE];
char userId[CHAR_BUFFER_SIZE];
char archId[CHAR_BUFFER_SIZE];
int retries; /* state as reported to CC & CLC */
char stateName[CHAR_BUFFER_SIZE]; /* as string */
int stateCode; /* as int */ /* state as NC thinks of it */
instance_states state; char keyName[CHAR_BUFFER_SIZE*4];
char privateDnsName[CHAR_BUFFER_SIZE];
char dnsName[CHAR_BUFFER_SIZE];
int launchTime; // timestamp of RunInstances request arrival
int bootTime; // timestamp of STAGING->BOOTING transition
int terminationTime; // timestamp of when resources are released (->TEARDOWN transition) virtualMachine params;
netConfig ncnet;
pthread_t tcb; /* passed into NC via runInstances for safekeeping */
char userData[CHAR_BUFFER_SIZE*10];
char launchIndex[CHAR_BUFFER_SIZE];
char groupNames[EUCA_MAX_GROUPS][CHAR_BUFFER_SIZE];
int groupNamesSize; /* updated by NC upon Attach/DetachVolume */
ncVolume volumes[EUCA_MAX_VOLUMES];
int volumesSize;
ncInstance; With that in place what was left was modifying the functions that
store/update the ncInstance data, and also the libvirt s function that
calls the gen_kvm_libvirt_xml script. I ve modified the following files:
- util/data.c
- node/handlers_kvm.c
- node/handlers. c,h
- node/test.c
- node/client-marshall-adb.c Most important the allocate_instance() function that is in charge of setting up
the instance metadata a prepare it to pass it to the handler, then
hyphervisor trough Libvirt s API. The function now has a new archId
parameter as well, to keep coherence with the field name. It also
handles whether the field is set or not. We (Eucalyptus developers and
I) haven t settled wether to initialize this field with a default value
or not. I d stepped up and initialized with a NULL value for this string
type var. if (archId != NULL)
strncpy(inst->archId, archId, CHAR_BUFFER_SIZE);
Stores the value only if it s passed to the function. This is essential to
don t break existing functionality and keep consistency for later releases.
With this we almost finished the part of extending Eucalyptus to support
arm images. It took quite long to get to that point. Next step: test. As I mentioned in my previous report the current Eucalyptus packaging
has issues. From the beginning I ve approached the pkg-eucalyptus team,
who were very hepful. So I myself set me up and joined the pkg team with
commit access to the repo. Even the GSoC is not intended for packaging
labour, I needed to clear things on that side because what we want is
adoption so anyone, say a DSA or a user, could setup a cloud that has
support for running arm instances under amd64 arch. Over the weeks between mid-term evaluation and past week I ve worked on
that front. Results were good, as you can see in the pkg-eucalyptus
mailing list and SVN repo.
Eucalyptus has a couple of issues that because of it s nature and complexity
triggered build errors. Those issues came from the Java side of the project and
few form the C-side. The C-part was a problem with AXIS2 Jar used to generate
stubs for later inclusion on build-time. There s no definitive solution to date
because one of the Makefile on gatherlog/Makefile issues a subshell call that
doesn t catches the AXIS_HOME environment variable. I ve worked around this
by defining it as a session env var from .zshrc. The other problem is related
to the Java libary versioning and most likely (we had to investigate further)
to groovy versioning. As I explained Eucalyptus has 3 components written in Java. The Java code uses
extensively the AXIS2 library for implementing webservices calls, the groovy
language for JVM, and many other Java libaries. Eucalyptus open source version
ships both online and offline versions. The difference is basically that
the offline version ships the Java libaries JAR files inside the source code,
so you can build with all required deps. Our former packaging used a file to list which Java libraries the package
depends on and it symlinked them using system wide installed packages. This
rather than helping out triggered a lot of problems I explaing with more detail
on the pkg-eucalyptus ml. I worked around this by telling the debian/rules
not to use such file to symlink and instead use the existing ones from
the source. The last weekend I managed to get to that point and built
sucessfully the software under Debian. I ve done that in order to have a
complete build-cycle for my changes, and then test. In fact today I ve
spotted and fixed a couple of bugs there. What s available now

To date we have a Eucalyptus branch where my patches are sent. The level of
extension is nearly complete, more testing needed. One missing bit is to
change the UI so that the user selects which arch her image is and that
value is passed to the corresponding functions on the Node Controller
component. We also have an ARM image that can be tested and later automated. I ve
recently learned about David Went s VMBuilder repo that extends Ubuntu s
vmbuilder to support image creationg. I might talk to him to adapt the tool
in order to support creation of arm images and use it to upload to an
Eucalyptus cloud. I ve also created a wiki page with different bits on which I was working on.
I ve yet need to craft it to be more educational. I guess I m going to use
this report as a source :) We also had more strong cooperation with the Eucalyptus project and
involved their developers on the packaging and this project, which is
something I consider a great outcome for this project and GSoC goals:
to attract more people to contribute to FLOSS. I ve also been contacted by
people from the Panda Project and we might cooperate also. Future work

I plan to keep working on the project regardless the GSoC ends. Our goal
as Debian could be integrating this into the Eucalyptus upstream
branch. We have good relations with them so expect news from that side.
I also plan to keep working with the Eucalyptus team and related
software in Debian, given that I m familiarized with the tools and
projects.
I m also planning to advocate SoC for Debian in my faculty. To date we
had already 4 students, IIRC, that have participated in the past. Project challenges and lessons
During the project I faced many challenges both in the personal side
and in the technical one. This part is a bit personal but I expect we
learn from this. First challenge was the where do I sit problem. My project was
particular, indeed quite different from the others on it s nature. I had
to work on a software that is not Debian s and then come and say Hey we
have this nice tool for you, come test it! . So, indeed, my focus was on
the understanding of such tool, Eucalyptus, and not much on looking back
at Debian because I didn t feel I had something to advertise yet and
indeed I got most valuable/useful feedback from Eucalyptus developers
rather than Debian s, which for the case is OK IMHO. I had great
mentoring. This situation probably looked a bit of isolation from my
side, from Debian s POV, but it was not intended it s just how it needed
to be IMHO. I felt that I wasn t contributing directly to the
project. I ve spoke about such thing with Steffen a couple of times. I d
like to say that it was also my concern but I probably failed on
communicating this, second challenge. Third challenge was more technical and related to the first one. Since I
was not writing any code for a Debian native package/program but instead for
other project I faced the situation of where to publish my
contributions. Eucalyptus had some issues on managing their FLOSS repo
and the current development one is outdated comparing to the
-src-offline 2.0.3 release. The project didn t fit pkg-eucalyptus repo
neither, even we thought to branch the existing trunk and ship my
contribs as quilt patches, but it involved a lot of more work and burn
on reworking something that has issues now. Later on time I branched the
bzr eucalyptus-devel Launchpad repo and synced there with the
src-offline code. I was slow to react on thing I can say. Fourth challenge was also technical and still dealing with. I was looking to
setup a machine to set a cloud and test my branch. The Dean of my faculty
kindly offered a machine but I ve never managed to get things arranged
with the technical team. Steffen and I discussed about this and evaluated
options, at the end we settled on pushing adoption rather than showcasing it.
Since for adoption we need a use case, I ve spend some SoC money on hardware
for doing this. I should have news this week on that side :) Finally, I d like to thank everyone on the Debian SoC team for this
opportunity to participate. I like to thank Steffen for his effort on
arranging things for me, Graziano from Eucalyptus for his advice on
working with the software code. The different people from Eucalyptus I
interacted with and showed interest on the project s sucess. My
classmates and teachers from the Computer Science School at UCSP in
Arequipa-Peru, friends from the computing community in Peru SPC and
finally my family. Thank you, I learned a lot, specially on the personal
side. As Randy Pausch put: The brick walls are there to give us a
chance to show how badly we want something. Resources
Please see the Wiki page[1] for any reference to the project s resources. 1- http://wiki.debian.org/SummerOfCode2011/BuildWithEucalyptus/ProjectLog Best regards,
Rudy

16 August 2011

John Goerzen: Being Helpful

After a couple of recent storms, we had some large branches down in our yard. Today was a new milestone: it was the first time out boys helped me clean up storm damage. We all trooped outside, and for a minute, the boys kind of ran around while I found a saw. We needed to saw off a few branches that were down but still connected. I did one. Jacob wanted to try, but it was a little too hard. But on the second one, which went a little easier, he helped out and with my guidance actually did some sawing. He was very proud of his work. Then we got into the pickup always a highlight for the boys, even if we only drove it around on the yard. I brought it up close and started loading brush onto it. Jacob went around finding whatever pieces of wood he could carry, and happily threw them onto the pickup. Oliver, on the other hand, meticulously picked up smaller pieces of wood and carefully set them on the rear tire then shoved them off the back. This really annoyed Jacob because Oliver wasn t doing it right, so Jacob kept trying to find the wood Oliver was dropping. And then, of course, they got to play in the pickup bed. Next it was off to the brush pile (also on our yard) to unload our 5ft high pile. I got the really heavy stuff mostly off. I told the boys they could throw whatever they wanted off the pickup. Jacob threw things onto the brush pile, and Oliver, who doesn t really have any aim yet, just threw things and enjoyed trying to figure out where they went. But the big highlight came when the boys got to sit in the back while I pulled the truck a few feet forward and the rest of the brush just fell off the end all by itself ! I complimented both of them on their good work they both really did contribute. Terah commented that when they came in, they both kind of strutted in, looking very tall, proud, and like they did some Good Work with their dad. And later, when Terah said to Oliver, Dad tells me that you were very helpful, Oliver, Jacob overheard and came running in to say, I was very helpful too!

7 August 2011

Lars Wirzenius: Missing roles in software: build engineers

The construction, deployment, and use of software involves a number of roles. I do not have an exhaustive list, but I have a couple in my list that seem to be surprising to some developers. In the traditional Unix (mainframe) model, there are three roles: In my personal reality distortion bubble, this model is badly broken, and has been since the middle of the 1990s. Here's an updated model: A common misconception is that distro developers merely compile software and shove the binaries into glorified tarballs. That is, of course, part of the job, but it's also the part that most distro's tools completely automate. There's a lot more to do, though. For example, just keeping track of known security problems among thirty thousand packages, and making or finding fixes for them, and testing the fixes, and releasing updates for them, takes up quite a bit of effort. Another thing distro developers have to spend effort on is dealing with the multitude of build systems upstream projects use. The freedom to experiment with different solutions is one of the cornerstones of free software development. However, when it comes to build systems, many of them are not serious experiments, and instead are quick hacks thrown together to imperfectly solve problems that sometimes should not exist at all. This highlights one role prominently still missing: build engineer. Their task is to solve the problem of taking the upstream source code, dealing with build-time configuration issues, actually building the software, and then installing it in whatever location the person installing the software wants it installed. Further, their task is to provide the software to do this in a sensible manner. Build engineers already exist, of course. For example, the developers of the autotools suite are such. Every upstream project also has someone who sets up autotools for their project, or some other build tools, or writes them from scratch for that project. However, what's lacking, perhaps, is recognition that build engineering is an important part of software development. Mostly, build systems are set up (or, worse, written) by upstream developers to fit their narrow view of how things should work, a view anchored in the old, broken model of roles. This attitude matters, because for the most part, build systems do not seem to be written to the same quality standards as the rest of the upstream software. For example, when an upstream developer writes a Makefile, and does not consider distro development, they'll often make it hard to differentiate between install-time location and run-time location of various components of the software. Thus, it might be hard for the Debian packager to get the software installed into debian/tmp rather than /usr/local, because the build system hardcodes debian/tmp/usr/lib/foo/plugins as the location of the plugins, since that's where the software gets installed, even if, in the Debian scenario, it's not where it gets run from. Luckily for much software these problems have been solved or avoided. Indeed, as crufty as autotools is, it's one of the things that has kept upstreams sane about their build systems, by providing such good tools that it's more work to write a new, bad one than to use autotools. We currently have roving bands of translators, who move from project to project and translate them into various natural languages. I would like to see a future where there are people dedicated to build engineering, who go from project to project and set up or fix their build problems. A good build system, once set up, should require a minimum of work to keep working. I'd also like to see much improvement in the build software. As good as autotools has been to free software, I'm sure there's a lot of room for improvement. For example, autotools requires setting up a bunch of boilerplate files in each project, which seems like something that could be avoided by relying on specific conventions. However, those improvements are a separate topic. I might go into them some day, if I decide to start developing build tools myself, but until then, I'll let those who do that decide how they want to do it. PS. There are lots more missing roles in my model of free software development, of course. For example, security engineers (find and fix problems, but also provide ways to prevent problems, or limit their scope), internal politicians (keep the project's human wheels from squeaking too much), diplomats (keep different projects from clashing too much), release engineers, etc. Someone should make an exhaustive list.

5 August 2011

Rudy Godoy: Compute Clusters Integration for Debian Development and Building Report 4

Hello, this is the fourth report for the project. First, I d like to
apologize for being late, I took a short trip over the weekend and
missed the deadline. This time I ve good news for my project. As
expected I ve made the necesary changes to have Eucalyptus support ARM
images. The approach was extending and exposing an arch field that
will be used by libvirt s XML domain file. I m working currently on
having the involved functions working with the new extension and I
expect to have this done over the middle of the next week. What was done so far

- Extended the ncInstance_t struct on util/data.h adding the archId
parameter. This struct keeps the information regarding a node (nc),
say RAM, Kernel, etc. Currently it has no way to set an arch in order
to be used later. typdef struct ncInstance_t
...
char reservationId[CHAR_BUFFER_SIZE];
char userId[CHAR_BUFFER_SIZE];
char archId[CHAR_BUFFER_SIZE]; // arch field added
int retries;
...

The Eucalyptus team was OK with this approach that I ve proposed few
weeks ago. For all my changes the approach is maintain compatibility
with existing features.
We still have to define if we are setting a default value, say amd64,
or keep it unset until the user sets one. For now we will be supporting
amd64 and arm as valid fields. - Modified the allocate_intance() function to support the archId
field. This involved adding a new parameter and detecting whether its
present and setting de value accordingly. This function is later used
by the doRunInstance() function in node/handlers_kvm.c - Modified the involved functions that are called by the KVM handlers
and result in passing parameters to the XML file generator for being
used by libvirt s and KVM and have the image running under
Eucalyptus. The new parameters and field are not required and only
being used if they are set with a value. The archId parameter has been
added as the last one, so existing function calls can work. This could
be rearranged later, but will involve a more extensive revision with
Eucalyptus developers. - I ve set a git local repo in order to keep track of the changes to
Eucalyptus. Since this part is a sort of a Eucalyptus branch I took
this approach. Steffen and I are discussing whether to release this as
a pkg-eucalyptus branch (as dpatch, using existing packaging). Either
way I m putting the diff files form my git repo over the wiki and will
put them on my site too. I plan to use git-svn in order to send them
later to the pkg-eucalyptus repo once we are settled with this. - Updated the project s wiki[1] page with information regarding all that s
involved on my work, so it can be re-played or resumed later. 1- http://wiki.debian.org/SummerOfCode2011/BuildWithEucalyptus/ProjectLog What I plan to to over the next weeks
-
- Finish the modifications to the relevant functions. Expected dealine:
next week.
- Do integration tests:
Write a test program, there s an example under node/ that could be
helpful.
Generate the XML domain definition from Eucalyptus functions. The
full cycle: handler_kvm -> allocate_instance -> get_instance_xml ->
have the XML file properly generated -> qemu-kvm runs the instance
using the proper emulator (arm in our case).
- Test runnning the image I ve worked on the project s begining.
- Fixing bits and cooperate with the pkg-eucalyptus team on the
packaging bits.
- Talk to Eucalyptus so they adopt our patches.
- Rock and roll :) What would be nice to have after the project ends
-
- The Debian packaging *needs* work. I ve already discussed this on the
pkg-eucalyptus mailing list. I ve also committed myself to help on
that side even after the SoC ends.
- Talk with upstream and Ubuntu to coordinate the packaging efforts and
modifications to the software.
- Get adopters to use our work :) By now, few weeks about to finish the SoC, I can say that it s been a
good experience and as I stated I m planning to keep maintaing/working
on the software after that. I ve already joined the pkg-eucalyptus team
and plan to contribute actively. I m sad I didn t managed to get Debconf
this year, time was a constraint again, hopefully we can meet in 2012! I
guess that all by now! Best regards,
Rud

4 August 2011

Aigars Mahinovs: Debconf 11 postmortem

Another year, another Debconf and now it is passed. Pictures are processes and all are now uploaded. We are still missing 25 names in the Debconf 11 group photo. With 265 people and 86 Mpix it is the highest resolution image we have had so far (Spain image had a bit more pixels, but a lot of them were outside the actual photo) and the largest number of people (Edinburgh photo had 248 people). The video team produced hundreds of gigabytes of footage, we had very interesting talks and debates and sometimes the AdHoc meeting room on the side was overcrowded with people in BoFs that were not on the initial schedule. It has been a very special kind of conference. As it always is. :)
As always, there were also some problems. I did not read the debconf-team mailing list, but I hear there was plenty of fun discussions to be had in the run-up to the event, some information on getting to the venue was not quite clear (that was quickly fixed as first people arriving to Debcamp were documenting their experience), the organizational dance with the food tickets was more elaborate than usually, both the day trip and the formal dinner were more self-driven than expected, vegetarian complaint level was about the same as usual (which is considered high by some) and the wireless in the hotel was very weak, also Saturday weddings are quite loud and run very late apparently.
To compensate for that we had: very sunny Debcamp, cheap beer (!!!), great quality accommodations, very good looking venue, good network at the venue (after initial scalability issues with the wireless), good food (that did sometimes run out, though), short distances to all locations, oh and did I mention cheep beer? What do we learn from this for next year? I am told that Debconf12 team has already had some meetings specifically to identify and learn from some shortcomings and strengths of this years conference. However, I think everyone of us might have an idea or two to add to the mix as well. For example, I think that BoF rooms are great and we should expand and solidify on that concept: take a page from unconference playbook have a board on-site where people can book slots for their BoF in one of the two rooms for tomorrow. Not far in advance and without moderation requirement of penta scheduling. In the evening the stickers can be taken down and tomorrows BoFs can be added to the schedule at that point, so the people can see online what is going on.
Also I think that we need some kind of coverage of the BoF room(s) from the video team. Even if that is in a form of a single self-service omnidirectional microphone where people could just walk into the room, press a single button and the audio would start streaming and recording.
In addition we could have a screen with the IRC channel of the room projected onto a screen on a side of the stage, to simplify getting questions off the channel. Recording that channel (as a text log) could also be useful.
It feels like post-conference assembly of the talk lists, presentations, links to the relevant recordings, photos, chat logs, ideas, comments, for every talk is very problematic in penta, should we perhaps just use wiki for that?
Any other ideas? And to finish off some more personal travel-inspired rambling. After visiting Paris, Zagreb, Banja Luka and (very shortly) Vienna on this trip it was very interesting to see similarities and draw some parallels, especially with the added context of Berlin and Riga. The similarities in city planning, street shops, brands and the way people go about their lives in these capitals are quite remarkable. I would even say that a trend can be seen where capitols of countries/regions that had troubles in the past (Bosnia more recently; Riga, Zagreb and East Berlin a bit more into the past) are now moving from their different pasts and converging towards the vision of the European capitol city living we see in cities like Paris. If you look at individual pieces you can even recognise milestones along this route and see when specific cities reach them (possibly in a different order than others). With that in mind you can even try to predict what might come next, like looking at Banja Luka and thinking that they might reach the point of upgrading their buses in the next 2-3 years, or comparing Riga and Zagreb we can see that Riga needs to soon choose a street and close it to car traffic to have more caf s, beer gardens and shops concentrated in that area. Or looking at Paris we might find a compelling idea, like clustering a lot of businesses of the same type (such as kitchen design) in the same street, so the customers know where they can go to have all the best choices available within a block. All in all this years Debconf for me was one with the most diverse experiences and the one that felt closest to home. I hope to see you all good people again next year! P.S. I do hope that Debconf12 dates are nailed down in the next 2 months, so we can start talking to air plane companies for group discounts immediately.

3 August 2011

Lars Wirzenius: Obnam 0.19 (backup software)

From the depths of the Andromeda galaxy I bring you version 0.19 of the backup program called Obnam, just released and travelling to the Terran Internet faster than light! This is a BETA release. Some day soon it may well end up being in Debian, unless you prevent that by reporting bugs faster than I can fix them. Meanwhile, my code.liw.fi has .debs for amd64, i386, and armel. Or you can install it from source, presumably (I never do). The NEWS items as listed below. INCOMPATIBILITY CHANGES: BUG FIXES: USER VISIBLE CHANGES: INTERNAL STUFF:

29 July 2011

Jaldhar Vyas: hello_minix-i386.deb

NEW YORK, Jul 29, 2011 PreventaConf The crowd of thousands gathered here to attend PreventaConf 2011 were abuzz over the unveiling of the culmination of 3 years of aimless arsing about^ww^w concentrated development which has led to the porting of dpkg to the MINIX 3 operating system. dselect on Minix Women fainted, men wept openly, medical personnel were kept busy dealing with the many instances of heart seizure, multisensory hallucination and spontaneous combustion which accompanied this momentous event. Much remains to be done. Still this is a major milestone of which the Debian Minix developers are justifiably proud.

20 July 2011

Aigars Mahinovs: Paris preDebConf11

Before arriving to DebConf11, I arranged my flights so that I had a day and a night in Paris as I have not been there before. So after checking out of my work apartment in Tampere, going to work there with all the things from that apartment, then flying back to Riga, going to a midnight screening of the last Potter movie and finally getting back home around 2am, I quickly checked that I have everything needed for the conference and went off to sleep only to wake up at 8am again, pack everything (into a single carry-on bag, after a year of experience travelling by air weekly and watching Up in the Air twice) and be off to the plain to Paris. The CDG airport is quite huge, even compared to something like JFK it feels bigger and more concrete. And yet so dark and lacking interesting features that the only photo I got from it was this IMG_4710 I then used the B line train to get to my hotel(9.1 ), drop my stuff there and another metro ticket (1.7 ) to go to Arc de Triomphe (M tro Charles de Gaulle-Etoile), went up there and walked the length of the Champs-Elys es right up to the Louvre. Then I walked to the river, crossed it to a small island to see the Notre Dame and then went back following the other bank of the river Seine seing both some more normal non-touristy streets of central Paris and also some other magnificent buildings, like Les Invalides and ending up at the Eifel tower. It was a rather easy walk that was only spoiled by some rain in the end so I did not get to take some night photos of the Eiffel tower as I wanted. But the experience still was breathtaking. The scale of the buildings was impressive. The pervasive gold-gilded decorations do expand on the splendour and the same time the full access to all the previously-royal places brings them down to people. The city looks well built and the planning mostly makes sense, especially if you are on foot. Car drivers might have a different opinion about it considering the maze of roads and traffic circles and underpasses I saw in central Paris. It is often said that Riga is a mini-Paris. Mostly, that is said in Riga. But walking the back streets of Paris I can see where that comes from the architecture, the street design, the small cafes and some cobblestone roads and some historical buildings with a few modern building thrown into the mix, and a tower too. Visually Riga does look like Paris a lot, but for the feeling, I still think that Riga feels more like Berlin with its mix of eastern and western Europe. I had one incident when some girl in front of Louvre started to ask me to sign something, gesturing to show that she was mute, it looked like a donation racket, but I did not get to find it out in detail as a swift security guard appeared and started running for the girl, who immediately took off running along with several friends. I guess the guards know them very well. On a more positive note, in one of many confetterie shops I noticed the macaron sweets that my friend advertised loudly some time ago, so I decided to buy and try some. They cost 7 for a box of 6. The taste was very, very sweet. Basically it tastes like froffed up and then frozen sugar made into cookie form with a cream layer between two cookies. IMG_4806 So basically it is like an Oreo cookie, but much, much, much more tender, airy and sweet. The two of the 6 cookies were partly crushed while I tenderly carried them for 5 minutes in their box while searching for a place to sit down and eat them. IMG_4815 After that I also had a proper dinner in a restaurant that was not too near the touristy paths, but still it was a bit lacking. The starter was quite nice I mean it is hard to screw up cutting off a piece of foie gras (basically a piece of liver paste with a layer of extra geese fat), but the main dish was disappointing I ordered a medium rare steak. First of all it came with a side of fries, that in itself was an insult to Paris, baked potatoes would have fit much better with this dish. And despite the main part of the steak being very nicely done to medium rare, the cut was done badly and the edges of the steak had sinew that was too chewy to eat or even cut normally. Other interesting things I liked about Paris:
* high street shops of fashion labels and car manufacturers, that showed off the concept cars and also sold small branded accessories, such as Mercedes Benz key chains IMG_4723 * Statues some made of stone, some of iron, some covered in gold, everywhere and made with great quality IMG_4768 * Bicycle rentals, I mostly saw the locals using them IMG_4754 * Segway tourists IMG_4773 * And countless classic postcard shots of very recognizable architecture IMG_4738 The Eiffel tower was immensely impressive. It towers over everything around, especially because there simply is nothing high enough around it. It would not have been so impressive in the middle of New York, for example. The age and openness of the construction makes you really feel the massiveness of this marvel of functional architecture. But in the end Paris also looks like a great city to just kick back and relax in IMG_4770 P.S. I am amazed that when I put a link to a Flickr image (such as http://www.flickr.com/photos/aigarius/5950187165/in/set-72157627230456160 in a separate paragraph in my WordPress article text, something along the way translates them into proper linked images with a reasonable size. Progress is amazing.

5 July 2011

Rudy Godoy: Compute Clusters Integration for Debian Development and Testing Report 3

Hello this is the third report regarding my project. As usual, quick
summary first. From my last report I have progressed much little than I
would like but, as stated previously also, I m finishing semester
and between exams and projects it takes almost all my time. However, I m
free of academic duties next week and I ll double the speed on the project. What was done so far

- Started working on the modifications to the Eucalyptus code, as
expected.
I ve almost complete the code for the Libvirt s XML Domain
definition. I ve abstracted the wrapper in order to support a broader
number of arches. Previously it was tailored for supporting x86/amd64
based machines. The idea is to have a list of supported arches and
generate the XML file according that. For ARM some parameters and
elements are different than for the others. This would allow for
people to support another architectures, given they are well supported
by KVM-Qemu.
Started working on changes to util/data.c to make sure the correct
parameters are being passed to the XML generator and to libvirt kvm
calls.
- Define and start to setup the testing / demo environment. Basically
involves having one machine to run KVM as hypervisor so the Eucalyptus
handler uses it to run ARM kvm-qemu images.
- Over this weekend I ll be publising the code changes on my
website. Steffen suggested to create some docs regarding the progress
too. I ll do that once I feel I m on track again for my current
pending work, that s coding. What I ll be working over the next two weeks

- My primary goal is to catch up on the work pace and finish the neccesary
code for Eucalyptus to support ARM images. This means to finish
modifications to:
node/handlers_kvm.c handling nc creation. Little changes are need
here from what I ve indentified already.
util/data. c,h handling instance creation. This is the part where
most of the work shall be done. I ve started to work here already.
- Test the code I ve complete already and send the patches to the teams
involved. To date I m still working on a local environment, because no
usable patches have been generated already, just little pieces here
and there.
- Since this is a sort of gluing project, that is code here and there
and make it work together. I m in the process of finding a machine to
be able to test parts of it once they are complete. I m still
contacting people to get such machine, in anycase I gues I can manage
to get one myself. I can t use my current machine since it involves
reinstalling and working at the OS level.
- For the project s demo I ll use both machines, one as the front-end
and the kvm one for NCs (nodes). That s the goal. What is required after that

- I d like to release a package that includes my changes, so it can be
installed either from official packages or my own packages, this
involves:
Updating the packaging bits to the pkg-Eucalyptus repo.
Talking to upstream about it.
- Write the reference Steffen suggested. I believe that I can finish the first part in one week, because of the
work I ve done in the past weeks to understand the Eucalyptus code. I m
also will be talking more to the Eucalyptus team and also to the
maintainers about such changes. I guess that s all for now.

20 June 2011

Rudy Godoy: Compute Clusters Integration for Debian Development and Building Report 2

Hello, this is the 2nd report regarding my project. I'll offer a brief
summary of how are we doing and then the details, so you don't have to
read everything at all.
I have to start saying that over the last two weeks I made a lot of
progress on the project. The most important part: Debian ARM image
integration with Eucalyptus is now cleared. The work to be done for the
next two weeks is pretty much fully coding to achieve what we've committed:
supporting Debian ARM images for Eucalyptus.
What was done so far
--------------------
- I've have identified and mapped what needs to be modified over the
  eucalyptus source for supporting ARM. I'm puting the details on the
  next section. I think this is the most crucial part that is now
  sucessfully achieved.
- I started a Project Log[1] on the wiki, so anyone can check and
  eventyally use the information I've already gathered. The log is being
  updated constantly. You can also check what I've done and what I'm
  working on.
- We got feeback[2] from Eucalyptus developers and packaging
  team regarding developers and packaging team regarding the ultimate
  source for debian/ packaging bits. This was quite important because
  in the past I've managed to build Eucalyptus from source using the debian/
  included in the upstream tarball. However the people in charge
  told us that they were using a different one, which indeed was
  different and resulted in different output than I expected. I'm
  still working on fixing the bits to get a clean build.
- I've joined the pkg-eucalyptus team and I've been given commit rights
  to the repo[2].
1- http://wiki.debian.org/SummerOfCode2011/BuildWithEucalyptus/ProjectLog
2- http://lists.alioth.debian.org/pipermail/pkg-eucalyptus-maintainers/2011-June/000289.html
What I'll be working over the next two weeks
--------------------------------------------
- Heavily code on the different bits that I've identified and need to be
  changed. They are:
  C code:
  - node/handlers_kvm. c,h  - handles nc creation. nc's are node
  clusters for Eucalyptus, the code is in charge of node creation. It
  prepares the "machine" that is able to run a compatible image.
  - util/data. c,h  - handles instance creation. The code over there
  implements the functions node/handlers_* use, like instance creation
  and libvirt's API parameter passing. Since libvirt's parameters for
  ARM vary from x86's ones, we need to support that.
  Perl code:
  - tools/gen_kvm_libvirt_xml - wrapper tool for creating KVM/QEMU
  libvirt's XML Domain definition. This needs to be heavily
  changed/updated - redesigned, to support ARM architecture.
- Depending on the previous task progress: Modify the Debian packaging
  bits for Eucalyptus and push them to the pkg-eucalyptus's team
  repository. This is not crucial for my project's success but I'd like
  to work with Debian official builds. I've yet to commit the
  Eucalyptus team version to the debian/ dir. I expect to have fixed
  the bits to get a successful and clean build, then commit.
Other plans
-----------
- I expect a slow down on the work pace over the second week due end of
  semester exams on the week from July 3 to July 9. After that week I'm
  on vacations and will be fully dedicated to the project.
- Code changes to Eucalyptus source will be pushed upstream. I would
  not like to keep a separate repo for this. I'll talk about this with
  Steffen. Changes to Debian bits will be pushed to the appropiate repos.
Other bits
----------
When the project started I noted it was too vague for some people and
it's results/output were unclear. This concerned me. However, while the
project started and we got lot of feedback things got much clear and I'd
like to take the opportunity to explain it a bit.
Currently, developers targeting ARM devices are struggling[1] with having
a development environment. There are missing bits that sometimes are not
in place and result on hard time for them. Also, due mobile devices
growth and vendors using Linux-based[2] operating systems it's a nice
opportunity for Debian to keep it's well known fame of: supporting every
arch over the sun. How this project fits, well by offering the ability
for developers to run ARM-based Debian images over x86/x86_64 CPU's
would impact the way they could be more productive and for Debian to be
more adopted as development environment.
I can say that I'm very enthusiast of this opportunity and I'm
certain that the outcome will result in more adoption. Even the Ubuntu
people is now pushing more and more efforts for ARM official support,
due the facts I've stated.
1- http://lists.debian.org/debian-arm/2011/06/msg00020.html
2- http://events.linuxfoundation.org/events/android-builders-summit/back

8 June 2011

Matt Zimmerman: DEX finishes first batch of derivative patches for Debian

It s been a few months since Zack and I announced the DEX project, which aims to improve the Debian ecosystem by working jointly with derivative distributions. Our first milestone The goal of our first project, nicknamed ancient-patches, was to clear out an old batch of a few hundred Ubuntu patches whose status was unclear. We couldn t tell which ones had been merged into Debian, which were waiting in the BTS, and which had yet to be submitted to Debian. All of them were several years old. I m pleased to announce that this project is now complete. Thanks to help from David Paleino, Colin Watson, Nathan Handler and Steve Langasek, we were able to clear over 95% of the patches in a matter of days. These were the easy ones: patches which were obsolete, or had already been applied. We discussed the remainder, and resolved all of the patches whose status was still unclear. This left the harder ones: patches stalled in the BTS, and patches where there was no consensus about what to do with them. One of the stalled patches was merged into Debian via an NMU, eliminating the delta between Debian and Ubuntu. Another had been submitted to Debian by a third party, but was no longer shipping in Ubuntu, so we considered it obsolete for purposes of this project. This has left only two patches out of the original list of 277. Both of them are filed in the BTS and have been discussed with the relevant maintainer team. One of them is expected to be obsoleted when a new upstream version is packaged, which implements similar functionality. The other is being discussed with the upstream developers, but there is no conclusion yet about whether it can be merged upstream or in Debian. Conclusions Although we weren t quite able to clear the whole list, we still consider the project to be a success because: What s next In the most recent DEX update on debian-derivatives, I highlighted a few important events for DEX: We re looking forward to seeing DEX develop further. If you d like to get involved, come and join us on the debian-derivatives mailing list or IRC (#debian-derivatives on freenodeOFTC). Matt Zimmerman and Stefano Zacchiroli

4 June 2011

Rudy Godoy: GSoC: Status report 01

Hello, this is the official project report for the Debian GSoC admins and everyone. It s been few days since the official coding phase started. Most of the time I ve been figuring out the details regarding integration of non-x86 images to the Eucalyptus IaaS and working on the ARM image, as you ll learn later. I m also in the process of starting to design the set of tools we ll be delivering as part of the project. The project s goal is to be a useful resource for Debian developers and porters. I ve joined some porters list in order to learn what are the things they are struggling with and how the project can help to address them. I ll be starting a wiki page with such topics. Bonding period:
Bonding period has been more fruitful than I expected. As mentioned in my previous post, Steffen and I got in touch with the Eucalyptus team, since they show their early interest on the project. We are coordinating cooperation between our teams. Besides my mentor and I defined to start working on the ARM image. Coordination went OK. We ll be having meetings every two weeks. Project Status:
Resuming Dominique s work I began working on the ARM image. Aur lien s work was a great resource. Currently I have a working and updated Debian testing ARM image using the versatile kernel that can be used under qemu-arm. I m currently making tests and figuring out the next phase which is cloudify it. Over this weekend I expect to finish on that side and then began on the design and implementation of the tools that will be the project s result.
Until now I haven t faced much issues, but the Eucalyptus part since the docs are targeting x86 and amd_64 archs. However I think I can run qemu-arm on top. I ll talk to them about the idea.
Future plans: week 3: - Finish tests with the ARM image. - Design the base tool-set, thinking integration with dpkg-buildpackage and qemubuilder. week 4-8: - Write the necessary code. mid-term evaluation. Other bits: Latest days have been quite busy for me, since I m still have to attend class, my semester ends in five weeks. This week was quite overwhelming with media , my University published an article (Spanish) about me and the program. I m the third student of the CS school participating in GSoC, in the past we had Gentoo Project and KDE students. Later, a local Cable TV program interview me on the same matter, it went fine. I managed to talk about Debian and what we do (we get free Advertising yay!) :) also about Google s Summer of Code program and invited students to participate both in FLOSS projects and the program. Still don t know if they have put it on air, I think it will be next week.

Next.

Previous.