Search Results: "ana"

3 June 2012

Johannes Schauer: port bootstrap build-ordering tool report 1

A copy of this post is sent to soc-coordination@lists.alioth.debian.org as well as to debian-bootstrap@lists.mister-muffin.de. Diary May 21 May 22 May 23 May 24 May 25 May 29 May 30 May 31 June 1 June 2 Results I learned a good chunk of ocaml and how to use dose3 and libcudf. I created a gitorious project and a git repository for all the sourcecode.
git clone git://gitorious.org/debian-bootstrap/bootstrap.git
The git as of now contains 30 commits and 1197 lines of ocaml code. So far, 62 emails have been exchanged between me and Pietro and Wookey. I created a mailinglist for this project where all email exchange so far is publicly accessible in the archives. You can also download all of the email exchange in mbox format. Everybody is welcome to join and/or read the list. What seems to be finished: the program that finds the minimal amount of source packages that have to be cross compiled to end up with a minimal build system. What it does is:
  1. get all essential packages
  2. get their runtime dependencies
  3. get build-essential plus runtime dependencies
  4. get all source packages that are necessary to build 1.-3. those are the packages that have to be cross compiled
  5. get a list of all packages that are built by source packages from 4.
  6. add all packages from 1.,2.,3. and 5. plus all arch:all packages to a universe
  7. use Depsolver.trim on that universe to figure out which of those packages are actually installable
The result of 7. will then contain a list of packages that are available automatically on the foreign system due to cross compiled source packages and arch:all packages. For Debian Sid, the output of my program is:
# (1) number of packages with priority:required: 62
# (2) plus, number of dependencies of priority:required packages: 20
# (3) plus, build-essential and dependencies: 31
# number of source packages to build the above: 71
# number of additional packages built from the above source packages: 292
# (4) number of packages of those plus arch:all packages that are installable: 6421
# total number of installable packages (1)+(2)+(3)+(4): 6534
For Ubuntu Natty it is:
# (1) number of packages with priority:required: 96
# (2) plus, number of dependencies of priority:required packages: 7
# (3) plus, build-essential and dependencies: 31
# number of source packages to build the above: 87
# number of additional packages built from the above source packages: 217
# (4) number of packages of those plus arch:all packages that are installable: 2102
# total number of installable packages (1)+(2)+(3)+(4): 2236
So for Debian, 71 source packages definitely have to be made cross compilable while for Natty, the number is 87. The last two days I was toying around with these minimal systems to see how big the number of source packages is, that can be built on top of them without running into dependency cycles. After installing the binary packages that were built, I checked again until no new packages could be built. For Natty, I was only able to find 28 additional packages that can be built on top of the 2236 existing ones. This means that a number of dependency cycles prevent building anything else. In the coming two weeks I will focus on coming up with a tool that cleverly helps the user to identify packages that would be useful to have for building more packages (probably determined by how many packages depend on it - debhelper is an obvious candidate). The tool would then show why that crucial package is not available (in case of debhelper because some of its runtime dependencies are not available and require debhelper to be built) and how the situation can best be resolved. The possible methods to do so are to identify a package that is part of a cycle and either cross compile it or let it have staged build dependencies.

Johannes Schauer: cross-compilable and bootstrappable Debian

When packaging software for Debian, there exist two important assumptions:
  1. Compilation is done natively
  2. Potentially all of Debian is available at compile time
Both assumptions make the life of a package maintainer much easier and they do not create any problem unless you are one of the unlucky few who want to run Debian on an architecture that it does not yet exist for. You will then have to use other distributions like OpenEmbedded or Gentoo which you compiled (or retrieved otherwise) for that new architecture to hack a core of Debian source packages until they build a minimal Debian system that you can chroot into and continue natively building the rest of it. But even if you manage to get that far you will continue to be plagued by cyclic build and runtime dependencies. So you start to hack source packages so that they drop some dependencies and you can break enough cycles to advance step by step. The Debian ports page lists 24 ports of Debian, so despite its unpleasant nature, porting it is something that is not done seldom. The process as laid out above has a number of drawbacks: If Debian would provide a set of core packages that are cross-compilable and which suffice for a minimal foreign build system, and if it would also have enough source packages that provide a reduced build dependency set so that all dependency cycles can be broken, building Debian for a yet unknown architecture could be mostly automated. The benefits would be: With three of this year's GSoC projects, this dream seems to come into reach. There is the "Multiarch Cross-Toolchains" project by Thibaut Girka and mentored by Hector Oron and Marcin Juszkiewicz. Cross-compiling toolchains need packages from the foreign architecture to be installed alongside the native libraries. Cross-compiler packages have been available through the emdebian repositories but always were more of a hack. With multiarch, it is now possible to install packages from multiple architectures at once, so that cross-compilation toolchains can be realized in a proper manner and therefor can also enter the main archives. Besides creating multiarch enabled toolchains, he will also be responsible for making them build on the Debian builld system as cross-architecture dependencies are not yet supported. There is also the "Bootstrappable Debian" project by Patrick "P. J." McDermott and mentored by Wookey and Jonathan Austin. He will make a small set of source packages multiarch cross-compilable (using cross-compilers provided by Thibaut Girka) and add a Build-Depends-StageN header to critical packages so that they can be built with reduced build dependencies for breaking dependency cycles. He will also patch tools as necessary to recognize the new control header. And then there is my project: "Port bootstrap build-ordering tool" (Application). It is mentored by Wookey and Pietro Abate. In contrast to the other two, my output will be more on the meta-level as I will not modify any actual Debian package or patch Debian tools with more functionality. Instead the goal of this project is threefold:
  1. find the minimal set of source packages that have to be cross compiled
  2. help the user to find packages that are good candidates for breaking build dependency cycles through added staged build dependencies or by making them cross-compilable
  3. develop a tool that takes the information about packages that can be cross compiled or have staged build dependencies to output an ordering with which packages must be built to go from nothing to a full archive
More on that project in my follow-up post.

2 June 2012

Petter Reinholdtsen: Debian Edu interview: Mike Gabriel

Back in 2010, Mike Gabriel showed up on the Debian Edu and Skolelinux mailing list. He quickly proved to be a valuable developer, and thanks to his tireless effort we now have Kerberos integrated into the Debian Edu Squeeze version. Who are you, and how do you spend your days? My name is Mike Gabriel, I am 38 years old and live near Kiel, Schleswig-Holstein, Germany. I live together with a wonderful partner (Angela Fu ) and two own children and two bonus children (contributed by Angela). During the day I am part-time employed as a system administrator and part-time working as an IT consultant. The consultancy work touches free software topics wherever and whenever possible. During the nights I am a free software developer. In the gaps I also train in becoming an osteopath. Starting in 2010 we (Andreas Buchholz, Angela Fu , Mike Gabriel) have set up a free software project in the area of Kiel that aims at introducing free software into schools. The project's name is "IT-Zukunft Schule" (IT future for schools). The project links IT skills with communication skills. How did you get in contact with the Skolelinux/Debian Edu project? While preparing our own customised Linux distribution for "IT-Zukunft Schule" we were repeatedly asked if we really wanted to reinvent the wheel. What schools really need is already available, people said. From this impulse we started evaluating other Linux distributions that target being used for school networks. At the end we short-listed two approaches and compared them: a commercial Linux distribution developed by a company in Bremen, Germany, and Skolelinux / Debian Edu. Between 12/2010 and 03/2011 we went to several events and met people being responsible for marketing and development of either of the distributions. Skolelinux / Debian Edu was by far much more convincing compared to the other product that got short-listed beforehand--across the full spectrum. What was most attractive for me personally: the perspective of collaboration within the developmental branch of the Debian Edu project itself. In parallel with this, we talked to many local and not-so-local people. People teaching at schools, headmasters, politicians, data protection experts, other IT professionals. We came to two conclusions: First, a technical conclusion: What schools need is available in bits and pieces here and there, and none of the solutions really fit by 100%. Any school we have seen has a very individual IT setup whereas most of each school's requirements could mapped by a standard IT solution. The requirement to this IT solution is flexibility and customisability, so that individual adaptations here and there are possible. In terms of re-distributing and rolling out such a standardised IT system for schools (a system that is still to some degree customisable) there is still a lot of work to do here locally. Debian Edu / Skolelinux has been our choice as the starting point. Second, a holistic conclusion: What schools need does not exist at all (or we missed it so far). There are several technical solutions for handling IT at schools that tend to make a good impression. What has been missing completely here in Germany, though, is the enrolment of people into using IT and teaching with IT. "IT-Zukunft Schule" tries to provide an approach for this. Only some schools have some sort of a media concept which explains, defines and gives guidance on how to use IT in class. Most schools in Northern Germany do not have an IT service provider, the school's IT equipment is managed by one or (if the school is lucky) two (admin) teachers, most of the workload these admin teachers get done in there spare time. We were surprised that only a very few admin teachers were networked with colleagues from other schools. Basically, every school here around has its individual approach of providing IT equipment to teachers and students and the exchange of ideas has been quasi non-existent until 2010/2011. Quite some (non-admin) teachers try to avoid using IT technology in class as a learning medium completely. Several reasons for this avoidance do exist. We discovered that no-one has ever taken a closer look at this social part of IT management in schools, so far. On our quest journey for a technical IT solution for schools, we discussed this issue with several teachers, headmasters, politicians, other IT professionals and they all confirmed: a holistic approach of considering IT management at schools, an approach that includes the people in place, will be new and probably a gain for all. What do you see as the advantages of Skolelinux/Debian Edu? There is a list of advantages: international context, openness to any kind of contributions, do-ocracy policy, the closeness to Debian, the different installation scenarios possible (from stand-alone workstation to complex multi-server sites), the transparency within project communication, honest communication within the group of developers, etc. What do you see as the disadvantages of Skolelinux/Debian Edu? Every coin has two sides: Technically: BTS issue #311188, tricky upgradability of a Debian Edu main server, network client installations on top of a plain vanilla Debian installation should become possible sometime in the near future, one could think about splitting the very complex package debian-edu-config into several portions (to make it easier for new developers to contribute). Another issue I see is that we (as Debian Edu developers) should find out more about the network of people who do the marketing for Debian Edu / Skolelinux. There is a very active group in Germany promoting Skolelinux on the bigger Linux Days within Germany. Are there other groups like that in other countries? How can we bring these marketing people together (marketing group A with group B and all of them with the group of Debian Edu developers)? During the last meeting of the German Skolelinux group, I got the impression of people there being rather disconnected from the development department of Debian Edu / Skolelinux. Which free software do you use daily? For my daily business, I do not use commercial software at all. For normal stuff I use Iceweasel/Firefox, Libreoffice.org. For serious text writing I prefer LaTeX. I use gimp, inkscape, scribus for more artistic tasks. I run virtual machines in KVM and Virtualbox. I am one of the upstream developers of X2Go. In 2010 I started the development of a Python based X2Go Client, called PyHoca-GUI. PyHoca-GUI has brought forth a Python X2Go Client API that currently is being integrated in Ubuntu's software center. For communications I have my own Kolab server running using Horde as web-based groupware client. For IRC I love to use irssi, for Jabber I have several clients that I use, mostly pidgin, though. I am also the Debian maintainer of Coccinella, a Jabber-based interactive whiteboard. My favourite terminal emulator is KDE's Yakuake. Which strategy do you believe is the right one to use to get schools to use free software? Communicate, communicate, communicate. Enrol people, enrol people, enrol people.

1 June 2012

C.J. Adams-Collier: Vacation Tasks

So, I m taking vacation this week and next. Here s a (partial) list of tasks that are left to do: That s probably enough, if not too much. Here s some of the bits I ve done this week: And probably some other stuff ;-)

Alexander Reichle-Schmehl: Release Critical Bug report for Week 22

The bug webinterface of the Ultimate Debian Database currently knows about the following release critical bugs:
In Total:1412
Affecting Wheezy:977
Wheezy only:272
Remaining to be fixed in Wheezy:705
Of these 705 bugs, the following tags are set:
Pending in Wheezy:51
Patched in Wheezy:118
Duplicates in Wheezy:57
Can be fixed in a security Update:24
Contrib or non-free in Wheezy:9
Claimed in Wheezy:2
Delayed in Wheezy:4
Otherwise fixed in Wheezy:99
Ignoring all the above (multiple tags possible) 427 bugs need to be fixed by Debian Contributors to get Debian 7.0 Wheezy released. However, with the view of the Release Managers, 867 need to be dealt with for the release to happen. Please see Interpreting the release critical bug statistics for an explanation of the different numbers.

Pietro Abate: apt-get with external solvers : call for testers

Last year we invited David to work with us for a few days to add a generic interface to apt to call external solvers. After a few iterations, this patch finally landed in master and recently (about 3 months ago), in debian unstable.
  [ David Kalnischkies ]
  * [ABI-Break] Implement EDSP in libapt-pkg so that all front-ends which
    use the internal resolver can now be used also with external
    ones as the usage is hidden in between the old API
  * provide two edsp solvers in apt-utils:
    - 'dump' to quickly output a complete scenario and
    - 'apt' to use the internal as an external resolver
Today the new version of apt-cudf was upload in unstable and with it the latest bug fixes that makes it ready for daily use. I've used it quite a lot myself to upgrade my machine and it seems working pretty well so far... The most important difference with the old version is the support for multi-arch enabled machines. This marks an important milestone in our efforts to integrate external solvers, built using different technologies, directly into apt. From a user prospective, this means that (s)he will have the possibility to check if there exists a better (best ?) solution to an installation problem then what proposed by the internal apt solver. Moreover, even if apt-get gives very satisfactory answers, there are occasions where it fails miserably, leaving the user wondering how to unravel the complex web of dependencies to accomplish his task. Available cudf solvers in debian are at the moment : aspcud, mccs and packup. From an architectural point of view this is accomplished by abstracting the installation problem via a simple textual protocol (EDSP) and using an external tool to do the heavy duty translation. Since all solvers now available in debian are not meant to be debian-specific, using them involve a two step translation. The EDSP protocol specification is for the moment "hidden" in the apt documentation. I hope to find a better place for it soon : it would be cool if other package managers as smart or cupt could add an implementation on EDSP in their code so to automatically benefit of this technology. To make it happen, Apt first creates an EDSP file that is then handled to apt-cudf that takes care of the translation to cudf and back into EDSP that is then read back by apt. Apt-cudf is the bridge between edsp and the external solvers and takes care of doing the book keeping and to select the right optimization criteria. Roberto recently wrote a very nice article explaining how to use apt-get with an external solver. In a nutshell, if you want to try this out you just need to install apt-cudf, one external solver like aspcud from the university of Pasdam and then call apt-get using the --solver option (that is not yet documented #67442 ). For example :
apt-get install -s gnome --solver aspcud
This will install gnome while using the a optimization criteria that tries to minimizing the changes on the system. Various other optimization criteria for all apt-get default actions can be specified in the apt-cudf configuration file /etc/apt-cudf.conf I hope the new release of apt-cudf make it into testing before the freeze. Time to test !!!

31 May 2012

Russell Coker: Links May 2012

Vijay Kumar gave an interesting TED talk about autonomous UAVs [1]. His research is based on helicopters with 4 sets of blades and his group has developed software to allow them to develop maps, fly in formation, and more. Hadiyah wrote an interesting post about networking at TED 2012 [2]. It seems that giving every delegate the opportunity to have their bio posted is a good conference feature that others could copy. Bruce Schneier wrote a good summary of the harm that post-911 airport security has caused [3]. Chris Neugebauer wrote an insightful post about the drinking culture in conferences, how it excludes people and distracts everyone from the educational purpose of the conference [4]. Matthew Wright wrote an informative article for Beyond Zero Emissions comparing current options for renewable power with the unproven plans for new nuclear and fossil fuel power plants [5]. The Free Universal Construction Kit is a set of design files to allow 3D printing of connectors between different types of construction kits (Lego, Fischer Technic, etc) [6]. Jay Bradner gave an interesting TED talk about the use of Open Source principles in cancer research [7]. He described his research into drugs which block cancer by converting certain types of cancer cell into normal cells and how he shared that research to allow the drugs to be developed for clinical use as fast as possible. Christopher Priest wrote an epic blog post roasting everyone currently associated with the Arthur C. Clarke awards, he took particular care to flame Charles Stross who celebrated The Prestige of such a great flaming by releasing a t-shirt [8]. For a while I ve been hoping that an author like Charles Stross would manage to make more money from t-shirt sales than from book sales, Charles is already publishing some of his work for free on the Internet and it would be good if he could publish it all for free. Erich Schubert wrote an interesting post about the utility and evolution of Favebook likes [9]. Richard Hartmann wrote an interesting summary of the problems with Google products that annoy him the most [10]. Sam Varghese wrote an insightful article about the political situation in China [11]. The part about the downside of allowing poorly educated people to vote seems to apply to the US as well. Sociological Images has an article about the increased rate of Autism diagnosis as social contagion [12]. People who get their children diagnosed encourage others with similar children to do the same. Vivek wrote a great little post about setting up WPA on Debian [13]. It was much easier than expected once I followed that post. Of course I probably could have read the documentation for ifupdown, but who reads docs when Google is available? Related posts:
  1. Links March 2012 Washington s Blog has an informative summary of recent articles about...
  2. Links April 2012 Karen Tse gave an interesting TED talk about how to...
  3. Links February 2012 Sociological Images has an interesting article about the attempts to...

Richard Hartmann: Potpourri II

Traffic jams suck; but they do give you a chance to think and blog. Truckers hams are telling me that four trucks are totalled and need to be removed by the THW. As the trucker next to me just broke out his gas grill and several people are joining him in a spontaneous BBQ, I don't expect to be moving any time soon. Kicking things along This is coming along nicely RAID 4 Unless you are forced to support legacy systems or playing RAIDemon and gotta try them all, there is no valid use case for RAID 4. So doing so is a patently bad idea. RAID 5 is always the better option as you are not relying on a single disk which contains the checksums, but use striping to distribute them. RAID 4 has a chance of 1/n that a failed disk will force all remaining disks to read all respective sectors and then the controller/CPU to XOR everything. There is a reason why RAID 3 was replaced with RAID 4; just as there's a reason why RAID 4 was replaced by RAID 5. Don't use RAID 4. Seriously. Tom Tom vs OSM In case you have been following the news in the GPS circles, you will have noticed that TomTom pretty gave up on innovating and resigned themselves to dying a slow death. Well that and to start a nice amount of FUD. While they have been rebuked pretty thoroughly already, I wanted to add my own POV. There is a kernel of truth in what TomTom said inasmuch they state that pedestrian paths are better than car paths. Of course, while they got the relative quality right, they are wrong about the absolute quality of the data. They claim pedestrian is OK and motorized is bad. In my experience, maps for motorized traffic are good while pedestrian tracks are mind-boggingly great. These days, most people (and probably all who read this) will be at least somewhat familiar with the quality of OSM in the random cities they visit over the year. The common verdict is usually "very nice". But that's not where OSM shines. It's in the lesser-known places where you don't expect too many tech-minded people to come along where it starts to really impress me. While I found a few stubs on a dead-end street in Akureyri which were not mapped, every foot trail I ever walked along in Iceland has been mapped. ski-doo tracks on Svalbard? Check. Backyard in Novosibirsk? Check, as well. And we are talking "packed earth track in between garages opening up into a 'playground' where a man and a pregnant women are passing a bottle back and forth while smoking and watching over their two kids who play next to a middle-aged man who is lying on the ground wasted, using a rock as a pillow all the while an extremely blingy Mercedes with two muscle-packed thugs is cruising by, all in the time it took us to walk through". Mapped. Horse tracks and hiking trails in Mongolia? Yep. Hutongs in Beijing including alleys so narrow you have to walk single file? Of course. Richard Fairhurst is right that the plural of anecdote is not data, but OSM is good. TomTom is afraid for a reason. As an aside, I find the picture used in TomTom's newsletter extremely funny; I am sure they didn't intend this, but hey. Windows 100-300 MiB for any random driver installation package is par for the course. Be it GFX, audio, or network card. If you plug in a different USB mouse, Windows 7 will take several seconds while "setting up" a standard USB HID device. And hitting ctrl-alt-del in the login screen or while logged in does not give you a quick way to shut down or reboot the machine. As I am breaking out the Windows installation every month to install updates and every six months to actually use this does not affect me a lot. Still, it's amazing that so many people are putting up with this. UEFI It's nice to see that Matthew Garrett is still going strong. I am not sure if this is the best way, but it seems to be a workable solution while in between a rock and a hard place. Not sure about the requirements of GPLv3 with regards to the keys Fedora will use to sign GRUB 2 with. From what I remember without rereading the license, as long as Fedora is not selling a system on which you can not disable secure boot they should be in the clear. IANAL, IIRC, etc. Wrapping up And I have not moved a single meter, other than making way to a police motorbike, in over an hour. Pity I don't have anything to BBQ with me; smells delicious over there.

Matthew Garrett: Implementing UEFI Secure Boot in Fedora

Edit 16:17EDT 31/5/12: Clarification of who gets the $99
Edit 02:10EDT 01/6/12: Clarification that it's a one-off payment

(Brief disclaimer - while I work for Red Hat, I'm only going to be talking about Fedora here. Anything written below represents only my opinions and my work on Fedora, not Red Hat's opinions or future plans)

Fedora 17 was released this week. It's both useful and free, and serves as a welcome addition to any family gathering. Do give it a go. But it's also noteworthy for another reason - it's the last Fedora release in the pre-UEFI secure boot era. Fedora 18 will be released at around the same time as Windows 8, and as previously discussed all Windows 8 hardware will be shipping with secure boot enabled by default. While Microsoft have modified their original position and all x86 Windows machines will be required to have a firmware option to disable this or to permit users to enrol their own keys, it's not really an option to force all our users to play with hard to find firmware settings before they can run Fedora. We've been working on a plan for dealing with this. It's not ideal, but of all the approaches we've examined we feel that this one offers the best balance between letting users install Fedora while still permitting user freedom.

Getting the machine booted
Most hardware you'll be able to buy towards the end of the year will be Windows 8 certified. That means that it'll be carrying a set of secure boot keys, and if it comes with Windows 8 pre-installed then secure boot will be enabled by default. This set of keys isn't absolutely fixed and will probably vary between manufacturers, but anything with a Windows logo will carry the Microsoft key[1].

We explored the possibility of producing a Fedora key and encouraging hardware vendors to incorporate it, but turned it down for a couple of reasons. First, while we had a surprisingly positive response from the vendors, there was no realistic chance that we could get all of them to carry it. That would mean going back to the bad old days of scouring compatibility lists before buying hardware, and that's fundamentally user-hostile. Secondly, it would put Fedora in a privileged position. As one of the larger distributions, we have more opportunity to talk to hardware manufacturers than most distributions do. Systems with a Fedora key would boot Fedora fine, but would they boot Mandriva? Arch? Mint? Mepis? Adopting a distribution-specific key and encouraging hardware companies to adopt it would have been hostile to other distributions. We want to compete on merit, not because we have better links to OEMs.

An alternative was producing some sort of overall Linux key. It turns out that this is also difficult, since it would mean finding an entity who was willing to take responsibility for managing signing or key distribution. That means having the ability to keep the root key absolutely secure and perform adequate validation of people asking for signing. That's expensive. Like millions of dollars expensive. It would also take a lot of time to set up, and that's not really time we had. And, finally, nobody was jumping at the opportunity to volunteer. So no generic Linux key.

The last option wasn't hugely attractive, but is probably the least worst. Microsoft will be offering signing services through their sysdev portal. It's not entirely free (there's a one-off $99 fee to gain access edit: The $99 goes to Verisign, not Microsoft - further edit: once paid you can sign as many binaries as you want), but it's cheaper than any realistic alternative would have been. It ensures compatibility with as wide a range of hardware as possible and it avoids Fedora having any special privileges over other Linux distributions. If there are better options then we haven't found them. So, in all probability, this is the approach we'll take. Our first stage bootloader will be signed with a Microsoft key.

Bootloaders
We've decided to take a multi-layer approach to our signing for a fairly simple reason. Signing through the Microsoft signing service is a manual process, and that's a pain. We don't want to have bootloader updates delayed because someone needs to find a copy of Internet Explorer and a smartcard and build packages by hand. Instead we're writing a very simple bootloader[2]. This will do nothing other than load a real bootloader (grub 2), validate that it's signed with a Fedora signing key and then execute it. Using the Fedora signing key there means that we can build grub updates in our existing build infrastructure and sign them ourselves. The first stage bootloader should change very rarely, and we don't envisage updating it more than once per release cycle. It shouldn't be much of a burden on release management.

What about grub? We've already switched Fedora 18 over to using grub 2 by default on EFI systems, but it still needs some work before it's ready for secure boot. The first thing is that we'll be disabling the module loading. Right now you can load arbitrary code into grub 2 at runtime, and that defeats the point of secure boot. So that'll be disabled. Next we'll be adding support for verifying that the kernel it's about to boot is signed with a trusted key. And finally we'll be sanitising the kernel command line to avoid certain bits of functionality that would permit an attacker to cause even a signed kernel to launch arbitrary code. These restrictions will all vanish if secure boot is disabled.

Kernel
Secure boot is built on the idea that all code that can touch the hardware directly is trusted, and any untrusted code must go through the trusted code. This can be circumvented if users can execute arbitrary code in the kernel. So, we'll be moving to requiring signed kernel modules and locking down certain aspects of kernel functionality. The most obvious example is that it won't be possible to access PCI regions directly from userspace, which means all graphics cards will need kernel drivers. Userspace modesetting will be a thing of the past. Again, disabling secure boot will disable these restrictions.

Signed modules are obviously troubling from a user perspective. We'll be signing all the drivers that we ship, but what about out of tree drivers? We don't have a good answer for that yet. As before, we don't want any kind of solution that works for us but doesn't work for other distributions. Fedora-only or Ubuntu-only drivers are the last thing anyone wants, and this really needs to be handled in a cross-distribution way.

Wait signed what
Secure boot is designed to protect against malware code running before the operating system. This isn't a hypothetical threat. Pre-boot malware exists in the wild, and some of it is nastier than you expect. So obviously bootloaders need to be signed, since otherwise you'd just replace the signed bootloader with an unsigned one that installed malware and booted your OS.

But what about the kernel? The kernel is just code. If I take a signed Linux bootloader and then use it to boot something that looks like an unsigned Linux kernel, I've instead potentially just booted a piece of malware. And if that malware can attack Windows then the signed Linux bootloader is no longer just a signed Linux bootloader, it's a signed Windows malware launcher and that's the kind of thing that results in that bootloader being added to the list of blacklisted binaries and suddenly your signed Linux bootloader isn't even a signed Linux bootloader. So kernels need to be signed.

And modules? Again, modules are just code. It's a little trickier, but if your signed kernel loads an unsigned module then that unsigned module can set up a fake UEFI environment and chain into a compromised OS bootloader. Now the attacker just has to include a signed kernel and a minimal initramfs that loads their malware module. It'd slow down boot by a couple of seconds, but other than that it'd be undetectable. X? If you can access the registers on a GPU then you can get the GPU to DMA over the kernel and execute arbitrary code. Trickier again, but still achievable - and if you've locked down every other avenue of attack, it's even attractive.

If we produce signed code that can be used to attack other operating systems then those other operating systems are justified in blacklisting us. That doesn't seem like a good outcome.

Customisation
A lot of our users want to build their own kernels. Some even want to build their own distributions. Signing our bootloader and kernel is an impediment to that. We'll be providing all the tools we use for signing our binaries, but for obvious reasons we can't hand out our keys. There's three approaches here. The first is for a user to generate their own key and enrol it in their system firmware. We'll trust anything that's signed with a key that's present in the firmware. The second is to rebuild the shim loader with their own key installed and then pay $99 and sign that with Microsoft. That means that they'll be able to give copies to anyone else and let them install it without any fiddling. The third is to just disable secure boot entirely, at which point the machine should return to granting the same set of freedoms as it currently does.

But I don't trust Microsoft
A system in custom mode should allow you to delete all existing keys and replace them with your own. After that it's just a matter of re-signing the Fedora bootloader (like I said, we'll be providing tools and documentation for that) and you'll have a computer that will boot Fedora but which will refuse to boot any Microsoft code. It may be a little more awkward for desktops because you may have to handle the Microsoft-signed UEFI drivers on your graphics and network cards, but this is also solvable. I'm looking at ways to implement a tool to allow you to automatically whitelist the installed drivers. Barring firmware backdoors, it's possible to configure secure boot such that your computer will only run software you trust. Freedom means being allowed to run the software you want to run, but it also means being able to choose the software you don't want to run.

You've sold out
We've been working on this for months. This isn't an attractive solution, but it is a workable one. We came to the conclusion that every other approach was unworkable. The cause of free software isn't furthered by making it difficult or impossible for unskilled users to run Linux, and while this approach does have its downsides it does also avoid us ending up where we were in the 90s. Users will retain the freedom to run modified software and we wouldn't have accepted any solution that made that impossible.

But is this a compromise? Of course. There's already inequalities between Fedora and users - trademarks prevent the distribution of the Fedora artwork with modified distributions, and much of the Fedora infrastructure is licensed such that some people have more power than others. This adds to that inequality. It's not the ideal outcome for anyone, and I'm genuinely sorry that we weren't able to come up with a solution that was better. This isn't as bad as I feared it would be, but nor is it as good as I hoped it would be.

What about ARM
Microsoft's certification requirements for ARM machines forbid vendors from offering the ability to disable secure boot or enrol user keys. While we could support secure boot in the same way as we plan to on x86, it would prevent users from running modified software unless they paid money for a signing key. We don't find that acceptable and so have no plans to support it.

Thankfully this shouldn't be anywhere near as much of a problem as it would be in the x86 world. Microsoft have far less influence over the ARM market, and the only machines affected by this will be the ones explicitly designed to support Windows. If you want to run Linux on ARM then there'll be no shortage of hardware available to you.

Is this all set in stone?
No. We've spent some time thinking about all of this and are happy that we can implement it in the Fedora 18 timescale, but there's always the possibility that we've missed something or that a new idea will come up. If we can increase user freedom without making awful compromises somewhere else then we'll do it.

[1] In fact, chances are that everything will carry the Microsoft key. Secure boot requires that UEFI drivers also be signed. The signing format only permits a single signature per binary. For compatibility, approximately all add-on hardware shipped will be signed with Microsoft's key, and that means that all system vendors have to recognise Microsoft's key in order to permit that hardware to run on their systems.
[2] Current source is here. It relies on a port of the UEFI crypto library and OpenSSL that I have building with some handholding, and which I'll upload as soon as possible.

comment count unavailable comments

Paul Tagliamonte: my sbuilder setup (sb)

I ve expanded a bit on my idea (that is, the sbuilder wrapper to help aid with managing more then one sbuild chroot), and it got out of hand very quickly. It s gone from one small shell script to a set of 8, and growing. Quickly. Until I spin these off into their own repo, they ll be in my dot-bin Before getting too far, please relize this is only how I m using sbuild. There are many other (more) sane configurations that you can (and should) use. This was not made to be useful for anyone other then me :) I ve added the ability to use unionfs-fuse, set up a way to login interactively and a script to create new chroots without having them register as a global schroot. Firstly, forgive me for my sins. This is all a huge hack, but it s use isn t the worst thing in the world. As of the time of this writing, I now have a gcc and clang chroot for each of stable, testing and unstable. Now the big question how do you use this. Rather, this is more of a how do I use sbuilder :) Firstly, I ve created a file for each schroot set that I m using. Here s my /etc/schroot/chroot.d/sid.
[sid]
type=directory
description=Debian sid/amd64 autobuilder
directory=/var/sbuild/sid
groups=root,sbuild
root-groups=root,sbuild
script-config=sbuild/config
It s very similar to the one that sbuilder-createchroot creates. If you re just looking for aufs / unionfs support, keep in mind schroot has some great support. check man 5 schroot.conf. Next, create a dist.d directory in /var/sbuild such as /var/sbuild/sid.d. In the future, this might get automagically created by the sb-create script. For now, it just expects that exists. The arguments for all the sb* commands is always in the order of variant, dist. variant defaults to generic , which I ve set as an alias to the generic chroot, and the dist defaults to sid . At some point, I ll change the default to be the last UNRELEASED entry in the changelog. So, to create a new chroot to become your clang chroot, you may choose to invoke it as sb-create clang, or more explicitly sb-create clang sid. To create a new chroot to become your stable / gcc chroot, you may create it with sb-create gcc stable. After you have a chroot, you can start to use the sb or sbo scripts. sbo is just a wrapper around the sb script that sets an envvar to tell the sb-prehook and sb-posthook to set up the unionfs overlay. The unionfs overlay routine replaces the dist symbolic link (such as /var/sbuild/testing) with a folder, and mounts the real chroot (such as /var/sbuild/testing.d/testing-gcc) under the unionfs divert point (which is something like /var/sbuild/testing.overlay.d/testing-gcc). This lets us make sure the chroot is never affected by a routine build, even if it s designed to trash a chroot (unless, of course, it uses any number of ways to get outside the chroot and trash the pristine directory this is to prevent mistakes, not malicious scripts). By the way, for those interested, there s a small issue with what I m describing it s very subtle. Basically, there s a few assumptions made by how schroot operates, and handles sessions . Since it ll keep stuff mounted if the session is still active, it makes purging all this a real pain. The magic behind the scripts is in the sb-endsessions routine which will properly end all the schroot sessions for that dist, and since we have our own high-level locking system, we shouldn t have a situation where this kills off another build. The code (which might have issues) looks like:
SESSIONS= schroot --all-sessions -l   grep "^session:$DIST-" 
for session in $SESSIONS; do
    echo "W: Ending session $session"
    schroot -e -c $session
done
By backing of schroot s mounts, we can do our mounting / removing without worrying about klobbering schroot. Now, there s also the case of wanting to interactively log into a chroot. For that, there s sb-login and sb-maint, both take the usual variant / dist arguments. sb-login logs in as the current user, and sb-maint logs in as the root user. Eventually, I ll be adding some scripts to upgrade all the build chroots for a dist at once.

30 May 2012

Thomas Goirand: New release of MLMMJ (version 1.2.18.0) uploaded in Debian

MLMMJ stands for Mailing List Manager Made Joyful. To me, it s the best mailing list manager available. Not only its written in C (no slow interpreted language here ), but also it is easy and very convenient to setup. No hack is needed to make it run on a multi-domain environment, and no need to run a special subdomain for your lists (yes, I m thinking about you, silly Mailman ). If you didn t know, MLMMJ is used to handle relatively high traffic lists: all mailing lists for Gentoo are using MLMMJ for example. MLMMJ 1.2.18.0 has just been release upstream, few days ago (on the 29 of may). It s with a great pleasure that I packaged (and uploaded to SID) the latest version, and wrote the changelog, where I wrote that 6 Debian specific patches (out of 8) could be removed! One of the funny changes are the renaming of the mlmmj-recieve into mlmmj-receive as it always should have been (with a symlink to keep backward compatibility), and the removal of the .sh extension to the mlmmj-make-ml (which was patched in Debian to be kept policy compliant: that s one of the removed patches ). Even if I did a few quick functional tests before uploading, I d be happy to get some feedback before Wheezy is out, so please test and eventually bug-report!

Lucas Nussbaum: Debian archive rebuilds on Amazon Web Services

I like to think that archive rebuilds play an important role in Debian Quality Assurance and Release Management efforts. By trying to rebuild every Debian package from source, one can identify packages that do not build anymore due to changes in other packages (compilers, interpreters, libraries, ). It is also a good way to stress-test all packages that are involved in building other packages. Since 2007, I had been running Debian archive rebuilds on the Grid 5000 testbed, a research infrastructure for performing experiments on distributed systems HPC/Grid/Cloud/P2P. I filed more than 6000 release-critical bugs in the process. Late last year, Amazon kindly offered us a grant to allow us to run such QA tests on Amazon Web Services. With S bastien Badia, we ported the rebuild infrastructure to AWS (scripts), and several rebuilds have already been carried out on AWS. On the technical level, 50 to 100 EC2 spot instances are started, and then controlled from a master instance using SSH. On build instances, a classic sbuild setup is used. Logs are retrieved to the master node after rebuilds, and build instances are simply shut down when there are no more tasks to process. Several tasks are processed simultaneously on each instance, and when they fail, they are retried again with no other concurrent build on the same instance, to eliminate random failures caused by load or timing issues. All the scripts are designed to support other kind of QA tests, not just rebuilds. Moving to Amazon Web Services will facilitate sharing the human workload of doing those tests. It is now possible for developers interested in custom tests to do them themselves (hint hint).

29 May 2012

Jon Dowland: Debian Day

After a relatively long period of flux, I've managed to settle into a routine where I've got all the time I can dedicate to open source stuff bunched together: every Tuesday night. I've started calling it "Debian Day", although Debian won't get all that time exclusively. One evening a week doesn't sound like much, but that's unfortunately a realistic amount given my other commitments. Still, having a firm idea of how much resource I have makes planning what I can and can't do much easier. It also means I need to be quite selective about what I spend my time on. This also impacts the type of interactions I can use. I can respond to email, bugs, etc., but I cannot effectively interact with folks on IRC, for example: the period between windows is too large for it to work effectively. Therefore, I can't work reliably with any team or area of Debian that relies on IRC to get things done. Tonight, then. I have recently been ripping some CDs, a task for which I use grip, despite it long since having been dropped by Debian. It still works very well, whereas Sound Juicer, which still pops up whenever I insert an audio CD, does not. I've been interested in helping out with Debian GNOME packaging for a while, and it seems the issue has been fixed upstream, but after asking around in #debian-gnome, the root cause is that the newer musicbrainz API packages have not been uploaded to Debian. (the existing package uses the old API). Two separate people appear to have been working on a new Music Brainz package, but neither filed an ITP bug. I've spent the largest chunk of my Debian time just trying to figure out what has been going on, sadly, but at least there's the likelyhood this will be fixed for wheezy. There's also the tail end of a musicbrainz-2.1 transition finishing up. Another thing I considered this evening was the future of debgtd in Debian stable. On reflection, and after sitting on the fence for several months, I've decided I don't have the time to fix it up enough to have it ready for wheezy, so with regret I've filed a removal bug. With any luck, at some point in the wheezy cycle I'll manage to put more time towards this, because I think it addresses a genuine need and could be useful to people. Finally, whenever I sit down and look at Debian stuff to finish up, I have a strong temptation to fix some issues belonging to the games team. In particular, Gustavo Panizzo has put a lot of work into vavoom packages, but sadly his requests for sponsorship are falling on deaf ears. There's also Johey Shmit's work on packaging zdoom which needs some attention to finish removing some embedded libraries. I have to keep reminding myself I left the team for a reason, to focus on something else, and finding that something else should be the priority. So all in all not the most uplifting evening: I'm starting to think I simply couldn't work effectively with the Debian GNOME team without being able to dedicate more time to use IRC effectively; I spent more time that could be necessary figuring out a bunch of other folks have potentially wasted each other's time by working on packages and not filing ITPs; and I've removed one of my own packages from the distribution altogether. It's not all doom and gloom, though: last week I packaged Simon Howard's lhasa, a library and tool for decompressing LHA files, as used by various old DOS games. (See? I just can't escape that stuff ) I also managed to 'resolve' two bup bugs, one RC. I also finished transitioning from my old PGP key to my new one.

Steve Kemp: A heady mixture of photography and programming

The past few weeks have consisted of a heady mixture of taking interesting pictures of cute people, and writing code. I spent a while getting to grips with seccomp filters, using the facilities present in recent GNU/Linux Kernels to filter system calls binaries are allowed to make. My initial test was to patch GNU Less to only allow it to open, read, and close files. The side-effect of this was that the built in shell-escape was closed, thus allowing me to test it. After that I toyed around with interfacing with spidermonkey seamonkey. In this regard I was less succesful, but I did manage to write code in C that would invoke javascript functions loaded dynamically. Similarly I could call from my (loaded) javascript code into functions defined in C. I don't have a use for a Javascript to/from C bridge, but I'm sure that time will come. Photography has been a constant distraction, I took some fun shots of the Edinburgh Marathon, and then distracted myself with a volunteer to take an abstract fishnet photograph. We went on to do some more fun shots begin careful to stay on the safe side of the NSFW limit. I think this is borderline NSFW, but we were both clear exactly what we wanted and we got it perfectly there. Next week will be quieter, but providing we don't have another mini-heatwave in Edinburgh I'll be cheerful regardless. ObQuote: One, two, Freddy's coming for you. - A Nightmare on Elm Street (original)

28 May 2012

Vincent Bernat: Integration of NetSNMP into an event loop

NetSNMP comes with its own event loop based on the select() system call. While you can build your program around it, you may prefer to use an existing event loop, either a custom one or something like libevent, libev or Twisted.

Own custom event loop Let s start with the easiest case: you have written your own event loop. This means you make a call to select(), poll(), epoll(), kqueue() or something similar. All those functions take a set of file descriptors and wait for any of them to become available in a specified time frame. Here is a typical use of select() (adapted from Quagga)
readfd = m->readfd;
writefd = m->writefd;
exceptfd = m->exceptfd;
timer_wait = thread_timer_wait (&m->timer);
num = select (FD_SETSIZE,
              &readfd, &writefd, &exceptfd,
              timer_wait);
if (num < 0)
   
    if (errno == EINTR)
      continue; /* signal received - process it */
    zlog_warn ("select() error: %s",
               safe_strerror (errno));
    return NULL;
   
if (num == 0)
   
    /* Timeout handling */
    thread_timer_process (&m->timer);
   
if (num > 0)
   
    thread_process_fd (&readfd);
    thread_process_fd (&writefd);
   
thread_process_fd (fds) function iterates on each file descriptor fd and executes the appropriate action if FD_ISSET (fds, fd) is true. Integrating NetSNMP into such an event loop is easy: NetSNMP provides the snmp_select_info() function which alters a set of file descriptors to insert its own. Here is the new code:
#if defined HAVE_SNMP
struct timeval snmp_timer_wait;
int snmpblock = 0;
int fdsetsize;
#endif
/* ... */
#if defined HAVE_SNMP
fdsetsize = FD_SETSIZE;
snmpblock = 1;
if (timer_wait)
   
    snmpblock = 0;
    memcpy(&snmp_timer_wait, timer_wait,
           sizeof(struct timeval));
   
snmp_select_info(&fdsetsize, &readfd,
                 &snmp_timer_wait, &snmpblock);
if (snmpblock == 0)
  timer_wait = &snmp_timer_wait;
#endif
num = select (FD_SETSIZE,
              &readfd, &writefd, &exceptfd,
              timer_wait);
if (num < 0)   /* ... */  
#if defined HAVE_SNMP
if (num > 0)
  snmp_read(&readfd);
else if (num == 0)
   
    snmp_timeout();
    run_alarms();
   
netsnmp_check_outstanding_agent_requests();
#endif
/* ... */
snmp_select_info() may modify the provided set of file descriptors (and therefore, its size). It may also modify the provided timer in case it needs to schedule an action before the original timeout. snmpblock is a tricky variable. While select() can be called with a timeout set to NULL, this is not the case of snmp_select_info(): you have to pass a valid pointer. snmpblock is set to 0 if the provided timeout must be considered or to 1 otherwise. snmp_select_info() will set snmpblock to 0 if and only if it alters the timeout. Here are a two examples of integration. Both are for a subagent but the code for a manager is the same:

Third-party event loop With a third party event loop, you don t have access to the select() system call anymore. Therefore, you cannot alter it with snmp_select_info(). Instead, at each iteration of the loop, a list of SNMP related file descriptors is kept up-to-date with the help of snmp_select_info(): new ones are added and old ones are removed.

libevent Let s assume that snmp_fds is the list of current SNMP related events1. A function levent_snmp_update() calls snmp_select_info() to update this list. Here is a partial implementation (error handling removed and some declarations omitted, look at the complete version from lldpd):
static void
levent_snmp_update()
 
    int maxfd = 0;
    int block = 1;
    FD_ZERO(&fdset);
    snmp_select_info(&maxfd, &fdset, &timeout, &block);
    /* We need to untrack any event whose FD is not in  fdset 
       anymore */
    for (snmpfd = TAILQ_FIRST(snmp_fds);
         snmpfd;
         snmpfd = snmpfd_next)  
        snmpfd_next = TAILQ_NEXT(snmpfd, next);
        if (event_get_fd(snmpfd->ev) >= maxfd  
            (!FD_ISSET(event_get_fd(snmpfd->ev), &fdset)))  
            event_free(snmpfd->ev);
            TAILQ_REMOVE(snmp_fds, snmpfd, next);
            free(snmpfd);
          else
            FD_CLR(event_get_fd(snmpfd->ev), &fdset);
     
    /* Invariant: FD in  fdset  are not in list of FD */
    for (int fd = 0; fd < maxfd; fd++)
        if (FD_ISSET(fd, &fdset))  
            snmpfd->ev = event_new(base, fd,
                EV_READ   EV_PERSIST,
                levent_snmp_read,
                NULL);
            event_add(snmpfd->ev, NULL);
            TAILQ_INSERT_TAIL(snmp_fds, snmpfd, next);
         
    /* If needed, handle timeout */
    evtimer_add(snmp_timeout, block?NULL:&timeout);
 
Then, replace the main loop (usually, a call to event_base_dispatch()) with this:
do  
    if (event_base_got_break(base)  
        event_base_got_exit(base))
        break;
    netsnmp_check_outstanding_agent_requests();
    levent_snmp_update();
  while (event_base_loop(base, EVLOOP_ONCE) == 0);
Here is how are defined the two callbacks:
static void
levent_snmp_read(evutil_socket_t fd, short what, void *arg)
 
    FD_ZERO(&fdset);
    FD_SET(fd, &fdset);
    snmp_read(&fdset);
    levent_snmp_update();
 
static void
levent_snmp_timeout(evutil_socket_t fd, short what, void *arg)
 
    snmp_timeout();
    run_alarms();
    levent_snmp_update();
 

Twisted reactor As a second example, here is how to integrate NetSNMP into Twisted reactor. Twisted is an event-driven network programming framework written in Python. It does not come with support for SNMP2. Because of the language mismatch, the integration of NetSNMP in Twisted needs to be done using a C extension or the ctypes module. Twisted event loop is called a reactor. Like for libevent, events need to be registered. It is possible to register file descriptor-like objects using a class implementing a handful of methods. Here is the implementation of such a class (adapted from PyNetSNMP, a subproject of Zenoss using the ctypes module):
class SnmpReader:
    "Respond to input events"
    implements(IReadDescriptor)
    def __init__(self, fd):
        self.fd = fd
    def doRead(self):
        netsnmp.snmp_read(self.fd)
    def fileno(self):
        return self.fd
Like for libevent, we have a function to update the list of SNMP related events (stored as a mapping between file descriptors and SnmpReader instances):
class Timer(object):
    callLater = None
timer = Timer()
fdMap =  
def updateReactor():
    "Add/remove event handlers for SNMP file descriptors and timers"
    fds, t = netsnmp.snmp_select_info()
    for fd in fds:
        if fd not in fdMap:
            reader = SnmpReader(fd)
            fdMap[fd] = reader
            reactor.addReader(reader)
    current = Set(fdMap.keys())
    need = Set(fds)
    doomed = current - need
    for d in doomed:
        reactor.removeReader(fdMap[d])
        del fdMap[d]
    if timer.callLater:
        timer.callLater.cancel()
        timer.callLater = None
    if t is not None:
        timer.callLater = reactor.callLater(t, checkTimeouts)
Contrary to libevent, we cannot alter the main loop to call updateReactor() at each iteration. Therefore, it must be called after each SNMP related function. For another example, have a look at my equivalent implementation as a C extension.

Miscellaneous

Lack of asynchronicity Integrating NetSNMP into an event-based program is not risk-free. I have written a fairly comprehensive article on the lack of asynchronicity in NetSNMP AgentX protocol implementation. I urge you to read it if you want to integrate a SNMP subagent into an existing program. On the manager side, you will get similar drawbacks when using SNMPv3. To retrieve or manipulate management information using SNMPv3, it is necessary to know the identifier of the remote SNMP protocol engine. This identifier can be configured directly on each manager but it is usually discovered by querying SNMP-FRAMEWORK-MIB:: snmpEngineID, as described in RFC 5343. Unfortunately, this discovery is done synchronously by NetSNMP. There is a bug report for this issue but it seems difficult to fix. I have not been able to come up with a proper patch but I have described a workaround around snmp_sess_async_send(). There is no such problem with SNMPv2.

Limitation of file descriptor sets snmp_select_info() and snmp_read() uses the fd_set type to handle a set of file descriptors. This type should be manipulated with FD_CLR(), FD_ISSET(), FD_SET() and FD_ZERO(). Those may be defined as functions but they usually are macros. Moreover, no file descriptor greater than FD_SETSIZE (which is usually set to 1024) can be handled with the fd_set type. This means that if you have a file descriptor greater than 1024, you won t be able to use snmp_select_info() with it. Starting from NetSNMP 5.5, you can use snmp_select_info2() and snmp_read2() instead of snmp_select_info() and snmp_read(). They use the netsnmp_large_fd_set. If you want to keep compatibility with NetSNMP 5.4, here is another twist. The fd_set type is usually a fixed-size array of long integers. FD_CLR(), FD_ISSET() and FD_SET() are size-independant. The size only matters for FD_ZERO() which is not used by either snmp_select_info() or snmp_read(). You can therefore allocate a larger fd_set. A common way is to compile your program with -D__FD_SETSIZE=4096. You can still use FD_ZERO() yourself. However, this is not portable (GNU C Library only). Another way is to allocate your own array of long integers and cast it to fd_set. You ll have to redefine FD_ZERO():
typedef struct  
   long int fds_bits[4096/sizeof(long int)];
  my_fdset;
#undef FD_ZERO
#define FD_ZERO(fdset) memset((fdset), 0, sizeof(my_fdset))
 
   /* ... */
   my_fdset fdset;
   FD_ZERO(&fdset);
   /* ... */
   rc = snmp_select_info(&n, (fd_set *)&fdset, &timeout, &block);
   /* ... */
 

Threads NetSNMP provides two API:
  • the traditional session API which is not thread-safe and
  • the single session API.
In the above examples, I have used the traditional session API. If you are using threads, you need to ensure that all SNMP operations are done in a single thread. Otherwise, you need to adapt the examples to use the single session API. snmp_select_info() is part of the traditional session API. You should replace it with snmp_sess_select_info() and keep a list of SNMP sessions. This API is not available for the agent side of NetSNMP.

  1. The event list is global but some code could be added to bind a different list for each base in case you use different bases. This is not needed in the case of lldpd which uses only one event base.
  2. TwistedSNMP is an SNMP protocol implementation for Twisted based on PySNMP but is not maintained anymore. PySNMP comes with examples on how to integrate with Twisted.

Thibaut Girka: Multiarch Cross-toolchains

Hi! I have been selected to work on multiarch cross-toolchains as part of the Google summer of Code. Although coding officially started last week, I have been busy with exams and other uni stuff at the time. Anyway, let's talk about the project itself, that is, multiarch cross-toolchains. What is multiarch? In order to understand what the project is about, one need to know what multiarch is. I'll quickly describe it, and I invite anyone interested to read the spec (really, it's interesting, and not that long). Basically, multiarch lets you install binary packages for multiple architectures on one system, and let package managers resolve dependencies as one would expect. This works by identifying four sets of packages: Anyway, multiarch is an amazing thing that may be of use in various cases, such as running i386-only software on an amd64 system, running armel software on an armhf system, running i386 software on an armhf system using qemu-user-static and binfmt, cross-grading (switching, for instance, from i386 to amd64 on a running system), and, in our case, easing cross-compilation. State of cross-toolchains in Debian There aren't really cross-toolchains available in Debian (although some can be built from source packages), but they are available in emdebian's repositories. However, those packages predate multiarch, and don't make use of it. Instead, they depend on special packages converted using dpkg-cross. Such packages are merely arch: all versions of foreign packages, with files moved to arch-qualified paths to prevent conflicts with normal packages. Such packages thus have to be generated from real packages, either manually or using some sort of automation. In addition, this additional step brings several issues, like keeping in sync converted packages with real packages from the archive. With multiarch, such packages are rendered useless, and may in fact bring additional problems, since converted and multiarch packages may share the same files, and thus may not be co-installable. Switching from dpkg-cross to cross-arch deps Cross-toolchains packages currently depend on a few packages converted using dpkg-cross (ex: libc6-$ARCH-cross and libc6-dev-$ARCH-cross for some packages). They need to depend on the real package with the right arch (ex: libc6:$ARCH, libc6-dev:$ARCH). The obvious way to do so is by annotating the dependency with a specific arch qualifier (ex: Depends: libc6:armhf ). This is, however, not covered by the multiarch spec. Note that some lib*-$ARCH-cross packages will remain (for instance, libgcc1-armhf-cross). Indeed, those packages are different from the packages mentioned above in that they are of the architecture of the build system and actually part of the toolchain itself.
The above isn't actually true. Indeed, those packages are runtime library cross-built during gcc's build. Likewise, cross-toolchains may build-depend on foreign packages (ex: libc6-dev). Thus, specific arch qualifiers must be handled in the Build-Depends source package field too. Changing dpkg and apt to support that should be fairly straightforward. In fact, I already have a few patches, but I need to refresh them, maybe clean them up a little, and push them a little harder to get them accepted!

Pietro Abate: managing puppet manifest with gitolite

Managing the puppet manifest using a vcs is a best practice and there is a lot of material on the web. The easier way to do it, is to use git directly in the directory /etc/puppet and use a simple synchronization strategy with an external repo, either to publish your work, or simply to keep a backup somewhere. Things are a bit more complicated when you would like to co-administer the machine with multiple people. Setting up user accounts, permission and everything can be a pain in the neck. Moreover working from your desktop is always more comfortable then logging in as root on a remote system and make changes there ... The solution I've chosen to make my life a bit easier is to use gitolite, that is a simple git gateway that works using ssh public keys for authentication and does not require the creating of local users on the server machine. Gitolite is available in debian and installing it is super easy : apt-get install gitolite . If you use puppet already you might be tempted to use puppet to manage your gitolite installation. This is all good, but I don't advice you to use modules like this one http://forge.puppetlabs.com/gwmngilfen/gitolite/1.0.0 as it's going to install gitolite from source and on debian it's not necessary ... For my purposes, I didn't find necessary to manage gitolite with puppet as all the default config options where good enough for me. Once your debian package is installed, in order to initialize your repo, you just need to pass to gitolite the admin public key, that is your .ssh/id_rsa.pub key and then run this command:
sudo -H -u gitolite gl-setup /tmp/youruser.pub
This will create the admin and testing repo in /var/lib/gitolite/repositories and setup few other things. At this point you are ready to test you gitolite installation by cloning the admin repo :
git clone gitolite@example.org:gitolite-admin.git
Gitolite is engineered to use only the gitolite use to manage all your repositories. To add more repositories and users you should have a look at the documentation and then editing the file conf/gitolite.conf to add your new puppet repository. At this point, you can go two ways. If you use git to manage your puppet directory, you can just make a copy of it somewhere and then add gitolite as a remote
git remote add origin gitolite@example.org:puppet.git
If you didn't use git before, you can just copy the manifest in your new git repository, make a first commit and push it on the server.
git push origin master
Every authorized users can now use git to clone your puppet repository, hack, commit, push ...
git clone gitolite@example.org:puppet
One last step is to add a small post-receive hook on the server to synchronize your gitolite repository with the puppet directory in /etc : This will sync your main puppet directory and trigger changes on the nodes for the next puppetd run. First I created a small shell script in /usr/local/bin/puppet-post-receive-hook.sh :
#!/bin/bash
umask 0022
cd /etc/puppet
git pull -q origin master
This script presuppose that your git repo in /etc/puppet has the gitolite repo as origin ... Then I added a simple hook in the gitolite git repo that calls this script using sudo :
sudo /usr/local/bin/puppet-post-receive-hook.sh
And since you are at it you should also add a pre-commit hook to check the manifest syntax. This will save you a lot of useless commits. If you use a more complicated puppet setup using environments (I'm not there yet, and I don't think my setup will evolve in that direction in the near future), you can use puppet-sync that seems a neat script to do the job. For the moment this setup works pretty well. I'm tempted to explore mcollective to trigger puppet runs on my nodes, but I'm there yet...

26 May 2012

Vincent Sanders: Each morning sees some task begun, each evening sees it close; Something attempted, something done, has earned a night's repose.

Thursday saw all the Collabora employees at Cambridge office go out and socialise at the beer festival. They seemed to have selected a wonderful day for it, the sun was shining and it was warm and blue sky day.

Alas, I had to attend some customer conference calls and work on some time sensitive research so I could not go to the ball as it were. At about eight my brain had run out of steam so I decided to call it a day and go and meetup with people at the festival for an hour or two.

The queue when I arrived dissuaded me from that notion. I asked one of the stewards and they indicated it would take at least an hour from where the queue finished.

So I decided to wend my way home along the bank of the Cam. I proceeded slowly along and to my utter surprise bumped into Ben Hutchings and his Solarflare work colleagues having their own soiree. I was immediately invited to sit and converse. Pretty quickly I was inveigled into accepting a glass of wine by John Aspden from his floating bar (AKA houseboat).

From here on my evening was a pleasant one of amusing new people, easy conversation and a definite pondering if the host would be discovering the delights of Cam swimming as he became progressively inebriated!

So although I missed the festival I did manage to have an enjoyable time. A big thanks to the solarflare guys and especially John who was the consummate host and provided me with far too much alcohol.

Craig Small: JFFNMS 0.9.3

jffnms version 0.9.3 has been released today. This is a vast improvement over the 0.9.x releases and anyone using that train is strongly recommended to upgrade.So what changed? What didn t change! A nice summary would be fixing a lot of things that were broken or needed some tweaking. A really, really big thanks to Marek for all the testing and bug reports and also patient just run this and tell me what it says tests he did too. If something wasn t right before and works now, it is quite likely it is working because Marek told me how it broke. A brief overview of what has changed: You can download the file off sourceforge at https://sourceforge.net/projects/jffnms/files/JFFNMS%20Releases/

Charles Plessy: Cloud-init uploaded to Debian.

Each instance (virtual machine) started from the same image is configured identically. The same account and the same procedure will be used to connect to it. If the machine image is public and if the instance is open to the Internet, which is common when using commercial providers such as Amazon, an attacker could connect randomly to known IPs from the cloud, and eventually access to an instance before its owner. To individualise the connection to each instance, one can pass informations at startup, called instance metadata, which will be available to this instance only at a fixed HTTP URL. If the owner provides a public SSH key, this can be used to ensure that only the owner of the private key can connect. After the next update of network-console, this method of connection will be available for the Debian Installer. Likewise for the Debian images produced, by installing the cloud-init packages, that I uploaded to the experimental section of our archive. This package is far from ready. It lacks init scripts, it is not translated, and to be honest, I have not tested it. But to have it in Debian makes me benefit from the package and bugs tracking systems. Do not hesitate to report some, or better, to participate to the maintenance of this package. Cloud-init has many other interesting functionalities. I would like to thank Scott Moser, the upstream developer and maintainer of the Ubuntu package, for his cooperation during the creation of the Debian package, as I made large modifications, in particular the removal of the part managing the menu.lst for pv-grub, which will have its own package or will be transferred somewhere else.

Next.

Previous.