Search Results: "ed"

1 June 2012

Petter Reinholdtsen: SOAP based webservice from Dell to check server support status

A few years ago I wrote how to extract support status for your Dell and HP servers. Recently I have learned from colleges here at the University of Oslo that Dell have made this even easier, by providing a SOAP based web service. Given the service tag, one can now query the Dell servers and get machine readable information about the support status. This perl code demonstrate how to do it:
use strict;
use warnings;
use SOAP::Lite;
use Data::Dumper;
my $GUID = '11111111-1111-1111-1111-111111111111';
my $App = 'test';
my $servicetag = $ARGV[0] or die "Please supply a servicetag. $!\n";
my ($deal, $latest, @dates);
my $s = SOAP::Lite
    -> uri('http://support.dell.com/WebServices/')
    -> on_action( sub   join '', @_   )
    -> proxy('http://xserv.dell.com/services/assetservice.asmx')
    ;
my $a = $s->GetAssetInformation(
    SOAP::Data->name('guid')->value($GUID)->type(''),
    SOAP::Data->name('applicationName')->value($App)->type(''),
    SOAP::Data->name('serviceTags')->value($servicetag)->type(''),
);
print Dumper($a -> result) ;
The output can look like this:
$VAR1 =  
          'Asset' =>  
                     'Entitlements' =>  
                                       'EntitlementData' => [
                                                             
                                                              'EntitlementType' => 'Expired',
                                                              'EndDate' => '2009-07-29T00:00:00',
                                                              'Provider' => '',
                                                              'StartDate' => '2006-07-29T00:00:00',
                                                              'DaysLeft' => '0'
                                                             ,
                                                             
                                                              'EntitlementType' => 'Expired',
                                                              'EndDate' => '2009-07-29T00:00:00',
                                                              'Provider' => '',
                                                              'StartDate' => '2006-07-29T00:00:00',
                                                              'DaysLeft' => '0'
                                                             ,
                                                             
                                                              'EntitlementType' => 'Expired',
                                                              'EndDate' => '2007-07-29T00:00:00',
                                                              'Provider' => '',
                                                              'StartDate' => '2006-07-29T00:00:00',
                                                              'DaysLeft' => '0'
                                                             
                                                          ]
                                      ,
                     'AssetHeaderData' =>  
                                          'SystemModel' => 'GX620',
                                          'ServiceTag' => '8DSGD2J',
                                          'SystemShipDate' => '2006-07-29T19:00:00-05:00',
                                          'Buid' => '2323',
                                          'Region' => 'Europe',
                                          'SystemID' => 'PLX_GX620',
                                          'SystemType' => 'OptiPlex'
                                         
                    
         ;
I have not been able to find any documentation from Dell about this service outside the inline documentation, and according to one comment it can have stability issues, but it is a lot better than scraping HTML pages. :) Wonder if HP and other server vendors have a similar service. If you know of one, drop me an email. :)

Pietro Abate: apt-get with external solvers : call for testers

Last year we invited David to work with us for a few days to add a generic interface to apt to call external solvers. After a few iterations, this patch finally landed in master and recently (about 3 months ago), in debian unstable.
  [ David Kalnischkies ]
  * [ABI-Break] Implement EDSP in libapt-pkg so that all front-ends which
    use the internal resolver can now be used also with external
    ones as the usage is hidden in between the old API
  * provide two edsp solvers in apt-utils:
    - 'dump' to quickly output a complete scenario and
    - 'apt' to use the internal as an external resolver
Today the new version of apt-cudf was upload in unstable and with it the latest bug fixes that makes it ready for daily use. I've used it quite a lot myself to upgrade my machine and it seems working pretty well so far... The most important difference with the old version is the support for multi-arch enabled machines. This marks an important milestone in our efforts to integrate external solvers, built using different technologies, directly into apt. From a user prospective, this means that (s)he will have the possibility to check if there exists a better (best ?) solution to an installation problem then what proposed by the internal apt solver. Moreover, even if apt-get gives very satisfactory answers, there are occasions where it fails miserably, leaving the user wondering how to unravel the complex web of dependencies to accomplish his task. Available cudf solvers in debian are at the moment : aspcud, mccs and packup. From an architectural point of view this is accomplished by abstracting the installation problem via a simple textual protocol (EDSP) and using an external tool to do the heavy duty translation. Since all solvers now available in debian are not meant to be debian-specific, using them involve a two step translation. The EDSP protocol specification is for the moment "hidden" in the apt documentation. I hope to find a better place for it soon : it would be cool if other package managers as smart or cupt could add an implementation on EDSP in their code so to automatically benefit of this technology. To make it happen, Apt first creates an EDSP file that is then handled to apt-cudf that takes care of the translation to cudf and back into EDSP that is then read back by apt. Apt-cudf is the bridge between edsp and the external solvers and takes care of doing the book keeping and to select the right optimization criteria. Roberto recently wrote a very nice article explaining how to use apt-get with an external solver. In a nutshell, if you want to try this out you just need to install apt-cudf, one external solver like aspcud from the university of Pasdam and then call apt-get using the --solver option (that is not yet documented #67442 ). For example :
apt-get install -s gnome --solver aspcud
This will install gnome while using the a optimization criteria that tries to minimizing the changes on the system. Various other optimization criteria for all apt-get default actions can be specified in the apt-cudf configuration file /etc/apt-cudf.conf I hope the new release of apt-cudf make it into testing before the freeze. Time to test !!!

Raphaël Hertzog: My Debian Activities in May 2012

This is my monthly summary of my Debian related activities. If you re among the people who made a donation to support my work (338.26 , thanks everybody!), then you can learn how I spent your money. Otherwise it s just an interesting status update on my various projects. Dpkg Like last month, I did almost nothing concerning dpkg. This will probably change in June now that the book is out The only thing worth noting is that I have helped Carey Underwood who was trying to diagnose why btrfs was performing so badly when unpacking Debian packages (compared to ext4). Apparently this already resulted in some btrfs improvements. But not as much as what could be hoped. The sync_file_range() calls that dpkg are doing only force the writeback of the underlying data and not of the meta-data. So the numerous fsync() that follow still create many journal transactions that would be better handled as one big transaction. As a proof of this, replacing the fsync() with a sync() brings the performance on par with ext4. (Beware this is my own recollection of the discussion, while it should be close to the truth, it s probably not 100% accurate when speaking of the brtfs behaviour) Packaging I uploaded new versions of smarty-gettext and smarty-validate because they were uninstallable after the removal of smarty. The whole history of smarty in Debian/Ubuntu has been a big FAIL since the start. Once upon a time, there was a smarty package and some plugins. Everything was great except that the files were installed in a way that differs from the upstream recommendations. So Ubuntu changed the path in their version of the package and did not check whether it broke anything else (and it did break all the plugins). Despite the brokenness of the plugins, this divergence survived for years. So several packages that were using Smarty were modified to use dpkg-vendor to use the correct path depending on whether it was built on Debian or Ubuntu. In 2010, Smarty 3.0 has been released and instead of upgrading the smarty package to this version, one of the smarty co-maintainers introduced a smarty3 package that used yet another path (despite the fact that smarty 3 had a mode to be compatible with smarty 2).
At some point, I informed him that he had to handle the migration of users of smarty to smarty3 he acknowledged and then lost interest in smarty ( I m no longer using it ) and did nothing. After some more bitrot, smarty has been forcefully orphaned in August 2011 by a member of the security team. And in March this year, it has been removed from unstable despite the fact that it still had reverse dependencies (usually removals only happen when they impact no other packages, I don t know why this wasn t the case here). At least the brokenness attracted some attention to the situation and Mike Gabriel contacted me about it. I offered him to take over the various packages since they all needed a real maintainer and he accepted. I sponsored his uploads of all smarty related packages (bringing in the latest upstream versions at the same time). In the end, the situation is looking better now, except that there s no migration path from users who rely on smarty in Squeeze. They will discover that they need smarty3 in Wheezy and that the various paths have to be adjusted. It s probably acceptable since the new upstream versions are no longer backwards compatible with smarty 2 The Debian Administrator s Handbook At the start of the month, I was busy preparing the release of the book. I introduced the publican-debian package to unstable, it s a Publican brand (aka a set of CSS and XSL stylesheets to tailor the output of Publican) using the Debian colors and using the Debian logo. This brand is used by the book. I also created the debian-handbook package and setup the public Git repository on alioth.debian.org. I was ready or so I thought. A few hours after the announce, the website became unusable because the numerous visitors were exhausting the maximum number of client connections. And I could not increase the limit due to Apache s memory usage (with PHP and WordPress). We quickly off-loaded most of the static files traffic to another machine and we setup bittorrent. The problem was solved for the short term. Thousands of persons downloaded the ebook and to this date, 135 copies of the paperback have been sold. Then I took a one-week vacation. Even though I had no Internet at the place I was, I wandered in the street to find a Freewifi wifi network (customers of the Free ISP can use those freely) to stay on top of incoming email. We quickly received some bug reports and I dealt with the easy ones (typos and the like) on the fly. When I came back at home, I manually placed 54 lulu orders for the people who opted for the paperback as reward during the fundraising campaign. A bit tedious but it had to be done (if only Lulu supported a way to batch many orders at once ). I also wanted a long term solution to avoid the use of an external host to serve static files (should a new traffic spike arrive ). So I installed nginx as a front-end. It serves static files directly, as well as WordPress pages which have been cached by wp-super-cache. Apache is still here listening on a local port and responding to the remaining queries forwarded by nginx. Once I ll migrate to wheezy, I might completely ditch apache in favor of php5-fpm to handle the PHP pages. Last but not least, I wanted to bootstrap the various translations that people offered to contribute. I wrote some documentation for interested translators and blogged about it. It s shaping up nicely check it out if you re interested to help! Thanks See you next month for a new summary of my activities.

No comment Liked this article? Click here. My blog is Flattr-enabled.

Steve Kemp: I've posted my javascript stuff

I previously mentioned some work I'd done with mixing Javascript & C. The code is now visible here: This consists of a binary which allows you to run javascript files, those javascript files have some extra methods available to them, allowing this sample file to execute and do things: Not great work, but still a useful exercise. ObQuote: "Please! I don't wanna go back there, you don't know what it's like to be treated as a freak!" - Shrek

Francois Marier: Proper indentation of Javascript files using js2-mode in emacs

If you use emacs for Javascript or Node.js development, you should have a look at js2-mode (apt-get install js2-mode in Debian / Ubuntu). In addition to providing syntax highlight, it will parse your Javascript and issue errors and warnings. (It's not as pedantic as jslint or jshint so you'll probably want to fix all of them before committing your files.) Unfortunately the default indentation style looks a bit crazy and adds excessive indentation to your code. Here's an example:

navigator.id.secret.generate(identity, function (plainKey, wrappedKey)
var data = wrappedKey: wrappedKey ;
$.post('/loggedin', data, function (res)
$("#message").text('All done');
, 'json');
, function (error)
$("#message").text('ERROR: ' + error);
);
It turns out that indenting Javascript properly is really hard, but you can turn on a special flag which will cause the indentation to "bounce" between different strategies, starting with the most likely one. This is what I now have in my ~/.emacs:
(custom-set-variables
'(js2-basic-offset 2)
'(js2-bounce-indent-p t)
)
You can find and configure all other options by issuing the following command:
M-x customize-group [RET] js2-mode [RET]
So yes, you can have both the reasonable indentation of the standard js-mode and the helpful warnings and errors of js2-mode!

31 May 2012

Petter Reinholdtsen: First monitor calibration using ColorHug

A few days ago my color calibration gadget ColorHug arrived in the mail, and I've had a few days to test it. As all my machines are running Debian Squeeze, where the calibration software is missing (it is present in Wheezy and Sid), I ran the calibration using the Fedora based live CD. This worked just fine. So far I have only done the quick calibration. It was slow enough for me, so I will leave the more extensive calibration for another day. After calibration, I get a ICC color profile file that can be passed to programs understanding such tools. KDE do not seem to understand it out of the box, so I searched for command line tools to use to load the color profile into X. xcalib was the first one I found, and it seem to work fine for single monitor setups. But for my video player, a laptop with a flat screen attached, it was unable to load the color profile for the correct monitor. After searching a bit, I discovered that the dispwin tool from the argyll package would do what I wanted, and a simple
dispwin -d 1 profile.icc
later I had the color profile loaded for the correct monitor. The result was a bit more pink than I expected. I guess I picked the wrong monitor type for the "led" monitor I got, but the result is good enough for now.

Maximilian Attems: New klibc release 2.0

The new klibc features prominently the merged stdio branch (causing the version bump). The buffered I/0 allows a faster fstype and thus faster boot. Plus klibc gained the support of several stream I/O functions, for details see the longer story on: klibc 2.0 release announcement or klibc git repository. The dhcp cmd ipconfig can now generate enough info for a proper lease file too and got fixed concerning the min length of it's replies. Google sponsored the /etc/fstab or cmdline boot mounting in kinit. 2.0 is shipping several arch fixes including i386 signal fixes, m68 support, s390x, sh4 compile fixes. There is still portability work todo: #439181 ia64 shared lib buggy (longdated unresolved ia64 mystery), #634890 amrhf all binaries segfault (maybe signal related), status on parisc is unknown, avr32 patch is untested (might as well just merge) and sh4 SIGILL. We got nice arch testcoverage thanks to mksh testsuite and working mksh klibc support. We are confident of this release on all this archs: alpha, armel, m68k, mips, ppc, ppc64, sparc, s390, s390x and x86. (:

Russell Coker: Links May 2012

Vijay Kumar gave an interesting TED talk about autonomous UAVs [1]. His research is based on helicopters with 4 sets of blades and his group has developed software to allow them to develop maps, fly in formation, and more. Hadiyah wrote an interesting post about networking at TED 2012 [2]. It seems that giving every delegate the opportunity to have their bio posted is a good conference feature that others could copy. Bruce Schneier wrote a good summary of the harm that post-911 airport security has caused [3]. Chris Neugebauer wrote an insightful post about the drinking culture in conferences, how it excludes people and distracts everyone from the educational purpose of the conference [4]. Matthew Wright wrote an informative article for Beyond Zero Emissions comparing current options for renewable power with the unproven plans for new nuclear and fossil fuel power plants [5]. The Free Universal Construction Kit is a set of design files to allow 3D printing of connectors between different types of construction kits (Lego, Fischer Technic, etc) [6]. Jay Bradner gave an interesting TED talk about the use of Open Source principles in cancer research [7]. He described his research into drugs which block cancer by converting certain types of cancer cell into normal cells and how he shared that research to allow the drugs to be developed for clinical use as fast as possible. Christopher Priest wrote an epic blog post roasting everyone currently associated with the Arthur C. Clarke awards, he took particular care to flame Charles Stross who celebrated The Prestige of such a great flaming by releasing a t-shirt [8]. For a while I ve been hoping that an author like Charles Stross would manage to make more money from t-shirt sales than from book sales, Charles is already publishing some of his work for free on the Internet and it would be good if he could publish it all for free. Erich Schubert wrote an interesting post about the utility and evolution of Favebook likes [9]. Richard Hartmann wrote an interesting summary of the problems with Google products that annoy him the most [10]. Sam Varghese wrote an insightful article about the political situation in China [11]. The part about the downside of allowing poorly educated people to vote seems to apply to the US as well. Sociological Images has an article about the increased rate of Autism diagnosis as social contagion [12]. People who get their children diagnosed encourage others with similar children to do the same. Vivek wrote a great little post about setting up WPA on Debian [13]. It was much easier than expected once I followed that post. Of course I probably could have read the documentation for ifupdown, but who reads docs when Google is available? Related posts:
  1. Links March 2012 Washington s Blog has an informative summary of recent articles about...
  2. Links April 2012 Karen Tse gave an interesting TED talk about how to...
  3. Links February 2012 Sociological Images has an interesting article about the attempts to...

Dominique Dumont: cme fix dpkg now track and display changes

Hello I ve released Config::Model 2.018 which is now able to track the changes done with cme to your configuration data. Thw GUI now features a new menu entry File -> show unsaved change that will give you a list of the changes done since running the command or since the last save (whichever occured last). For instance: This feature is also available in non-graphic mode. As an example, let s refresh the lcdproc package:
$ cme fix dpkg
Fixing...
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Warning in 'control source Standards-Version' value '3.9.2': Current standards version is 3.9.3
Warning in 'copyright Format' value 'http://dep.debian.net/deps/dep5/': Format does not match the recommended URL for DEP-5
Changes:
- control source Standards-Version: '3.9.2' -> '3.9.3' # applied fix
- copyright Format: 'http://dep.debian.net/deps/dep5/' -> 'http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/' # applied fix
The Changes list can be reused to update the changelog, but the author does not dare injecting this directly in debian/changelog Let s check the resulting changes:
$ git diff
diff --git a/debian/control b/debian/control
index 11a5b8a..2b81192 100644
--- a/debian/control
+++ b/debian/control
@@ -20,7 +20,7 @@ Build-Depends: autoconf,
                libxosd-dev,
                pkg-config,
                texinfo
-Standards-Version: 3.9.2
+Standards-Version: 3.9.3
 Vcs-Browser: http://git.debian.org/?p=collab-maint/lcdproc.git;a=summary
 Vcs-Git: git://git.debian.org/collab-maint/lcdproc.git
 Homepage: http://www.lcdproc.org/
diff --git a/debian/copyright b/debian/copyright
index 6cea444..4b48619 100644
--- a/debian/copyright
+++ b/debian/copyright
@@ -1,4 +1,4 @@
-Format: http://dep.debian.net/deps/dep5/
+Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
 Upstream-Name: lcdproc
 Upstream-Contact: William W. Ferrell 
 Source: http://www.lcdproc.org/
domi@ylum:~/debian-dev/lcdproc-stuff/lcdproc$ 
Last but not least, this track and display changes features is also available for other models, like OpenSsh, lcdproc, multistrap All the best
Tagged: Config::Model, dpkg, Perl

Julian Andres Klode: Memory allocators

In the past days, I looked at various memory allocators in a quest to improve performance in my multi-threaded test cases of a reference counting language runtime / object allocator (a fun project). It turns out the glibc s memory allocator is relatively slow, especially if you do loops that create one element and destroy one element at the same time (for example, map() on a list you no longer use after you passed it to map). To fix this problem, I considered various options. The first option was to add a thread-local cache around malloc(). So whenever we want to free() an object, we place it on a thread-local list instead, and if a malloc() request for an object of that size comes in, we reuse the old object. This fixes the problem with the lists, but experiments shown another problem with the glibc allocator: Allocating many objects without releasing others (let s say appending an element to a functional list). I started testing with tcmalloc instead, and noticed that it was several times faster (reducing run time from 6.75 seconds to 1.33 seconds (5 times faster)). As I do not want to depend on a C++ code base, I decided to write a simple allocator that allocates blocks of memory via mmap(), splits those into objects, and puts the objects on a local free list. That allocator performed faster than tcmalloc(), but was also just a simple test case, and not really useable, due to potential fragmentation problems (and because the code was statically linked in, causing thread-local storage to be considerably faster, as it does not need to call a function on every TLS access, but can rather go through a segment register). I then discovered jemalloc, and tried testing with this. It turned out that jemalloc was even faster than tcmalloc in all test cases, and also required about the same amount of memory as tcmalloc (and slightly more than the custom test code, as that combined reference counting with memory allocator metadata and thus had less meta data overhead). Using jemalloc, the list building test performs in 0.9X seconds now (instead of tcmalloc s 1.33 or glibc s 6.75 seconds), requires 0 major page faults (tcmalloc has 2), and uses 5361472 kb memory (tcmalloc uses 5290240, glibc requires 7866608). Given that jemalloc is written in C, I can only recommend it to everyone in need of a scalable memory allocator. Depending on your workload, it might require less memory than tcmalloc (at least that s the case at Facebook) and is most likely faster than tcmalloc. It also provides tcmalloc-compatible profiling facilities (as Facebook needed them). Furthermore, jemalloc is also the memory allocator used by FreeBSD, and is used by Mozilla projects, and on several Facebook projects, and should thus be relatively stable and useable. All tests were run with 4 threads, on a dual-core laptop with hyper-threading, using (e)glibc 2.13, tcmalloc 2.0, and jemalloc 3.0. The tests may not be representative of real world performance, and do not account for fragmentation. Should I try replacing Python s custom caching of malloc() calls with simply calling jemalloc, and run a few Python benchmarks? Anyone interested in that?
Filed under: General

Richard Hartmann: Potpourri II

Traffic jams suck; but they do give you a chance to think and blog. Truckers hams are telling me that four trucks are totalled and need to be removed by the THW. As the trucker next to me just broke out his gas grill and several people are joining him in a spontaneous BBQ, I don't expect to be moving any time soon. Kicking things along This is coming along nicely RAID 4 Unless you are forced to support legacy systems or playing RAIDemon and gotta try them all, there is no valid use case for RAID 4. So doing so is a patently bad idea. RAID 5 is always the better option as you are not relying on a single disk which contains the checksums, but use striping to distribute them. RAID 4 has a chance of 1/n that a failed disk will force all remaining disks to read all respective sectors and then the controller/CPU to XOR everything. There is a reason why RAID 3 was replaced with RAID 4; just as there's a reason why RAID 4 was replaced by RAID 5. Don't use RAID 4. Seriously. Tom Tom vs OSM In case you have been following the news in the GPS circles, you will have noticed that TomTom pretty gave up on innovating and resigned themselves to dying a slow death. Well that and to start a nice amount of FUD. While they have been rebuked pretty thoroughly already, I wanted to add my own POV. There is a kernel of truth in what TomTom said inasmuch they state that pedestrian paths are better than car paths. Of course, while they got the relative quality right, they are wrong about the absolute quality of the data. They claim pedestrian is OK and motorized is bad. In my experience, maps for motorized traffic are good while pedestrian tracks are mind-boggingly great. These days, most people (and probably all who read this) will be at least somewhat familiar with the quality of OSM in the random cities they visit over the year. The common verdict is usually "very nice". But that's not where OSM shines. It's in the lesser-known places where you don't expect too many tech-minded people to come along where it starts to really impress me. While I found a few stubs on a dead-end street in Akureyri which were not mapped, every foot trail I ever walked along in Iceland has been mapped. ski-doo tracks on Svalbard? Check. Backyard in Novosibirsk? Check, as well. And we are talking "packed earth track in between garages opening up into a 'playground' where a man and a pregnant women are passing a bottle back and forth while smoking and watching over their two kids who play next to a middle-aged man who is lying on the ground wasted, using a rock as a pillow all the while an extremely blingy Mercedes with two muscle-packed thugs is cruising by, all in the time it took us to walk through". Mapped. Horse tracks and hiking trails in Mongolia? Yep. Hutongs in Beijing including alleys so narrow you have to walk single file? Of course. Richard Fairhurst is right that the plural of anecdote is not data, but OSM is good. TomTom is afraid for a reason. As an aside, I find the picture used in TomTom's newsletter extremely funny; I am sure they didn't intend this, but hey. Windows 100-300 MiB for any random driver installation package is par for the course. Be it GFX, audio, or network card. If you plug in a different USB mouse, Windows 7 will take several seconds while "setting up" a standard USB HID device. And hitting ctrl-alt-del in the login screen or while logged in does not give you a quick way to shut down or reboot the machine. As I am breaking out the Windows installation every month to install updates and every six months to actually use this does not affect me a lot. Still, it's amazing that so many people are putting up with this. UEFI It's nice to see that Matthew Garrett is still going strong. I am not sure if this is the best way, but it seems to be a workable solution while in between a rock and a hard place. Not sure about the requirements of GPLv3 with regards to the keys Fedora will use to sign GRUB 2 with. From what I remember without rereading the license, as long as Fedora is not selling a system on which you can not disable secure boot they should be in the clear. IANAL, IIRC, etc. Wrapping up And I have not moved a single meter, other than making way to a police motorbike, in over an hour. Pity I don't have anything to BBQ with me; smells delicious over there.

Marc 'Zugschlus' Haber: atop in unstable

Eight days ago, I uploaded atop 1.26-1 to DELAYED/8, listing me as new maintainer. This means that the package has in the mean time appeared in unstable, and I hope that it ll swiftly migrate to testing.

Matthew Garrett: Implementing UEFI Secure Boot in Fedora

Edit 16:17EDT 31/5/12: Clarification of who gets the $99
Edit 02:10EDT 01/6/12: Clarification that it's a one-off payment

(Brief disclaimer - while I work for Red Hat, I'm only going to be talking about Fedora here. Anything written below represents only my opinions and my work on Fedora, not Red Hat's opinions or future plans)

Fedora 17 was released this week. It's both useful and free, and serves as a welcome addition to any family gathering. Do give it a go. But it's also noteworthy for another reason - it's the last Fedora release in the pre-UEFI secure boot era. Fedora 18 will be released at around the same time as Windows 8, and as previously discussed all Windows 8 hardware will be shipping with secure boot enabled by default. While Microsoft have modified their original position and all x86 Windows machines will be required to have a firmware option to disable this or to permit users to enrol their own keys, it's not really an option to force all our users to play with hard to find firmware settings before they can run Fedora. We've been working on a plan for dealing with this. It's not ideal, but of all the approaches we've examined we feel that this one offers the best balance between letting users install Fedora while still permitting user freedom.

Getting the machine booted
Most hardware you'll be able to buy towards the end of the year will be Windows 8 certified. That means that it'll be carrying a set of secure boot keys, and if it comes with Windows 8 pre-installed then secure boot will be enabled by default. This set of keys isn't absolutely fixed and will probably vary between manufacturers, but anything with a Windows logo will carry the Microsoft key[1].

We explored the possibility of producing a Fedora key and encouraging hardware vendors to incorporate it, but turned it down for a couple of reasons. First, while we had a surprisingly positive response from the vendors, there was no realistic chance that we could get all of them to carry it. That would mean going back to the bad old days of scouring compatibility lists before buying hardware, and that's fundamentally user-hostile. Secondly, it would put Fedora in a privileged position. As one of the larger distributions, we have more opportunity to talk to hardware manufacturers than most distributions do. Systems with a Fedora key would boot Fedora fine, but would they boot Mandriva? Arch? Mint? Mepis? Adopting a distribution-specific key and encouraging hardware companies to adopt it would have been hostile to other distributions. We want to compete on merit, not because we have better links to OEMs.

An alternative was producing some sort of overall Linux key. It turns out that this is also difficult, since it would mean finding an entity who was willing to take responsibility for managing signing or key distribution. That means having the ability to keep the root key absolutely secure and perform adequate validation of people asking for signing. That's expensive. Like millions of dollars expensive. It would also take a lot of time to set up, and that's not really time we had. And, finally, nobody was jumping at the opportunity to volunteer. So no generic Linux key.

The last option wasn't hugely attractive, but is probably the least worst. Microsoft will be offering signing services through their sysdev portal. It's not entirely free (there's a one-off $99 fee to gain access edit: The $99 goes to Verisign, not Microsoft - further edit: once paid you can sign as many binaries as you want), but it's cheaper than any realistic alternative would have been. It ensures compatibility with as wide a range of hardware as possible and it avoids Fedora having any special privileges over other Linux distributions. If there are better options then we haven't found them. So, in all probability, this is the approach we'll take. Our first stage bootloader will be signed with a Microsoft key.

Bootloaders
We've decided to take a multi-layer approach to our signing for a fairly simple reason. Signing through the Microsoft signing service is a manual process, and that's a pain. We don't want to have bootloader updates delayed because someone needs to find a copy of Internet Explorer and a smartcard and build packages by hand. Instead we're writing a very simple bootloader[2]. This will do nothing other than load a real bootloader (grub 2), validate that it's signed with a Fedora signing key and then execute it. Using the Fedora signing key there means that we can build grub updates in our existing build infrastructure and sign them ourselves. The first stage bootloader should change very rarely, and we don't envisage updating it more than once per release cycle. It shouldn't be much of a burden on release management.

What about grub? We've already switched Fedora 18 over to using grub 2 by default on EFI systems, but it still needs some work before it's ready for secure boot. The first thing is that we'll be disabling the module loading. Right now you can load arbitrary code into grub 2 at runtime, and that defeats the point of secure boot. So that'll be disabled. Next we'll be adding support for verifying that the kernel it's about to boot is signed with a trusted key. And finally we'll be sanitising the kernel command line to avoid certain bits of functionality that would permit an attacker to cause even a signed kernel to launch arbitrary code. These restrictions will all vanish if secure boot is disabled.

Kernel
Secure boot is built on the idea that all code that can touch the hardware directly is trusted, and any untrusted code must go through the trusted code. This can be circumvented if users can execute arbitrary code in the kernel. So, we'll be moving to requiring signed kernel modules and locking down certain aspects of kernel functionality. The most obvious example is that it won't be possible to access PCI regions directly from userspace, which means all graphics cards will need kernel drivers. Userspace modesetting will be a thing of the past. Again, disabling secure boot will disable these restrictions.

Signed modules are obviously troubling from a user perspective. We'll be signing all the drivers that we ship, but what about out of tree drivers? We don't have a good answer for that yet. As before, we don't want any kind of solution that works for us but doesn't work for other distributions. Fedora-only or Ubuntu-only drivers are the last thing anyone wants, and this really needs to be handled in a cross-distribution way.

Wait signed what
Secure boot is designed to protect against malware code running before the operating system. This isn't a hypothetical threat. Pre-boot malware exists in the wild, and some of it is nastier than you expect. So obviously bootloaders need to be signed, since otherwise you'd just replace the signed bootloader with an unsigned one that installed malware and booted your OS.

But what about the kernel? The kernel is just code. If I take a signed Linux bootloader and then use it to boot something that looks like an unsigned Linux kernel, I've instead potentially just booted a piece of malware. And if that malware can attack Windows then the signed Linux bootloader is no longer just a signed Linux bootloader, it's a signed Windows malware launcher and that's the kind of thing that results in that bootloader being added to the list of blacklisted binaries and suddenly your signed Linux bootloader isn't even a signed Linux bootloader. So kernels need to be signed.

And modules? Again, modules are just code. It's a little trickier, but if your signed kernel loads an unsigned module then that unsigned module can set up a fake UEFI environment and chain into a compromised OS bootloader. Now the attacker just has to include a signed kernel and a minimal initramfs that loads their malware module. It'd slow down boot by a couple of seconds, but other than that it'd be undetectable. X? If you can access the registers on a GPU then you can get the GPU to DMA over the kernel and execute arbitrary code. Trickier again, but still achievable - and if you've locked down every other avenue of attack, it's even attractive.

If we produce signed code that can be used to attack other operating systems then those other operating systems are justified in blacklisting us. That doesn't seem like a good outcome.

Customisation
A lot of our users want to build their own kernels. Some even want to build their own distributions. Signing our bootloader and kernel is an impediment to that. We'll be providing all the tools we use for signing our binaries, but for obvious reasons we can't hand out our keys. There's three approaches here. The first is for a user to generate their own key and enrol it in their system firmware. We'll trust anything that's signed with a key that's present in the firmware. The second is to rebuild the shim loader with their own key installed and then pay $99 and sign that with Microsoft. That means that they'll be able to give copies to anyone else and let them install it without any fiddling. The third is to just disable secure boot entirely, at which point the machine should return to granting the same set of freedoms as it currently does.

But I don't trust Microsoft
A system in custom mode should allow you to delete all existing keys and replace them with your own. After that it's just a matter of re-signing the Fedora bootloader (like I said, we'll be providing tools and documentation for that) and you'll have a computer that will boot Fedora but which will refuse to boot any Microsoft code. It may be a little more awkward for desktops because you may have to handle the Microsoft-signed UEFI drivers on your graphics and network cards, but this is also solvable. I'm looking at ways to implement a tool to allow you to automatically whitelist the installed drivers. Barring firmware backdoors, it's possible to configure secure boot such that your computer will only run software you trust. Freedom means being allowed to run the software you want to run, but it also means being able to choose the software you don't want to run.

You've sold out
We've been working on this for months. This isn't an attractive solution, but it is a workable one. We came to the conclusion that every other approach was unworkable. The cause of free software isn't furthered by making it difficult or impossible for unskilled users to run Linux, and while this approach does have its downsides it does also avoid us ending up where we were in the 90s. Users will retain the freedom to run modified software and we wouldn't have accepted any solution that made that impossible.

But is this a compromise? Of course. There's already inequalities between Fedora and users - trademarks prevent the distribution of the Fedora artwork with modified distributions, and much of the Fedora infrastructure is licensed such that some people have more power than others. This adds to that inequality. It's not the ideal outcome for anyone, and I'm genuinely sorry that we weren't able to come up with a solution that was better. This isn't as bad as I feared it would be, but nor is it as good as I hoped it would be.

What about ARM
Microsoft's certification requirements for ARM machines forbid vendors from offering the ability to disable secure boot or enrol user keys. While we could support secure boot in the same way as we plan to on x86, it would prevent users from running modified software unless they paid money for a signing key. We don't find that acceptable and so have no plans to support it.

Thankfully this shouldn't be anywhere near as much of a problem as it would be in the x86 world. Microsoft have far less influence over the ARM market, and the only machines affected by this will be the ones explicitly designed to support Windows. If you want to run Linux on ARM then there'll be no shortage of hardware available to you.

Is this all set in stone?
No. We've spent some time thinking about all of this and are happy that we can implement it in the Fedora 18 timescale, but there's always the possibility that we've missed something or that a new idea will come up. If we can increase user freedom without making awful compromises somewhere else then we'll do it.

[1] In fact, chances are that everything will carry the Microsoft key. Secure boot requires that UEFI drivers also be signed. The signing format only permits a single signature per binary. For compatibility, approximately all add-on hardware shipped will be signed with Microsoft's key, and that means that all system vendors have to recognise Microsoft's key in order to permit that hardware to run on their systems.
[2] Current source is here. It relies on a port of the UEFI crypto library and OpenSSL that I have building with some handholding, and which I'll upload as soon as possible.

comment count unavailable comments

Paul Tagliamonte: my sbuilder setup (sb)

I ve expanded a bit on my idea (that is, the sbuilder wrapper to help aid with managing more then one sbuild chroot), and it got out of hand very quickly. It s gone from one small shell script to a set of 8, and growing. Quickly. Until I spin these off into their own repo, they ll be in my dot-bin Before getting too far, please relize this is only how I m using sbuild. There are many other (more) sane configurations that you can (and should) use. This was not made to be useful for anyone other then me :) I ve added the ability to use unionfs-fuse, set up a way to login interactively and a script to create new chroots without having them register as a global schroot. Firstly, forgive me for my sins. This is all a huge hack, but it s use isn t the worst thing in the world. As of the time of this writing, I now have a gcc and clang chroot for each of stable, testing and unstable. Now the big question how do you use this. Rather, this is more of a how do I use sbuilder :) Firstly, I ve created a file for each schroot set that I m using. Here s my /etc/schroot/chroot.d/sid.
[sid]
type=directory
description=Debian sid/amd64 autobuilder
directory=/var/sbuild/sid
groups=root,sbuild
root-groups=root,sbuild
script-config=sbuild/config
It s very similar to the one that sbuilder-createchroot creates. If you re just looking for aufs / unionfs support, keep in mind schroot has some great support. check man 5 schroot.conf. Next, create a dist.d directory in /var/sbuild such as /var/sbuild/sid.d. In the future, this might get automagically created by the sb-create script. For now, it just expects that exists. The arguments for all the sb* commands is always in the order of variant, dist. variant defaults to generic , which I ve set as an alias to the generic chroot, and the dist defaults to sid . At some point, I ll change the default to be the last UNRELEASED entry in the changelog. So, to create a new chroot to become your clang chroot, you may choose to invoke it as sb-create clang, or more explicitly sb-create clang sid. To create a new chroot to become your stable / gcc chroot, you may create it with sb-create gcc stable. After you have a chroot, you can start to use the sb or sbo scripts. sbo is just a wrapper around the sb script that sets an envvar to tell the sb-prehook and sb-posthook to set up the unionfs overlay. The unionfs overlay routine replaces the dist symbolic link (such as /var/sbuild/testing) with a folder, and mounts the real chroot (such as /var/sbuild/testing.d/testing-gcc) under the unionfs divert point (which is something like /var/sbuild/testing.overlay.d/testing-gcc). This lets us make sure the chroot is never affected by a routine build, even if it s designed to trash a chroot (unless, of course, it uses any number of ways to get outside the chroot and trash the pristine directory this is to prevent mistakes, not malicious scripts). By the way, for those interested, there s a small issue with what I m describing it s very subtle. Basically, there s a few assumptions made by how schroot operates, and handles sessions . Since it ll keep stuff mounted if the session is still active, it makes purging all this a real pain. The magic behind the scripts is in the sb-endsessions routine which will properly end all the schroot sessions for that dist, and since we have our own high-level locking system, we shouldn t have a situation where this kills off another build. The code (which might have issues) looks like:
SESSIONS= schroot --all-sessions -l   grep "^session:$DIST-" 
for session in $SESSIONS; do
    echo "W: Ending session $session"
    schroot -e -c $session
done
By backing of schroot s mounts, we can do our mounting / removing without worrying about klobbering schroot. Now, there s also the case of wanting to interactively log into a chroot. For that, there s sb-login and sb-maint, both take the usual variant / dist arguments. sb-login logs in as the current user, and sb-maint logs in as the root user. Eventually, I ll be adding some scripts to upgrade all the build chroots for a dist at once.

30 May 2012

Jaldhar Vyas: Dovecot 2.1.7 Uploaded to Unstable

This is the latest upstream version and will, barring any unforseen circumstances, be the version in wheezy, the upcoming stable release of Debian GNU/Linux. In order to ensure it is the high quality our users have come to expect from Debian stable, I need your help. First and foremost, please test the hell out of it. Upgrades during a stable release are in general not allowed unless the problem is security related so we need to shake all the bugs out now. I am particularly interested in knowing if there are any issues with upgrades from squeeze. I waved my hands ineffectually at the bug list but it needs more attention. Could some kind person go through them and let me know which ones are no longer valid, need further attention etc.? I attempted to add hardening flags but lintian still complains a lot. I think the warnings are probably false positives but I'm not a 100% sure so I would appreciate some advice from those who know more about such things. Those are the issues that come to mind at the moment but feel free to let me know if there's something I've missed.

Lior Kaplan: Debian packaging for beginners @ FOSDEM the video

I recently got a question by email about the lecture I presented at FOSDEM, and by that mail I discovered that the video of the lecture was uploaded to YouTube (FOSDEM 2012 Debian packaging for beginners). While I thank the person who did the upload, he/she did not keep the right license for the video which is CC-BY-SA 3.0 (as noted at FOSDEM archive). You can also grab the webm file directly. Well, the next task will be to listen to myself in order to improve for FOSDEM 2013 See you there.
Filed under: Debian GNU/Linux, FOSDEM

Thomas Goirand: New release of MLMMJ (version 1.2.18.0) uploaded in Debian

MLMMJ stands for Mailing List Manager Made Joyful. To me, it s the best mailing list manager available. Not only its written in C (no slow interpreted language here ), but also it is easy and very convenient to setup. No hack is needed to make it run on a multi-domain environment, and no need to run a special subdomain for your lists (yes, I m thinking about you, silly Mailman ). If you didn t know, MLMMJ is used to handle relatively high traffic lists: all mailing lists for Gentoo are using MLMMJ for example. MLMMJ 1.2.18.0 has just been release upstream, few days ago (on the 29 of may). It s with a great pleasure that I packaged (and uploaded to SID) the latest version, and wrote the changelog, where I wrote that 6 Debian specific patches (out of 8) could be removed! One of the funny changes are the renaming of the mlmmj-recieve into mlmmj-receive as it always should have been (with a symlink to keep backward compatibility), and the removal of the .sh extension to the mlmmj-make-ml (which was patched in Debian to be kept policy compliant: that s one of the removed patches ). Even if I did a few quick functional tests before uploading, I d be happy to get some feedback before Wheezy is out, so please test and eventually bug-report!

Sorina Sandu: Initial commit

Hello!

I'm Sorina and this summer I will be working on Debian Installer network setup during Google Summer of Code. I am a second year bachelor's student at Politehnica University of Bucharest, I am studying Computer Science. I very much enjoy what I am doing even if sometimes I feel that the amount of stuff I want/have to learn/do/work on is infinite and I don't really know where to start.

I am at the beginning of the project, I believe now I am accustomed to the way things work and I am waiting forward to finishing the exam period so that I could really start working.

The hardest part for me in what I've done so far is testing what I am doing. It took me a while to figure out at least a part of how the installer functions and to solve the problems that occurred while building. Luckily, there is documentation and internet and even more important, I had help and support from my mentors.

So far I have only written two patches for netcfg, for bugs #610752 and #636211. For the first one I had a version for a while now, but I was unable to test it (unfortunately, it's likely that you will hear this a lot in the future). The idea was to make the link detection timeout preseedable and for this I have added two new templates, asking the user to provide a value for that matter.

The second one involved a missing "Go Back" button. This seemed to be the result of not letting debconf know that the program is capable of handling the user pressing a back button. That could be solved using deb_capb.

As I said before, testing was my main problem. Eventually, I built the image, wrote it to an USB and rebooted my laptop. This served its purpose, since I had access to anything I needed such as the wireless interface and physical access to the network cable. Running it through a virtual machine did not help much in this case, because I was unable to provide those.

Another thing I considered to be hard was blog posting, but now that I have just finished my first post it seems that I overreacted.

That's all for now, have a nice day!

Michal Čihař: Four counferences in Prague this October

I'm participating on organizing this year's openSUSE conference and as we've finally finalized place and time and CFP is open, it's time to share some information. This will be special - there is not a single conference, but four of them are sharing same space and the motto - Bootstrapping awesome!!!. The 4th openSUSE conference will be held together with LinuxDays, Czech conference following tradition of canceled LinuxExpo, what should be the biggest free software event in Czech republic. In addition there will be also 12th SUSE Labs conference (so you can meet quite a lot of kernel hackers and other strange guys) and first Gentoo mini summit (the website is empty so far). All that will happen on weekend from 20th October to 21st October, the SUSE conferences will then continue for 22nd and 23rd October. I believe it will be great mixture of conferences and I hope to meet lot of people there.

Filed under: English Gentoo Suse 0 comments Flattr this!

Lucas Nussbaum: Debian archive rebuilds on Amazon Web Services

I like to think that archive rebuilds play an important role in Debian Quality Assurance and Release Management efforts. By trying to rebuild every Debian package from source, one can identify packages that do not build anymore due to changes in other packages (compilers, interpreters, libraries, ). It is also a good way to stress-test all packages that are involved in building other packages. Since 2007, I had been running Debian archive rebuilds on the Grid 5000 testbed, a research infrastructure for performing experiments on distributed systems HPC/Grid/Cloud/P2P. I filed more than 6000 release-critical bugs in the process. Late last year, Amazon kindly offered us a grant to allow us to run such QA tests on Amazon Web Services. With S bastien Badia, we ported the rebuild infrastructure to AWS (scripts), and several rebuilds have already been carried out on AWS. On the technical level, 50 to 100 EC2 spot instances are started, and then controlled from a master instance using SSH. On build instances, a classic sbuild setup is used. Logs are retrieved to the master node after rebuilds, and build instances are simply shut down when there are no more tasks to process. Several tasks are processed simultaneously on each instance, and when they fail, they are retried again with no other concurrent build on the same instance, to eliminate random failures caused by load or timing issues. All the scripts are designed to support other kind of QA tests, not just rebuilds. Moving to Amazon Web Services will facilitate sharing the human workload of doing those tests. It is now possible for developers interested in custom tests to do them themselves (hint hint).

Next.

Previous.