Search Results: "nate"

4 April 2011

Julien Danjou: Using advanced filter with mod_authnz_ldap

As you may know, Apache's mod_authzn_ldap allows to authenticate users in Apache HTTP server using an LDAP server. Unfortunately, it has a little implementation flaw. The filter used to authenticate the user is built by abusing the RFC 2255 which specifies the LDAP URL format. This format has an "attribute" field which is normally used to specify which attributes should be returned. But mod_authzn_ldap uses this attribute to compare with the username given by the client. That means that you have to have an attribute in your LDAP entries which matches the username, and you have to use it in the "attribute" part of the URL to get things working. Therefore, I wrote a patch to add a format string in the LDAP URL in order to user the provided username in the filter, and ignore the attribute part of the URL, which has no use in such a context anyway. The bug has been opened in ASF Bugzilla and has number #51005, with the patch. The patch is backward compatible with the current configuration format, which is not the best choice in theory, but probably the more pragmatic. I've no clue on the typical delay for patches in clusion in Apache HTTP server, so let's just wait'n see. Flattr this

3 April 2011

Petter Reinholdtsen: A Norwegian FixMyStreet have kept me busy the last few weeks

Here is a small update for my English readers. Most of my blog posts have been in Norwegian the last few weeks, so here is a short update in English. The kids still keep me too busy to get much free software work done, but I did manage to organise a project to get a Norwegian port of the British service FixMyStreet up and running, and it has been running for a month now. The entire project has been organised by me and two others. Around Christmas we gathered sponsors to fund the development work. In January I drafted a contract with mySociety on what to develop, and in February the development took place. Most of it involved converting the source to use GPS coordinates instead of British easting/northing, and the resulting code should be a lot easier to get running in any country by now. The Norwegian FiksGataMi is using OpenStreetmap as the map source and the source for administrative borders in Norway, and support for this had to be added/fixed. The Norwegian version went live March 3th, and we spent the weekend polishing the system before we announced it March 7th. The system is running on a KVM instance of Debian/Squeeze, and has seen almost 3000 problem reports in a few weeks. Soon we hope to announce the Android and iPhone versions making it even easier to report problems with the public infrastructure. Perhaps something to consider for those of you in countries without such service?

Raphaël Hertzog: March 2011 wrap up

Since I m soliciting donations to support my Debian work, the least I can do is explain what I do. You can thus expect to see an article like this one every month. Multi-Arch work I updated the code to use another layout for the control files stored in /var/lib/dpkg/info/. Instead of using a sub-directory per architecture (arch/package.type), we decided to use package:arch.type but only for packages which are Multi-Arch: same. dpkg is taking care to rename the files the first time it is executed with write rights and then updates /var/lib/dpkg/info/format to remember that the upgrade has been done and that we can rely on the new structure. I filed a few bugs on packages that are improperly accessing those internal files instead of using the appropriate dpkg-query interface. I sent a heads-up mail on -devel to make other people aware of those problems in the hope to discover most of them as early as possible. After that, the work stalled because Guillem went away for 2 weeks and thus stopped his review of my work. I hope he will quickly resume the review and that we will get something final this month. With the arrival of dpkg 1.16.0, it s now possible to start converting libraries to multi-arch even if full multi-arch support has not yet landed in dpkg proper. See http://wiki.debian.org/Multiarch/Bootstrapping for the detailed plan. If you re curious about Multi-Arch, you might want to read this article of Steve Langasek as well. Bug triage for dpkg in launchpad At the start of the month, there was close to 500 bugs reported against the dpkg package in Launchpad. Unfortunately most of it is noise many of the reported bugs are misfiled, they show an upgrade problem of a random package and that upgrade problem confuses update-manager which tries to configure an already configured package. This generates a second error that apport attributes to dpkg and the resulting bug report is thus filed on dpkg. There are literally hundreds of those that have to be reclassified. Michael Vogt and Brian Murray did some triaging, and I also spend quite some hours on this task. It s a bit frustrating as I tend to mark many reports Incomplete because there s no way they can be acted upon and many of them are so old that the reporter is unlikely to be able to provide supplementary information. But in the middle of this noise, there are some useful bug reports, like LP#739179 which enabled me to fix a regression even before it reached Debian Unstable (because Ubuntu runs a snapshot of dpkg with multiarch support). I subscribed to the Launchpad bugs for dpkg via the Debian Package Tracking System (thanks to the derivatives-bugs keyword) and will try to keep up with the incoming reports. Misc dpkg work The ftpmasters came up with a request for a new field (see 619131) in source packages. After a quick discussion and a round of review on debian-policy@l.d.o, I implemented the new Package-List field. This should allow the ftpmasters to save some time in NEW processing, but we deferred the change for the next dpkg version (1.16.1) to ponder a bit more on the design of the field. I also fixed a bunch of bugs (#619541, #605719, #598922, #616096) and merged a patch of Mark Hymers to recognize the new Built-Using field. Developers-reference work The review process for changes to the developers-reference is not working as it should. And I suffered from it while trying to integrate the patch I wrote for the Developer duties chapter (see #548867). We purposely changed the maintainer field from debian-doc to debian-policy in the hope to have more reviews of suggested changes and to seek some sort of consensus before committing anything. But we don t get more reviews and deciding to commit a patch is now even harder than it was (except for trivial stuff where personal opinions can t interfere). In my case, I only got the feedback of Charles Plessy which was very mixed to say the least. I tried to improve my patch based on what he expressed but I also clearly disagreed with some of his assertions and was convinced that my wording was in line with the dominant point of view within Debian. We tried to involve the release team in the discussion because most of what I documented was about helping making stable release happen, but nobody of the team answered. Instead of letting the situation (and my patch) rot, I solicited feedback from the DPL and from another developers-reference editor to see whether my patch was an improvement or not. After some more time, I went ahead and committed it. It was not pleasant for anyone. I don t know how we can improve this. Contrary to the policy, the developers-reference is a document that is not normative, I believe the result is better when we put some soul into it. But it s a real challenge when you seek a consensus and that the interest in reviewing changes is so low. DVD shop listed on debian.org In February, I launched a DVD shop whose benefits are used to fund my Debian work. Shortly after the launch I used the official form to be added to the official listing of Debian CD vendors and offered a few suggestions to deal with vendors who are selling unofficial images (with firmware in my case). A few weeks later, I got no answers: neither for my request nor for my suggestions, I mailed the cdvendors@debian.org team directly asking for a status update and quickly got an answer suggesting that Simon Paillard usually does the work and can t process the backlog due to some injury. At this point no concerns had been raised about adding me to the list. To save some time and some work for the team, I added myself to the list since I had commit rights and I informed them that I did it, so that they can review it. Shortly after I did that, Martin Zobel Helas objected to my addition. I cleared some misunderstandings but the discussion also lead to some changes to please everybody: the listing now indicates that some images are unofficial and I have prepared a special landing page for people coming from the Debian website through this listing. Debian column on OMG! Ubuntu I have always been a firm believer that it s important for Debian to reach out to the widest public with its message of freedom. Thus when Benjamin Humphrey contacted the debian-publicity team to find volunteers to write a Debian column on OMG! Ubuntu, I immediately jumped in. I wrote 4 articles over there. The tone is very different from my articles on my blog and I like that duality. Check out Debian is dying! Oh my word!, Debian or Ubuntu, which is the best place to contribute?, Are you contributing your share? and Ubuntu s CTO reveals DEX: an effort to close the gap with Debian. It s a great win-win situation, OMG! Ubuntu benefits from my articles, Debian s values are relayed further, and OMG! Ubuntu s large audience also helps me develop my own blog. Work on my book I had lots of paperwork to do this month (annual accounting stuff for my company) and I did not have as much time as I hoped for my book. Still I have a updated a few more chapters of my French book and I certainly hope to complete the update during April. This means that the work on the English translation could start in may. Work on my blog Just like for my book, it has been relatively difficult for me to cope with my policy of two articles every week. But I still managed to get quite some good stuff out. I interviewed Christian Perrier (Debian s translation coordinator) and also Bdale Garbee (chair of Debian s technical committee). I finished my series of Debian Cleanup Tips with 2 supplementary articles: The removal of firmware is causing troubles to quite some users so I wrote an article explaining how to deal with the problem. A regular reader also asked me to write an article about Jigdo, I executed myself because it was a good idea and that he has been very nice with me: Download ISO images of Debian CD/DVD at light speed with Jigdo. Last but not least, I shared my package maintainer pledge which inspired my developers-reference patch (see discussion above). Thanks Many thanks to all the people who showed their appreciation of my work. The 324.37 EUR that you gave me in February represented 2 days and a half of my time that I have spent working on the above projects. See you next month for a new summary of my activities.

2 comments Liked this article? Click here. My blog is Flattr-enabled.

1 April 2011

Josselin Mouette: GNOME.Asia 2011 hackfest

For the whole week, I ve been in Bangalore for the GNOME.Asia 2011 hackfest. I ve been delegated by Stefano to represent Debian here, and my employer EDF has agreed to cover for travel costs since they are very interested in first-hand information the future of the Linux desktop and sharing our work on scientific computing. It s been a really exciting week; I ve spent quite some time packaging missing pieces of GNOME 3.0 (well, the release candidate versions of course) in experimental, together with Fred Peters. I think it s reaching a usable state now, so we ll probably soon provide metapackages to make it easily installable. The latest developments of the Shell make it a very exciting piece of software, with a strong focus on usability. Many things were written about it, but in the end my main criticism would be that it lacks some functionality - for example, the combined clock/weather/locations applet will be greatly missed. The good news is that it is extremely customizable, and with all the libraries being made accessible through GObject introspection, there are many features that are accessible from it. If you know how to write JavaScript, now is the time to write your favorite extension. On the good news front, Vincent Untz also spent a lot of time improving the so-called legacy mode , which is more and more starting to look like the Shell without special effects, and with all the features from gnome-panel 2.x that are still here. We will try in Debian to cover all uses cases that there were for GNOME 2 with GNOME 3 technology, so that panel lovers are not left behind. I ve also proposed an update to the dh_gsettings proposal, which will provide the same functionality as dh_gconf and allow to easily set distribution-specific overrides. It is still missing a way to set mandatory settings, which might come as a problem for some corporate users, but this is planned for a future version of GSettings. Today, we re having a business track where I and representatives of other companies (Oracle, Lanedo, Dexxa) are sharing experiences about making money with free software. Unfortunately the local organizers didn t manage to gather many people, despite our being in a city with an incredible number of IT industries. Tomorrow, the public conference starts, and this should be the opposite: we re expecting around 1000 people, which is a great achievement for a free software conference. For an unrelated topic, being around so many GNOME hackers has some interesting side effects; I ve been added to Planet GNOME. So, hey, hello Planet GNOME readers!

Stefano Zacchiroli: formerly known as

pushing cross-distro collaboration to the next level In case you've missed the announcement, today it's a memorable day for Free Software. Please join the following distributions in welcoming Canterbury, the next generation community distribution: canterbury splash page
Kudos to the, yet unnamed, representatives of the above derivatives who have come up with and coordinated this initiative.

31 March 2011

Jordi Mallach: A tale of Trist nia and its Quadrennial Royal Ball

In one of the corners of what is now know as Europa, there was a rich, prosperous and beautiful kingdom known as Trist nia. In the past, not that long ago, it had been a number of smaller kingdoms and caliphates, all with their own cultures, religions and ways of life. Wars, and series of marriages of convenience eventually configured what ended up being the united kingdom of Trist nia. Throughout the years, some of the unified cultures grew and flourished, while others struggled to survive in their ever-shrinking areas of influence. A required introduction Sometimes, the minor cultures would suffer due to oppression coming from the delegates of the King, who would ban any expression of these cultures, as they were seen as a potential threat to the kingdom's stability and unity. For example, just a few decades before the main subject of this tale, the predecessor of the incumbent King took power by force, after crushing everyone who opposed his uprise during a bloody and hard civil war. His reign was ruthless and he imposed draconian laws uppon his people: usage and teaching of the minor languages was banned, and everyone was forced to use the language of the Centr lia region, in public or private. After four decades, the majority of the Tristanian people were sick enough of the situation to consider standing against their fear of the regime and demand freedom, but repression prevailed until the old general died. His place was taken by the King's grandson even if the people had expressed, just before the Great War, that they had had enough kings and demanded a ruler they could choose directly. Of course, the new King seemed a lot nicer than who they had been suffering for ages, so when asked if they accepted the new situation, an overwhelming majority said yes . However, there was a region, Verd lia, where the majority said no . Things were actually more complicated. Verdalians formed a traditional, proud society, and while the years of oppression had undoubtedly weakened it, they had managed to preserve their very unique culture, language and traditions healthy. The Verdal language was really weird to the ears of Centr lians and even other minor cultures of the Kingdom, and erudites struggled to find its real origins, not being able to reach plausible conclusions. Verdanians, as we already know, were a traditional society, living in a land of deep and poorly connected valleys. Little they knew or cared about the complicated matters of Centr lia and other regions. What made them happy was to take care their sheep and cows, keep a good fire in their living room and, every now and then, enjoy one of their log cutting contests. The impositions of the former dictator were too much for them, and some of them started sabotaging, assaulting and killing some of the dictator's soldiers, agents and officers. This was a huge risk at the time; getting caught meant death penalty for sure, and at first, even people from other regions were in favour of these actions. However, this popular support greatly diminished when the new King took the throne, as these minority continued with the killings, while most of the people saw it was no longer justified. The Royal Ball One of the very first measures the young King introduced was to organise the Royal Ball of Trist nia , a major event through which the people of the different regions would be able to elect their delegates to the Crown. Every four years, a Great Ball contest would happen in Centr lia, and the winners would be able to decide by their own on some of the matters that affected their region. Verdanians would send a few teams of dancers, each of which came from different towns or areas. Some Verdanian teams were happy about the King and the new political situation, but other teams weren't so much. And some others, while being simple non-violent dancers, were known to be supporters of the violent minority who kept on harassing, assaulting and even killing in their struggle for freedom of Verd nia . The Verdanian groups aligned with the different culture of Verd nia (including those who were said to support the violent) tended to get a lot more points in the dancing contest, and a majority of the elected delegates were appointed by them, making it easier to pass laws and edicts that favoured protection of their ways, traditions and language. No matter how hard they tried, the dancing groups closer to Centr lia kept losing to the majority. After many years of dance contests, these groups used their closeness to the King's court to pass the Ball Law of Tristanian, that would ban any dancing group which didn't condemn the assaults and killings that kept happening in Verd nia. The unsurprising result was that, with less dancing groups participating in the following Royal Ball, the Verdanian majority was broken and new delegates, friendly of the Centralian officers, were elected. Many people who had been in favour of assaults and killings began to question this strategy, and this political movement's unity started to break. In the end, the dancers decided to part ways with the violent; they wanted to dance in the next ball, and to do so, they wrote a letter to the King, in which they explicitly expressed their rejection of violent ways, and their embracing of dancing as the only means to drive their political agenda. An objective reading of the new Ball Law clearly showed that this was enough: the text only said the requisite for a dancing group was to disavow all kinds of violence. This wasn't really expected in Centr lia, so they started to add new requirements in an attempt to keep this group from the contest: their decisive majority in Verd nia was at stake. The Royal Ball was nearing and registrations for the contest would soon close. The Centralian government first argued that the dancing group should reject the violence coming from the Verdanian extremists in particular. The dancers did it. Then they argued that the dancers were the same people who had been supporting violence in Verd nia for years, and obviously their violence rejection statement was a lie. The dancers struggled to find new dancers who had not been involved in past dances. But it was not enough. They then claimed that this dance group should be quarantined for four years, until they could prove they really were serious about their new non-violent ideas. The dance group made a plea to the Trist nian Supreme Counsel, a group of sixteen experts in law of the Kingdom, and argued that all of these draconian requirements were not part of the law that was being enforced by the King. Their appeal to the elder counselors was in vain, though. They ruled this dancing group was as criminal as the violent minority they had once supported, and should by no means take part in the Royal Ball. As a last, desperate measure, the dancing group reached an agreement with other Verdanian dancers to join forces. They would adopt a new name and new dancing costume colours. Many feared this would only end up in the ban of the other dancing group. Unfortunately, the end of this story has not been written yet, but it will be completed very soon. Only time will tell if things continue being very sad and unfair in Trist nia, or if the dance contest will once again be impartial, with legitimate results.

28 March 2011

Rémi Vanicat: Booting rarely used os

For some time, I was looking for a way to change grub default entry for only one boot. I want to tell it that next time it should boot the entry 0, and the time after that, use the usual default (the entry 1). Founding documention for this is not as easy at it should, and the information on the debian wiki wasn't clear enough for me to understand. So I edit the wiki page to make clear that grub-reboot also available for those using grub2, and that it will only configure the boot, but not reboot the computer itself. As each time I wrote some text in English, I'm not sure my version correctly transmit my meaning. By the way, I'm changing my blog. The old one contain a post on my search of a solution for this. Thanks to stbuehler and Jochen for the indication leading to the correct solution for this. Thanks also to Vadim Solomin for a patch to make pm-hibernate notice the configured method for shutting-down the computer.

Raphaël Hertzog: People behind Debian: Bdale Garbee, chair of the technical committee

Bdale is a long-time Free Software believer, he has been contributing even before Debian existed in the prehistoric era of free software. :-) Anyone who went to a big Free Software conference has seen one of his colorful t-shirts. Or maybe you have heard the story where he got his beard shaved by Linus Torvalds to raise funds to protect the Tasmanian Devil. More seriously Bdale has played and continue to play a number of important roles in the Debian community. He also represents one of the biggest corporate sponsors (both for DebConf and for the servers that Debian owns): Hewlett Packard. My questions are in bold, the rest is by Bdale. Who are you? I made my first personal contribution of source code to what we now call Free Software in 1979. I started with HP in 1986 and for nearly a decade have served the company as Chief Technologist for Open Source & Linux. I am president of Software in the Public Interest, which is the umbrella organization providing legal and financial existence for Debian in the USA. I also represent users, developers, and Debian interests on a number of boards including at the Linux Foundation and the Freedombox Foundation. I m happily married with two children. Many people in Debian have met some or all of my family. They all joined me for Debconf in Edinburgh, and my daughter Elizabeth also attended in Caceres and New York. I joined Debian in 1994. I ve been responsible for a number of packages essential to our base system continuously since that time. But I ve also contributed to the project in many other ways over the years. I ran the first server that was fully dedicated to Debian. Ideas of mine influenced the development of project infrastructure, from the early design of our mirror network to structuring the archive around a package pool . I started or made significant early contributions to 5 ports of Debian to non-i386 architectures. I served as Debian Project Leader (DPL) in 2002-2003, was acting Secretary for a while, and have served on the Technical Committee for a number of years. Over the years, I ve also had some interesting hobbies. I helped design, build, and program pieces of various amateur radio satellites. I enjoy making physical things, and have many tools for working in wood and metals. My son and I are very active in the world of high power model rockets. And with my partner (and fellow Debian developer!) Keith Packard I m now running a small business making and selling open hardware and open source avionics for hobby rockets. You can read more about that at http://altusmetrum.org. You re the chair of the Debian technical committee. Can you quickly explain the role of this committee? I think many people assume the Technical Committee has a larger role in Debian than it really does. Section 6 of Debian s constitution defines the official role of the Technical Committee. Most importantly, the committee exists as a last resort place to resolve technical conflicts between Debian developers that they are unable to resolve by themselves. Most of the power in Debian is left in the hands of individual developers, who are usually able to collaborate with each other to make good technical decisions. So the Technical Committee s resolution process has only rarely been needed, which I think is a very good thing. From my point of view, the technical committee is not working. In many cases, the committee does not take any (timely) decision and just waits until the underlying situation has evolved to a point where the intervention of the committee is no longer needed. Do you agree with this and how can you explain it? I think it s very important for all of us to remember that everyone working on Debian does so voluntarily, and people who volunteer their time generally deserve a measure of respect and appreciation for their efforts. No issue is brought to the Technical Committee unless resolving it is expected to be really difficult, or at least contentious. And often, the issues brought to the committee have been as much or more about personality than technology. That makes some of them really hard to solve. So I do not agree that the technical committee is not working. It seems to me that the decisions that bog down and take a long time are the ones where arguments start out or become emotional instead of technical. In this context, if committee members can help lead public and private discussions in a way that causes a situation to evolve to the point where a decision is no longer needed, that may be healthier for the project in the long term than a quick vote that satisfies some contributors at the expense of others. The last important change that was made to try to revive the committee was the addition of two new members (Don Armstrong and Russ Allbery). Is there anything else that could be tried? The biggest improvement I could personally wish for is something people sending issues to the committee can help with. As the ultimate technical decision making body for a project whose output is mostly software, the more a request can be put in terms of a decision about source code, the easier it will be for us to make a decision. That won t always be possible, but when we re forced to try and dream up alternatives and then figure out whether anyone would actually be willing to write the code to implement those alternatives, the process takes a lot longer than choosing between competing patch sets or deciding whether a patch should be included. Besides your role in the technical committee, you have held the role of mediator/facilitator/advisor on numerous occasions. Because you re an old wise bearded guy who travels a lot and knows many Debian contributors I would like to thank you for all this work that few people notice. Are there been times where this has been a real burden for you? Thank you for mentioning this. I ve put a lot of my heart into Debian over the years, largely because it s a project and a community that continues to amaze and inspire me. I feel fortunate to have been able to meet and work on Debian with so many outstanding people from around the world. Many are now my friends, with all the silly and serious things being a friend implies. I ve been asked for and have given advice many times. I ve helped celebrate birthdays, marriages, new jobs, and the arrival of children. Sadly, I have also found myself having to try and find the right words to mark the loss of some of these friends The only time any of this feels like a burden is when there s some important problem that many people care about, that I m working behind the scenes to help fix, but can t really talk about publicly without causing more harm than good. It s distressing to have people think you don t care or aren t helping, when really you re doing everything you possibly can just not in a publicly visible way. Of course I understand that this is an impossible situation. If you can t see what s happening, there s no way to know if something is happening or not. That s why I advocate doing as much as possible in Debian, and SPI, and everywhere else I contribute in as open a way as possible. You have been Debian Project Leader and you promoted the vision of Debian as the Universal Operating System. What does universal mean for you? The biggest thing to me at the time was the idea that Debian could be anything. Those who chose to work on Debian would ultimately determine what Debian became. I also wanted to make sure we thought about as broad a set of potential users and collaborators as possible. But this vision provided a framework for pursuing a whole range of worthwhile increases in Debian s scope of utility, some of which I articulated in my DPL platforms, some others put forward. Internationalization, porting to more supported architectures, our inclusive and evolving approaches to accepting new developers and new packages, and so forth. I think this vision has served us well, and it pleases me that it has stayed a part of our collective thinking for so long. We re again in Debian s electoral period, what do you think of the work done by the current DPL? I m very happy with what I ve observed of Stefano s activities during his first year as DPL. He has an obvious enthusiasm for Debian, communicates well both in one to one interactions and in front of a crowd, and I think represents Debian very well. It is interesting that he s running unopposed for re-election this year. I choose to interpret that as evidence he s doing a good job, the project is running well, and nobody feels the need to try and take the job away From him. I m glad he s willing to continue in this role for another year. What s the most important thing that Debian should achieve in the wheezy timeframe? I don t yet have a very crisp personal wish-list for wheezy. But I would certainly like to see multiarch support finally completed! I m also very interested to see what comes from the CUT work. You have been an early supporter of multiarch , a project to allow easy installation of foreign architecture packages. It s on good track for Wheezy. Do you think it s an important milestone? My original motivation for requesting multiarch support was to enable support for 32-bit x86 binaries on ia64 Itanium systems, in the time leading up to the sarge release. I ended up creating the ia32-libs package, which I m not proud of. The emergence of 64-bit extensions to x86 (the amd64 architecture) made this a much broader issue. Today, I run a 64 bit kernel and a 32 bit user space on my notebook. There are problems with just moving entirely to 64 bit but I would like to be able to run some applications that work with large data sets in full 64 bit mode!
Thank you to Bdale Garbee for the time spent answering my questions. I hope you enjoyed reading his answers as I did. Subscribe to my newsletter to get my monthly summary of the Debian/Ubuntu news and to not miss further interviews. You can also follow along on Identi.ca, Twitter and Facebook.

4 comments Liked this article? Click here. My blog is Flattr-enabled.

26 March 2011

Mark Brown: ASoC conference 2011 Edinburgh, 4-5th May

There will be an ASoC conference in Edinburgh 4th-5th May this year, held in the Scotch Malt Whisky Society in Edinburgh. Full details are in the announcement if you ve got an interest in embedded audio on Linux I recommend you attend, there s a lot of development going on in this area right now and it promises to be a great opportunity to coordinate development plans and make sure things work well for everyone. Having met quite a few of the attendees who ve signed up already I can also say that it should be an opportunity to meet some really excellent people.

23 March 2011

Kapil Paranjape: John Willard Milnor: Abel Prize 2011

The Abel prize for 2011 has been awarded to John Willard Milnor.[1] In response to a query from Dr. R. Ramachandran who writes about Science and Science Policy for The Hindu'' newspaper group, I wrote the following short piece which others may find interesting as well. Milnor is well-known for his work on topology and geometry. He has also made significant contributions to algebra and even number theory. In addition, he has written a number of books which are loved by graduate students in Mathematics all over the world. One of the striking early results of Milnor was the example he gave of a seven dimensional space which is topologically a sphere but its geometric (differentiable) structure is different. This was the first example of an "exotic sphere". A nice way to state his main result (due to Ajit Sanzgiri) is that "Groups of homotopy spheres are homotopy groups of spheres". Milnor received the Fields' Medal in 1962. In addition, the work of a number of later Fields' medallists such as Donaldson, Thurston, Mori and Voevodsky can be seen as having roots in the work of Milnor.
A more personal perspective The first time I came across the name Milnor was when I heard that the only dimensions in which one can do algebra with division is 1, 2, 4 and 8; I was told that an "easy" proof was based on Characteristic Classes on which Milnor had written a nice book. In later years, I read a number of his other books like Topology from a differentiable viewpoint'', Morse theory'', Isolated singularities of complex hypersurfaces'' and Algebraic K-theory''. These books not only explained the results and definitions, but laid the foundations of my geometric intuition --- the same is probably true for many others in my generation. When I joined TIFR, Raghunathan was full of praise for the work of Milnor and how his deep ideas on differential topology would "lead somewhere". One of the first lectures in our graduate seminar was by Ajit Sanzgiri on Milnor's paper on exotic spheres --- the title of the talk was Groups of homotopy spheres are homotopy groups of spheres''. When Srinivas taught me (algebraic) K-theory, the only available reference text was Milnor's book (Milnor K-theory forms a crucial component of Voevodsky's early 21st century work that won him his Fields Medal!); since then Srinivas has written a more modern and comprehensive book on the topic. Later, when A. J. Parameswaran started work on his Ph.D. under the guidance of Srinivas, AJP and I read Milnor's book on isolated singularities together --- as a prelude to the sequel to Milnor's book (by Looijenga). Much of the modern work on the algebraic theory of singularities (which forms a crucial component of Mori's Fields medal winning work on terminal three dimensional singularities) starts with the notion of "Milnor number" and "Milnor fibrations". If the impact of a mathematician is to be measured not only by his own fantastic results but the great results of others that grow out of his work, then Milnor is certainly one of the greats of the latter half of the twentieth century. Much of the topology and geometry that I have used in my work relies on simpler versions of Milnor's results that were proved by his predecessors. So (unfortunately!) I cannot quote a result that I have proved which actually uses a theorem of Milnor's.
Groups of homotopy spheres are homotopy spheres'' I think the idea of joining two spaces by a tube to make a new space (called connected sum) is due to Whitney or Kervaire. This leads the "algebra of spaces" or to give it its mathematical name "the cobordism group". Milnor showed us how to perform calculations with this group. Another group studied by topologists combines the different ways in which a sphere of one dimension wraps around'' a sphere of a (possibly) different dimension; this is called a homotopy group of the second sphere. Milnor's brilliant idea was to show that in certain cases, the two groups obtained are the same. Said pithily, "Groups of homotopy spheres (exotic spheres)" are the same as "homotopy groups of spheres".
Some remarks A number of people have remarked that what we need in order for institutes like IISER to grow are teaching researchers''. The system of universities in the USA has thrown up a few shining examples --- Richard Feynman in Physics and John Milnor in Mathematics are two names that immediately spring to mind.

[1]The last time I wrote about the Abel Prize was to contribute to a guest post to Rahul Basu's science blog: The Far Side. Unfortunately, Rahul is no longer among us. This post is dedicated to the memory of Rahul Basu.

Gerfried Fuchs: LXC and NAT on notebook

Yesterday I was hinted towards lxc when I wondered what happened to openvz in unstable (which unfortunately isn't documented at all in the kernel changelogs, but that's a different story). So I started off taking a look. From a bit of experimenting around with it I consider it something that I want to play more with, and I want to share the problems I stumbled upon with you so that you don't have to figure them out on your own. First of all, LXC uses the cgroup kernel facility for resource management. The according file system isn't mounted by default, and LXC doesn't care for where it is mounted, it just needs to be. It seems like /sys/fs/cgroup seems to be the proper place (see 601757), so add the line cgroup /sys/fs/cgroup cgroup defaults 0 0 to your /etc/fstab file and sudo mount cgroup it. Next, it seems like bridging is the defacto standard for networking with lxc, but given that I want to use it on my notebook while being mobile I can't bind the bridge to any specific interface. To make this happen, one needs the bridge-utils package installed, and secondly, this is the path that I chose. I've added to /etc/network/interfaces this snippet:
auto br0
iface br0 inet static
    bridge_maxwait 0
    bridge_ports dummy0
    address 10.80.80.1
    netmask 255.255.255.0
This will bring up the bridge and act as gateway. For the running system, call sudo ifup br0. To make the host universally being able to work as gateway, of course ip_forward needs to be enabled. For this I added the line net.ipv4.ip_forward=1 to /etc/sysctl.d/local.conf (and for the running system, echo 1 into /proc/sys/net/ipv4/ip_forward). As I am using ferm for configuring the firewall on my notebook I have to add some parts into its configuration. This is the raw part that needs to get added, mix it into your existing configuration:
table filter  
    chain INPUT  
        # allow DNS queries from LXContainers
        proto (udp tcp) dport domain source 10.80.80.0/24 ACCEPT;
     
    chain FORWARD  
        # allow LXContainers into the net
        source 10.80.80.0/24 ACCEPT;
     
 
table nat  
    chain POSTROUTING  
        # NAT LXContainers
        source 10.80.80.0/24 MASQUERADE;
     
 
For DNS I installed dnsmasq so that I won't have to touch the /etc/resolv.conf inside the containers whenever I switch networks. So far for the host part, now to the actual containers. There is the /usr/lib/lxc/templates/lxc-debian helper script which uses debootstrap to create you a lenny chroot at least in the squeeze package this is hardwired, likewise with using cdn.debian.net. Copy the script and edit it to your likes if you feel like it. From what I understood it expects you to store the containers below /var/lib/lxc, I haven't yet tested for different places. So this was my commandline for that:
sudo /usr/lib/lxc/templates/lxc-debian -p /var/lib/lxc/vm0 A while later you'll end up below that directory with two entries: The config file and the rootfs subdirectory which is actually the bootstrapped distribution part. Now comes the configuration of the container. Open the config file with your favorite editor and add the following lines to the end:
lxc.utsname = vm0
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = br0
# lxc.network.name = eth0
lxc.network.hwaddr = 00:FF:80:80:80:80
lxc.network.ipv4 = 10.80.80.80/24
The network.name part is commented out, it defaults to that name internally; you though can change it to whatever you prefer. Caution, even though this is the documented approach, it does not work for Debian containers. It will always try to get its IP address through dhcp, lxc.network.ipv4 has no meaning for us. We need to change inside the rootfs the file etc/network/interfaces to read like this instead:
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet static
   address 10.80.80.80
   netmask 255.255.255.0
   gateway 10.80.80.1
I suggest to keep the config and the interfaces file aligned with respect to the ipv4 setting so if this gets fixed upstream you won't stumble into any surprises. Also like mentioned before, we need to change the nameserver entries inside the rootfs file etc/resolv.conf to read nameserver 10.80.80.1. Now it's time to start it up and log in! sudo lxc-start -n vm0 -d will start the container in the background, and sudo lxc-console -n vm0 will give you the login to the container. The default password for the root user is root, obviously you want to change that before you install any networking services into the container like ssh-server. In case you want to quit from that console notice the message upon starting it, it's bound to <Ctrl+a q>. One more issue that I had: The default route wasn't set. I had to manually call ip r a default via 10.80.80.1 dev eth0 to be able to use the network inside the container. It seems to be related to that netbase isn't installed by default. If you install it the default route will be set upon starting the container automatically. This should get you started, there is of course more to explore and experiment with. Actually it is also suggested to create a tarball from your vm0 after you did the basic setup and installed the basic components you want to have around so you won't have to bootstrap over and over again. Do this after you have shut down the container, either through a halt from a container shell or through sudo lxc-stop -n vm0. The tarball can then get extracted to a different directory and just needs minor tweaks in the config and rootfs/etc/network/interfaces file to not create any clash with other containers (lxc.rootfs, lxc.mount.entry, lxc.utsname, lxc.network.hwaddr and ipv4 address). About limiting the containers, you can do it dynamically through the cgroup file system, and set it permanently through the config file. See man lxc.conf about these settings, amongst others. Enjoy, use, experiment. With sudo lxc-checkconfig you will see what your kernel actually supports for your LXCs. You will most probably notice the missing for the memory controller, this is tracked in the Debian bug report 534964.

/debian permanent link Comments: 6 Flattr this

22 March 2011

Gunnar Wolf: Lets go to Nicaragua, 2012!

Ok, so finally it is official! We just had the DebConf 12 decision meeting. We saw two great proposals, from the cities of Belo Horizonte, Brazil and Managua, Nicaragua. If you are curious on the decision process: We held it over two IRC channels The moderated #debconf-team channel, where only the five members of the decision committee (Marga Manterola, Andrew McMillan, Jeremiah Foster, Holger Levsen, Moray Allan) and two members from each of the bids (Marco T lio Gontijo e Silva and Rafael Cunha de Almeida from Brazil; Leonardo G mez and Eduardo Rosales from Nicaragua) had voices, and the open #dc12-discuss channel where we had an open discussion. Of course, you can get the full conversation logs in those links. I have to thank and congratulate the Brazilian team as they did a great work... The decision was very tight. It was so tight, in fact, that towards the end of the winning all of the committee members were too shy to state the results - so I kidnapped the process by announcing the winner ;-) (I hope that does not cast a shadow of illegitimacy over it) And, very much worth noting, both teams were also very professional: In previous years, we have seen such decisions degenerate into personal attacks and very ugly situations. That has always been painful and unfortunate. And although the Brazilians will not be able to go celebrate tonight, the decision was received with civility, knowing it was a decision among equals, and a decision well carried out. Well, that's it I am very much looking forward for that peculiar two weeks when the whole Debian family meets, this year to be held in Banja Luka, Bosnia-Herzegovina, and I am very eager towards meeting in 2012 in Managua, Nicaragua! Yay!

Axel Beckert: Different Flavours of Planet Commandline

Since there were quite some requests for a Planet Commandline feed without the microblogging feeds included, I splitted Planet Commandline into different flavours. I m quite happy with that solution, because I must admit that the amount of microblogging postings in relation to normal blog postings was indeed higher than initially expected So from now on Planet Commandline has a basic flavour at http://planet-commandline.org/ and one with the microblogging feeds (climagic and commandlinefu) included at http://planet-commandline.org/+snippets/. For making this possible I hacked our Planet Venus wrapper to accept arbitary configuration snippets to be added at the end of the configuration as well as as sed-based modifications to the concatenated configuration before Planet Venus is run on them. This also allowed me to create further flavours of Planet Commandline: I hope nobody minds this diversification of Planet Commandline. Currently no combination of flavours is supported, but if there s a relevant demand for the one or the other combination of flavours I may have a look if that can be automated, too.

21 March 2011

Joerg Jaspert: Debian FTPMaster meeting 2011 - Alright Brain...Its all up to you

And so there went a day of work. Minutes done, come back tomorrow
Okok, more text: We did lots of stuff. There was breakfast, soon after we had Lunch, and guess, Dinner also. Now, would you please come back tomorrow?
Anyways, we did some things. One of those was to have PostgreSQL 9.0 and to enable the replication to the machine any DD can access. Which is, from today on, ries.debian.org. And for the first time ever (since that time ftp-master got closed to the public), DDs can now see the live status of the archive, literally a second after a package got accepted a dak ls on ries will show you that fact. Torsten worked on fixing up old remnants of code and database entries around contents and also tackled a problem between our usage of SQLalchemy and the python multiprocessing module. He also started working on the source contents and was weird enough to get us standardized logging into a dak tool. Mark worked with DSA on the Postgres 9.0 upgrade to fix what was to fix in out database and did some tests with our environment in squeeze and coordinated changes in python-apt needed for multiarch and xz. He then started working on some larger database changes around the built-using feature. I myself started off with getting generate-packages-sources to behave when there is an error instead of silently ignoring that. I also did a bit of groundwork to enable us to do NEW processing adding overrides for all affected binary packages, not just those from the architecture currently uploaded. That will allow stuff like linux-2.6 to pass NEW just once, not once per architecture. That involves a new field in our dsc files and the status of that is tracked in Bug #619131.

18 March 2011

Kai Wasserb ch: KVIrc updates (ahead)

For the users of Debian Squeeze we, the KVIrc maintainers, plan to upload the soon-to-be-released upstream bug fix release 4.0.4, packages of RC2 are available. RC2 is most likely going to be identical to the final 4.0.4 release. The update will be, if the SRMs agree (still need to ask them, but I wanted to wait for the final version before I do that), uploaded to stable-proposed-updates, maybe squeeze-updates and fix a few nasty bugs, which some users might experience, including graphical glitches when used under Gnome (upstream bug #1010), some crashes (e.g. upstream bugs #878, #879, #1093, #1098 and some without bug numbers like r4796), a regression in the Perl scripting support and a lot of translation updates (btw: if you use KVIrc and the translation for your mothertounge is lacking, consider to help! Coordinate your efforts on IRC.) Apart from that I just uploaded a new SVN snapshot of 4.1 into Sid, the next will follow in about ten days unless you report bugs, which make an earlier upload necessary. ;-)

Russell Coker: Some Postfix Scripts for dealing with Outbound Spamming

I ve just written some small scripts to help me manage spam emergencies on a mail server. I ve been doing this thing with a bit of manual effort for a while, but after having done it once from a phone I want to optimise it a bit to reduce painful typing.My observation is that when a system I run is used as an outbound mail relay for spamming I will notice this reasonably quickly by the queues getting big. Some portion of mail that is queued is delayed due to general network issues and anti-spam measures that neither accept nor reject mail cause a typical spam message to be more likely to be queued than a typical non-spam message. So I look for accounts that send large amounts of mail.The biggest mail server I run (and the only one to have an outbound spam problem) is configured to not allow users to fake their sender address (IE use an address from gmail.com or another server) unless they pay extra. So the spam I deal with tends to have a valid origin email address that I can use.Queue CountingThe following command provides a sorted list of the accounts that have the most mail in the queue. This usually means a maximum of 2 or 3 spammers at the end of the list:mailq grep ^[A-F0-9] cut -c 42-80 sort uniq -c sort -n tailThe following function declaration makes the command get-top-sender find the sender with the largest number of queue entries (which has been a spammer every time I ve checked such things) and assign their email address to the environment variable $TOP_SENDER. It has to be a shell function so that the environment variable will be set in the context of the shell and available to child processes.function get-top-sender() TOP_SENDER_LINE=$(mailq grep ^[A-F0-9] cut -c 42-80 sort uniq -c sort -n tail -1) ; export TOP_SENDER=$(echo $TOP_SENDER_LINE cut -f2 -d\ ) ; TOP_COUNT=$(echo $TOP_SENDER_LINE cut -f1 -d\ ); echo "Top Sender is $TOP_SENDER with $TOP_COUNT messages" ; Viewing the QueueThe following script uses the first parameter or the environment variable $TOP_SENDER to specify the user who s mail should be read. In the normal course of events user email won t be read by the sysadmin, but if we are to determine whether a slew of email is spam or not there s no other way. So far I haven t seen a large number of queued messages not be spam, but I expect it ll happen eventually and the user will be happy that I checked it instead of locking their account and deleting all the queued mail. For readers who don t do sysadmin work, this is why we always have entries in the Terms of Service about the possibility of your email being read to fix technical problems or to investigate possible breaches of the ToS from your end.#!/bin/bash
set -e
mkdir -p ~/tmp/$$
cd ~/tmp/$$
if [ "$1" != "" ]; then
TOP_SENDER="$1"
fi
if [ "$TOP_SENDER" = "" ]; then
echo "Specify the sender on the command-line or in \$TOP_SENDER"
exit 1
fi
for n in $(mailq grep ^[A-F0-9].*$TOP_SENDER cut -c 1-10) ; do
FILE=$(echo $n sed -e "s/^\(.\)/\/var\/spool\/postfix\/deferred\/\1\/\1/" -e "s/ .*$//")
if [ -f "$FILE" ]; then
postcat "$FILE" > $n
else
echo "File $FILE missing, probably active"
fi
done
less *
rm -rf ~/tmp/$$Note that this needs to be done with a single less command so that I can terminate it by pressing q and easily go to the next file with :n . It does waste some server resources by running postcat on all queue files relating to the user in question, but that doesn t matter, servers are supposed to be powerful enough to cope with some inefficiency in processing uncommon operations.The Final CommandAfter that I have a script that connects by ssh to all outbound mail relays and deletes bad messages from the queues and then connects to the database server to lock the account that was used for sending the mail. I m not publishing it because it s specific to the servers I run.So when NAGIOS reports that the queue is too large I have one script to find the most likely culprit, one script to view the queued email from that account, and a final script to lock the account and purge the spam.I ve also added a reminder of the command names to /etc/motd as I don t want to be running ls and set on a phone to discover command names that I ve forgotten.Any suggestions for improvements to this will be welcome.

16 March 2011

Daniel Kahn Gillmor: CryptLib and the OpenSSL License

I spent part of today looking at packaging Peter Gutmann's CryptLib for debian. My conclusion: it may not be worth my time. One of the main reasons i wanted to package it for debian is because i would like to see more OpenPGP-compliant free software available to debian users and developers, particularly if the code is GPL-compatible. Cryptlib makes it quite clear that it intends to be GPL-compatible:
cryptlib is distributed under a dual license that allows free, open-source use under a GPL-compatible license and closed-source use under a standard commercial license.
However, a significant portion of the cryptlib codebase (particularly within the bn/ and crypt/ directories) appears to derive directly from OpenSSL, and it retains Eric Young's copyright and licensing. This licensing retains the so-called "advertising clause" that is generally acknowledged to be deliberately incompatible with the GPL. (A common counterargument for this incompatibility is that OpenSSL should be considered a "System Library" for GPL'ed tools that link against it; whether or not you believe this for tools linked against OpenSSL, this counterargument clearly does not hold for a project that embeds and ships OpenSSL code directly, as CryptLib does) This does not mean that CryptLib is not free software (it is!), nor does it mean that you cannot link it against GPL'ed code (you can!). However, you probably can't distribute the results of linking CryptLib against any GPL'ed code, because the GPL is incompatible with the OpenSSL license. The un-distributability of derived GPL-covered works makes the CryptLib package much less appealing to me, since i want to be able to write distributable code that links against useful libraries, and many of the libaries i care about happen to be under the GPL. I also note that Peter Gutmann and CryptLib Security Software (apparently some sort of company set up to distribute CryptLib) may be in violation of Eric Young's License, which states:
 * 3. All advertising materials mentioning features or use of this software
 *    must display the following acknowledgement:
 *    "This product includes cryptographic software written by
 *     Eric Young (eay@cryptsoft.com)"
 *    The word 'cryptographic' can be left out if the rouines from the library
 *    being used are not cryptographic related :-).
A Google search for Eric Young's name on cryptlib.com yields no hits. I do find his name on page 340 of the Cryptlib manual, but that hardly seems like it covers "All advertising materials mentioning features or use of this software". I suppose it's also possible that CryptLib has negotiated an alternate license with Eric Young that I simply don't know about. In summary: i think CryptLib could go into debian, but its incorporation of OpenSSL-licensed code makes it less useful than i was hoping it would be. I have a few other concerns about the package after looking it over in more detail. I suspect these are fixable one way or another, but i haven't sorted them out yet. I'm not sure i'll spend the time to do so based on the licensing issues i sorted out above. But in case anyone feels inspired, the other concerns i see are:
embedded code copies
In addition to embedded (and slightly-modified) copies of bn/ and crypt/ from OpenSSL, CryptLib contains an embedded copy of zlib. Debian has good reasons for wanting to avoid embedded code copies. I've got it CryptLib building against the system copy of zlib, but the bits copied from OpenSSL seem to be more heavily patched, which might complicate relying on the system version.
executable stack
lintian reports shlib-with-executable-stack for the library after it is built. I don't fully understand what this means or how to fix it, but i suspect it derives from the following set of object files that also have an executable stack:
0 dkg@pip:~/src/cryptlib/cryptlib$ for foo in $(find . -iname '*.o'); do
>  if ! readelf -S "$foo"   grep -q .note.GNU-stack ; then
>   echo "$foo"
>  fi
> done
./shared-obj/desenc.o
./shared-obj/rc5enc.o
./shared-obj/md5asm.o
./shared-obj/castenc.o
./shared-obj/bfenc.o
./shared-obj/rmdasm.o
./shared-obj/sha1asm.o
./shared-obj/bn_asm.o
./shared-obj/rc4enc.o
0 dkg@pip:~/src/cryptlib/cryptlib$ 
They probably all need to be fixed, unless there's a good reason for them to be that way (in which case they need to be documented).
non-PIC code
lintian reports shlib-with-non-pic-code on the library after it is built. I don't fully understand what this means, or how to resolve it cleanly, but i imagine this is just a question of sorting out the details.
Anyone interested in using my packaging attempts as a jumping off point can start with:
git clone git://lair.fifthhorseman.net/~dkg/cryptlib
Oh, and please correct me about anything in this post; I'd be especially happy to hear that my analysis of the licensing issues above is wrong :) Tags: cryptlib, gpl, licensing, openssl

15 March 2011

Michal &#268;iha&#345;: GPL violation - getting more sources

It has been some time since I updated information about Gammu usage in proprietary software, so here it comes (see other posts tagged with gpl for more information). I still have troubles to explain Anwsoft what they are supposed to release. The good news is that they are still communicating and even providing some updated sources (see git repository), unfortunately they still construct fake projects rather than providing real sources of the application. Anyway if anybody is interested in digging in the sources and finding bits which might be interesting for porting to current Gammu code base, please go ahead, I probably won't have time for that in near future.

Filed under: English Gammu Gpl 3 comments Flattr this!

Colin Watson: Wubi bug 693671

I spent most of last week working on Ubuntu bug 693671 ("wubi install will not boot - phase 2 stops with: Try (hd0,0): NTFS5"), which was quite a challenge to debug since it involved digging into parts of the Wubi boot process I'd never really touched before. Since I don't think much of this is very well-documented, I'd like to spend a bit of time explaining what was involved, in the hope that it will help other developers in the future. Wubi is a system for installing Ubuntu into a file in a Windows filesystem, so that it doesn't require separate partitions and can be uninstalled like any other Windows application. The purpose of this is to make it easy for Windows users to try out Ubuntu without the need to worry about repartitioning, before they commit to a full installation. Wubi started out as an external project, and initially patched the installer on the fly to do all the rather unconventional things it needed to do; we integrated it into Ubuntu 8.04 LTS, which involved turning these patches into proper installer facilities that could be accessed using preseeding, so that Wubi only needs to handle the Windows user interface and other Windows-specific tasks. Anyone familiar with a GNU/Linux system's boot process will immediately see that this isn't as simple as it sounds. Of course, ntfs-3g is a pretty solid piece of software so we can handle the Windows filesystem without too much trouble, and loopback mounts are well-understood so we can just have the initramfs loop-mount the root filesystem. Where are you going to get the kernel and initramfs from, though? Well, we used to copy them out to the NTFS filesystem so that GRUB could read them, but this was overly complicated and error-prone. When we switched to GRUB 2, we could instead use its built-in loopback facilities, and we were able to simplify this. So all was more or less well, except for the elephant in the room. How are you going to load GRUB? In a Wubi installation, NTLDR (or BOOTMGR in Windows Vista and newer) still owns the boot process. Ubuntu is added as a boot menu option using BCDEdit. You might then think that you can just have the Windows boot loader chain-load GRUB. Unfortunately, NTLDR only loads 16 sectors - 8192 bytes - from disk. GRUB won't fit in that: the smallest core.img you can generate at the moment is over 18 kilobytes. Thus, you need something that is small enough to be loaded by NTLDR, but that is intelligent enough to understand NTFS to the point where it can find a particular file in the root directory of a filesystem, load boot loader code from it, and jump to that. The answer for this was GRUB4DOS. Most of GRUB4DOS is based on GRUB Legacy, which is not of much interest to us any more, but it includes an assembly-language program called GRLDR that supports doing this very thing for FAT, NTFS, and ext2. In Wubi, we build GRLDR as wubildr.mbr, and build a specially-configured GRUB core image as wubildr. Now, the messages shown in the bug report suggested a failure either within GRLDR or very early in GRUB. The first thing I did was to remember that GRLDR has been integrated into the grub-extras ntldr-img module suitable for use with GRUB 2, so I tried building wubildr.mbr from that; no change, but this gave me a modern baseline to work on. OK; now to try QEMU (you can use tricks like qemu -hda /dev/sda if you're very careful not to do anything that might involve writing to the host filesystem from within the guest, such as recursively booting your host OS ... [update: Tollef Fog Heen and Zygmunt Krynicki both point out that you can use the -snapshot option to make this safer]). No go; it hung somewhere in the middle of NTLDR. Still, I could at least insert debug statements, copy the built wubildr.mbr over to my test machine, and reboot for each test, although it would be slow and tedious. Couldn't I? Well, yes, I mostly could, but that 8192-byte limit came back to bite me, along with an internal 2048-byte limit that GRLDR allocates for its NTFS bootstrap code. There were only a few spare bytes. Something like this would more or less fit, to print a single mark character at various points so that I could see how far it was getting:
	pushal
	xorw	%bx, %bx	/* video page 0 */
	movw	$0x0e4d, %ax	/* print 'M' */
	int	$0x10
	popal
In a few places, if I removed some code I didn't need on my test machine (say, CHS compatibility), I could even fit in cheap and nasty code to print a single register in hex (as long as you didn't mind 'A' to 'F' actually being ':' to '?' in ASCII; and note that this is real-mode code, so the loop counter is %cx not %ecx):
	/* print %edx in dumbed-down hex */
	pushal
	xorw	%bx, %bx
	movb	$0xe, %ah
	movw	$8, %cx
1:
	roll	$4, %edx
	movb	%dl, %al
	andb	$0xf, %al
	int	$0x10
	loop	1b
	popal
After a considerable amount of work tracking down problems by bisection like this, I also observed that GRLDR's NTFS code bears quite a bit of resemblance in its logical flow to GRUB 2's NTFS module, and indeed the same person wrote much of both. Since I knew that the latter worked, I could use it to relieve my brain of trying to understand assembly code logic directly, and could compare the two to look for discrepancies. I did find a few of these, and corrected a simple one. Testing at this point suggested that the boot process was getting as far as GRUB but still wasn't printing anything. I removed some Ubuntu patches which quieten down GRUB's startup: still nothing - so I switched my attentions to grub-core/kern/i386/pc/startup.S, which contains the first code executed from GRUB's core image. Code before the first call to real_to_prot (which switches the processor into protected mode) succeeded, while code after that point failed. Even more mysteriously, code added to real_to_prot before the actual switch to protected mode failed too. Now I was clearly getting somewhere interesting, but what was going on? What I really wanted was to be able to single-step, or at least see what was at the memory location it was supposed to be jumping to. Around this point I was venting on IRC, and somebody asked if it was reproducible in QEMU. Although I'd tried that already, I went back and tried again. Ubuntu's qemu is actually built from qemu-kvm, and if I used qemu -no-kvm then it worked much better. Excellent! Now I could use GDB:
(gdb) target remote   qemu -gdb stdio -no-kvm -hda /dev/sda
This let me run until the point when NTLDR was about to hand over control, then interrupt and set a breakpoint at 0x8200 (the entry point of startup.S). This revealed that the address that should have been real_to_prot was in fact garbage. I set a breakpoint at 0x7c00 (GRLDR's entry point) and stepped all the way through to ensure it was doing the right thing. In the process it was helpful to know that GDB and QEMU don't handle real mode very well between them. Useful tricks here were: Single-stepping showed that GRLDR was loading the entirety of wubildr correctly and jumping to it. The first instruction it jumped to wasn't in startup.S, though, and then I remembered that we prefix the core image with grub-core/boot/i386/pc/lnxboot.S. Stepping through this required a clear head since it copies itself around and changes segment registers a few times. The interesting part was at real_code_2, where it copies a sector of the kernel to the target load address, and then checks a known offset to find out whether the "kernel" is in fact GRUB rather than a Linux kernel. I checked that offset by hand, and there was the smoking gun. GRUB recently acquired Reed-Solomon error correction on its core image, to allow it to recover from other software writing over sectors in the boot track. This moved the magic number lnxboot.S was checking somewhat further into the core image, after the first sector. lnxboot.S couldn't find it because it hadn't copied it yet! A bit of adjustment and all was well again. The lesson for me from all of this has been to try hard to get an interactive debugger working. Really hard. It's worth quite a bit of up-front effort if it saves you from killing neurons stepping through pages of code by hand. I think the real-mode debugging tricks I picked up should be useful for working on GRUB in the future.

14 March 2011

Axel Beckert: Planet Commandline officially online

Around the first bunch of postings in my Useful but Unknown Unix Tools, Tobias Klauser of inotail and Symlink fame came up with the idea of making a Planet (i.e. a blog aggregator) of all the comandline blogs and blog categories out there. A first Planet Venus running prototype based on the template and style sheets of Planet Symlink was quickly up and running. I just couldn t decide if I should use an amber or phosphor green style for this new planet. Marius Rieder finally had the right idea to solve this dilemma: Offer both, an amber and a phosphor green style. Christian Herzog pointed me to the right piece of code at A List Apart. So here is it, available in you favourite screen colors:

Planet Commandline For a beginning, the following feeds are included:

Which leads us to the discussion what kind of feeds should be included in Planet Commandline. Of course, all blogs or blog categories which (nearly) solely post neat tips and tricks about the command line in English are welcome. Microblogging feeds containing (only) small but useful command line tips are welcome, too, if they neither permanently contain dozens of posts per day nor have a low signal-to-noise ratio. Unfortunately most identi.ca groups do, so they re not suitable for such a planet. What I m though unsure about are non-English feeds. Yes, there s one in already, but I noticed this only after including Beat s Chr tertee and his FreeBSD command line tips are really good. So if it doesn t go overboard, I think it s ok. If there are too many non-English feeds, I ll probably split Planet Commandline off into at least three Planets: One with all feeds, one with English only and one with all non-English feeds or maybe even one feed per language. But for now that s still a long way off. Another thing I m unsure about are more propgram specific blogs like the impressive Mastering Emacs blog about mastering the world s best text editor . *g* (Yeah, I didn t include that one yet. But as soon someone shows me the vi-equivalent of that blog, I ll include both. Anyone thinks, spf13 s vim category is up to that?) Oh, and sure, any shell-specific (zsh, tcsh, bash, mksh, busybox) tips & tricks blogs don t count as program-specific blogs like some $EDITOR, $BROWSER, or $VCS specific blogs do. :-) Of course I m happy about further suggestions for feeds to include in Planet Commandline. Just remember that the feed should provide (at least nearly) exclusively command line tips, tricks or howtos. Suggestions for links to other commandline related planets are welcome, too.

Next.

Previous.