Search Results: "Daniel Burrows"

9 December 2021

David Kalnischkies: APT for Advent of Code

Screenshot of my Advent of Code 2021 status page as of today Advent of Code 2021
Advent of Code, for those not in the know, is a yearly Advent calendar (since 2015) of coding puzzles many people participate in for a plenary of reasons ranging from speed coding to code golf with stops at learning a new language or practicing already known ones. I usually write boring C++, but any language and then some can be used. There are reports of people implementing it in hardware, solving them by hand on paper or using Microsoft Excel so, after solving a puzzle the easy way yesterday, this time I thought: CHALLENGE ACCEPTED! as I somehow remembered an old 2008 article about solving Sudoku with aptitude (Daniel Burrows via archive.org as the blog is long gone) and the good same old a package management system that can solve [puzzles] based on package dependency rules is not something that I think would be useful or worth having (Russell Coker). Day 8 has a rather lengthy problem description and can reasonably be approached in a bunch of different way. One unreasonable approach might be to massage the problem description into Debian packages and let apt help me solve the problem (specifically Part 2, which you unlock by solving Part 1. You can do that now, I will wait here.) Be warned: I am spoiling Part 2 in the following, so solve it yourself first if you are interested. I will try to be reasonable consistent in naming things in the following and so have chosen: The input we get are lines like acedgfb cdfbe gcdfa fbcad dab cefabd cdfgeb eafb cagedb ab cdfeb fcadb cdfeb cdbaf. The letters are wires mixed up and connected to the segments of the displays: A group of these letters is hence a digit (the first 10) which represent one of the digits 0 to 9 and (after the pipe) the four displays which match (after sorting) one of the digits which means this display shows this digit. We are interested in which digits are displayed to solve the puzzle. To help us we also know which segments form which digit, we just don't know the wiring in the back. So we should identify which wire maps to which segment! We are introducing the packages wire-X-connects-to-Y for this which each provide & conflict1 with the virtual packages segment-Y and wire-X-connects. The later ensures that for a given wire we can only pick one segment and the former ensures that not multiple wires map onto the same segment. As an example: wire a's possible association with segment b is described as: 
Package: wire-a-connects-to-b
Provides: segment-b, wire-a-connects
Conflicts: segment-b, wire-a-connects
Note that we do not know if this is true! We generate packages for all possible (and then some) combinations and hope dependency resolution will solve the problem for us. So don't worry, the hard part will be done by apt, we just have to provide all (im)possibilities! What we need now is to translate the 10 digits (and 4 outputs) from something like acedgfb into digit-0-is-eight and not, say digit-0-is-one. A clever solution might realize that a one consists only of two segments so a digit wiring up seven segments can not be a 1 (and must be 8 instead), but again we aren't here to be clever: We want apt to figure that out for us! So what we do is simply making every digit-0-is-N (im)possible choice available as a package and apply constraints: A given digit-N can only display one number and each N is unique as digit so for both we deploy Provides & Conflicts again. We also need to reason about the segments in the digits: Each of the digit packages gets Depends on wire-X-connects-to-Y where X is each possible wire (e.g. acedgfb) and Y each segment forming the digit (e.g. cf for one). The different choices for X are or'ed together, so that either of them satisfies the Y. We know something else too through: The segments which are not used by the digit can not be wired to any of the Xs. We model this with Conflicts on wire-X-connects-to-Y. As an example: If digit-0s acedgfb would be displaying a one (remember, it can't) the following package would be installable: 
Package: digit-0-is-one
Version: 1
Depends: wire-a-connects-to-c   wire-c-connects-to-c   wire-e-connects-to-c   wire-d-connects-to-c   wire-g-connects-to-c   wire-f-connects-to-c   wire-b-connects-to-c,
         wire-a-connects-to-f   wire-c-connects-to-f   wire-e-connects-to-f   wire-d-connects-to-f   wire-g-connects-to-f   wire-f-connects-to-f   wire-b-connects-to-f
Provides: digit-0, digit-is-one
Conflicts: digit-0, digit-is-one,
  wire-a-connects-to-a, wire-c-connects-to-a, wire-e-connects-to-a, wire-d-connects-to-a, wire-g-connects-to-a, wire-f-connects-to-a, wire-b-connects-to-a,
  wire-a-connects-to-b, wire-c-connects-to-b, wire-e-connects-to-b, wire-d-connects-to-b, wire-g-connects-to-b, wire-f-connects-to-b, wire-b-connects-to-b,
  wire-a-connects-to-d, wire-c-connects-to-d, wire-e-connects-to-d, wire-d-connects-to-d, wire-g-connects-to-d, wire-f-connects-to-d, wire-b-connects-to-d,
  wire-a-connects-to-e, wire-c-connects-to-e, wire-e-connects-to-e, wire-d-connects-to-e, wire-g-connects-to-e, wire-f-connects-to-e, wire-b-connects-to-e,
  wire-a-connects-to-g, wire-c-connects-to-g, wire-e-connects-to-g, wire-d-connects-to-g, wire-g-connects-to-g, wire-f-connects-to-g, wire-b-connects-to-g
Repeat such stanzas for all 10 possible digits for digit-0 and then repeat this for all the other nine digit-N. We produce pretty much the same stanzas for display-0(-is-one), just that we omit the second Provides & Conflicts from above (digit-is-one) as in the display digits can be repeated. The rest is the same (modulo using display instead of digit as name of course). Lastly we create a package dubbed solution which depends on all 10 digit-N and 4 display-N all of them virtual packages apt will have to choose an installable provider from and we are nearly done! The resulting Packages file2 we can give to apt while requesting to install the package solution and it will spit out not only the display values we are interested in but also which number each digit represents and which wire is connected to which segment. Nifty! 
$ ./skip-aoc 'acedgfb cdfbe gcdfa fbcad dab cefabd cdfgeb eafb cagedb ab   cdfeb fcadb cdfeb cdbaf'
[ ]
The following additional packages will be installed:
  digit-0-is-eight digit-1-is-five digit-2-is-two digit-3-is-three
  digit-4-is-seven digit-5-is-nine digit-6-is-six digit-7-is-four
  digit-8-is-zero digit-9-is-one display-1-is-five display-2-is-three
  display-3-is-five display-4-is-three wire-a-connects-to-c
  wire-b-connects-to-f wire-c-connects-to-g wire-d-connects-to-a
  wire-e-connects-to-b wire-f-connects-to-d wire-g-connects-to-e
[ ]
0 upgraded, 22 newly installed, 0 to remove and 0 not upgraded.
We are only interested in the numbers on the display through, so grepping the apt output (-V is our friend here) a bit should let us end up with what we need as calculating3 is (unsurprisingly) not a strong suit of our package relationship language so we need a few shell commands to help us with the rest. 
$ ./skip-aoc 'acedgfb cdfbe gcdfa fbcad dab cefabd cdfgeb eafb cagedb ab   cdfeb fcadb cdfeb cdbaf' -qq
5353
I have written the skip-aoc script as a testcase for apt, so to run it you need to place it in /path/to/source/of/apt/test/integration and built apt first, but that is only due to my laziness. We could write a standalone script interfacing with the system installed apt directly and in any apt version since ~2011. To hand in the solution for the puzzle we just need to run this on each line of the input (~200 lines) and add all numbers together. In other words: Behold this beautiful shell one-liner: parallel -I ' ' ./skip-aoc ' ' -qq < input.txt paste -s -d'+' - bc (You may want to run parallel with -P to properly grill your CPU as that process can take a while otherwise and it still does anyhow as I haven't optimized it at all the testing framework does a lot of pointless things wasting time here, but we aren't aiming for the leaderboard so ) That might or even likely will fail through as I have so far omitted a not unimportant detail: The default APT resolver is not able to solve this puzzle with the given problem description we need another solver! Thankfully that is as easy as installing apt-cudf (and with it aspcud) which the script is using via --solver aspcud to make apt hand over the puzzle to a "proper" solver (or better: A solver who is supposed to be good at "answering set" questions). The buildds are using this for experimental and/or backports builds and also for installability checks via dose3 btw, so you might have encountered it before. Be careful however: Just because aspcud can solve this puzzle doesn't mean it is a good default resolver for your day to day apt. One of the reasons the default resolver has such a hard time solving this here is that or-groups have usually an order in which the first is preferred over every later option and so fort. This is of no concern here as all these alternatives will collapse to a single solution anyhow, but if there are multiple viable solutions (which is often the case) picking the "wrong" alternative can have bad consequences. A classic example would be exim4 postfix nullmailer. They are all MTAs but behave very different. The non-default solvers also tend to lack certain features like keeping track of auto-installed packages or installing Recommends/Suggests. That said, Julian is working on another solver as I write this which might deal with more of these issues. And lastly: I am also relatively sure that with a bit of massaging the default resolver could be made to understand the problem, but I can't play all day with this maybe some other day. Disclaimer: Originally posted in the daily megathread on reddit, the version here is just slightly better understandable as I have hopefully renamed all the packages to have more conventional names and tried to explain what I am actually doing. No cows were harmed in this improved version, either.
  1. If you would upload those packages somewhere, it would be good style to add Replaces as well, but it is of minor concern for apt so I am leaving them out here for readability.
  2. We have generated 49 wires, 100 digits, 40 display and 1 solution package for a grant total of 190 packages. We are also making use of a few purely virtual ones, but that doesn't add up to many packages in total. So few packages are practically childs play for apt given it usually deals with thousand times more. The instability for those packages tends to be a lot better through as only 22 of 190 packages we generated can (and will) be installed. Britney will hate you if your uploads to Debian unstable are even remotely as bad as this.
  3. What we could do is introduce 10.000 packages which denote every possible display value from 0000 to 9999. We would then need to duplicate our 10.190 packages for each line (namespace them) and then add a bit more than a million packages with the correct dependencies for summing up the individual packages for apt to be able to display the final result all by itself. That would take a while through as at that point we are looking at working with ~22 million packages with a gazillion amount of dependencies probably overworking every solver we would throw at it a bit of shell glue seems the better option for now.
This article was written by David Kalnischkies on apt-get a life and republished here by pulling it from a syndication feed. You should check there for updates and more articles about apt and EDSP.

19 January 2012

Obey Arthur Liu: State of the Aptitude (week 12)

Hi folks!

We're almost done for the official Summer of Code program and I thought that I couldn't let it end without another update, so here we are (you may thank Daniel Burrows for additional poking).

The Gtk+ interface for Aptitude is making great progress. The product is not final yet but already implements many of the planned ideas, with others to come.
Here are the screenies:




So, let's see the tasks presented in my previous update:

In some more details: we are now much closer to the final interface. Some things are not done yet, for example, notification bars la Iceweasel (or is it Firefox ?) are in the works for unobtrusive notifications about important things and a little guidance.

We are much more reliable now. We eliminated lots of memory management problems that resulted from, mmh, imperfect coding discipline. We also lock the interface when important backend stuff reload to prevent the user from shooting himself in the foot when triggering an install when the APT cache is reloading for example.. The backend refactoring took a sizable part of this second term but it was worth it. The code is now much cleaner and uses no more (at least much fewer) ugly unscalable hacks.

One very important feature of Aptitude is the ability to easily display all kinds of dependencies and navigate through them by hyperlinking. It's now working in the Gtk+ interface as well. Feel free to click on whatever looks like a package or a package version.

We now support the powerful Aptitude package searching syntax. Daniel Burrows is talking with Enrico Zini to bring the power of apt-xapian-index to Aptitude: autocompletion! lightning fast full-text searches! tag clouds! ponies!

There is still a lot to be done. I'm not going away after the GSoC program. I'm here to stay, for Aptitude and Debian. Daniel Burrows has been very helpful and supportive. The Debian community is as awesome as I knew it before participating in the program.

Daniel Burrows is planning to merge the Gtk+ interface into the main branch in September and packages may appear into Experimental. With the very interesting evolutions of the upcoming Adept-3, Lenny+1 will show very interesting changes in the area of packages managers.

Now for you. I need you to test my interface. All necessary instructions are on the development trac : http://dev.graffit.net/aptitude/trac.

Obey Arthur Liu: Aptitude 0.5.0 (aka Aptitude-gtk) released

Long time no post. Anyway, I have some good news.

The Gtk code for Aptitude has been merged some time ago into the main development trunk and we now have a release in Experimental.



For those that don't know about it, here's what it's all about : "The new frontend is is an effort to bring some of the design principles of the curses frontend to a GUI environment, while also exploiting the unique features a GUI gives us and exploring ways to deal with changes in the environment in the nine years since aptitude was first designed."

I had a very good time this summer working on Aptitude with Daniel Burrows in the Google Summer of Code program and I'm very glad we now have a real release. This version is by no means final or perfect but it's a good start.

Head for the blog post from Daniel for some other informations : [Daniel Burrows]

14 January 2012

Christian Perrier: Aptitude revival

Maybe many people have missed this: Aptitude package manager is undergoing a revival. Two fellows, Daniel Hartwig and Manuel A. Fernandez Montecelo, started triaging bugs and preparing a possible new version. Daniel Burrows, who maintained it for years (and did a really good job in this, given the high exposure of aptitude, which happened to be the recommended package managers during a few years), has currently less free time for doing the work. So, this is really great to see people motivated in taking the job over. In case you want to help, please joing the "aptitude-devel" mailing list on lists.alioth.debian.org. There is an Alioth project but Daniel is currently the only administrator. I might ask for admin privileges so that I can validate new team members. Of course, I personnally can't do much in this (C++ programming, coding, ah ah ah...things that are black magic for me), except my usual help for i18n/l10n. But, at minimum, I can make noise about this effort to encourage the volunteers who commit themselves to revive the project. And, I can certainly help by sponsoring their work (none of them is DD as of now) even if I am not skilled enough to review everything. I don't want to see good work wasted because people insist on nitpicking each and every commit line before "approving" it (any idea what I'm talking about, here? :-)).

21 January 2011

Rapha&#235;l Hertzog: People behind Debian: Michael Vogt, synaptic and APT developer

Michael and his daughter Marie

Michael has been around for more than 10 years and has always contributed to the APT software family. He s the author of the first real graphical interface to APT synaptic. Since then he created software-center as part of his work for Ubuntu. Being the most experienced APT developer, he s naturally the coordinator of the APT team. Check out what he has to say about APT s possible evolutions. My questions are in bold, the rest is by Michael. Who are you? My name is Michael Vogt, I m married and have two little daughters. We live in Germany (near to Trier) and I work for Canonical as a software developer. I joined Debian as a developer in early 2000 and started to contribute to Ubuntu in 2004. What s your biggest achievement within Debian or Ubuntu? I can not decide on a single one so I will just be a bit verbose. From the very beginning I was interested in improving the package manager experience and the UI on top for our users. I m proud of the work I did with synaptic. It was one of the earliest UIs on top of apt. Because of my work on synaptic I got into apt development as well and fixed bugs there and added new features. I still do most of the uploads here, but nowadays David Kalnischkies is the most active developer. I also wrote a bunch of tools like gdebi, update-notifier, update-manager, unattended-upgrade and software-properties to make the update/install situation for the user easier to deal with. Most of the tools are written in python so I added a lot of improvements to python-apt along the way, including the initial high level apt interface and a bunch of missing low-level apt_pkg features. Julian Andres Klode made a big push in this area recently and thanks to his effort the bindings are fully complete now and have good documentation. My most recent project is software-center. Its aim is to provide a UI strongly targeted for end-users. The goal of this project is to make finding and installing software easy and beautiful. We have a fantastic collection of software to offer and software-center tries to present it well (including screenshots, instant search results and soon ratings&reviews). This builds on great foundations like aptdaemon by Sebastian Heinlein, screenshots.debian.net by Christoph Haas, ddtp.debian.org by Michael Bramer, apt-xapian-index by Enrico Zini and many others (this is what I love about free software, it usually adds , rarely takes away ). What are your plans for Debian Wheezy? For apt I would love to see a more plugable architecture for the acquire system. It would be nice to be able to make apt-get update (and the frontends that use this from libapt) be able to download additional data (like debtags or additional index file that contains more end-user targeted information). I also want to add some scripts so that apt (optionally) creates btrfs snapshots on upgrade and provide some easy way to rollback in case of problems. There is also some interesting work going on around making the apt problem resolver a more plugable part. This way we should be able to do much faster development. software-center will get ratings&reviews in the upstream branch, I really hope we can get that into Wheezy. If you could spend all your time on Debian, what would you work on? In that case I would start with a refactor of apt to make it more robust about ABI breaks. It would be possible to move much faster once this problem is solved (its not even hard, it just need to be done). Then I would add a more complete testsuite. Another important problem to tackle is to make maintainer scripts more declarative. I triaged a lot of upgrade bug reports (mostly in ubuntu though) and a lot of them are caused by maintainer script failures. Worse is that depending on the error its really hard for the user to solve the problem. There is also a lot of code duplication. Having a central place that contains well tested code to do these jobs would be more robust. Triggers help us a lot here already, but I think there is still more room for improvement. What s the biggest problem of Debian? That s a hard question :) I mostly like Debian the way it is. What frustrated me in the past were flamewars that could have been avoided. To me being respectful to each other is important, I don t like flames and insults because I like solving problems and fighting like this rarely helps that. The other attitude I don t like is to blame people and complain instead of trying to help and be positive (the difference between it sucks because it does not support $foo instead of it would be so helpful if we had $foo because it enables me to let me do $bar ). For a long time, I had the feeling you were mostly alone working on APT and were just ensuring that it keeps working. Did you also had this feeling and are things better nowadays ? I felt a bit alone sometimes :) That being said, there were great people like Eugene V. Lyubimkin and Otavio Salvador during my time who did do a lot of good work (especially at release crunch times) and helped me with the maintenance (but got interested in other area than apt later). And now we have the unstoppable David Kalnischkies and Julian Andres Klode. Apt is too big for a single person, so I m very happy that especially David is doing superb work on the day-to-day tasks and fixes (plus big project like multiarch and the important but not very thankful testsuite work). We talk about apt stuff almost daily, doing code reviews and discuss bugs. This makes the development process much more fun and healthy. Julian Andres Klode is doing interesting work around making the resolver more plugable and Christian Perrier is as tireless as always when it comes to the translations merging. I did a quick grep over the bzr log output (including all branch merges) and count around ~4300 total commits (including all revisions of branches merged). Of that there ~950 commits from me plus an additional ~500 merges. It was more than just ensuring that it keeps working but I can see where this feeling comes from as I was never very verbose. Apt also was never my only project, I am involved in other upstream work like synaptic or update-manager or python-apt etc). This naturally reduced the time available to hack on apt and spend time doing the important day-to-day bug triage, response to mailing list messages etc. One the python-apt side Julian Andres Klode did great work to improve the code and the documentation. It s a really nice interface and if you need to do anything related to packages and love python I encourage you to try it. Its as simple as: 
import apt
cache = apt.Cache()
cache["update-manager"].mark_install()
cache.commit()
Of course you can do much more with it (update-manager, software-center and lots of more tools use it). With pydoc apt you can get a good overview. The apt team always welcomes contributors. We have a mailing list and a irc channel and it s a great opportunity to solve real world problems. It does not matter if you want to help triage bugs or write documentation or write code, we welcome all contributors. You re also an Ubuntu developer employed by Canonical. Are you satisfied with the level of cooperation between both projects? What can we do to get Ubuntu to package new applications developed by Canonical directly in Debian? Again a tricky question :) When it comes to cooperation there is always room for improvement. I think (with my Canonical hat on) we do a lot better than we did in the past. And it s great to see the current DPL coming to Ubuntu events and talking about ways to improve the collaboration. One area that I feel that Debian would benefit is to be more positive about NMUs and shared source repositories (collab-maint and LowThresholdNmu are good steps here). The lower the cost is to push a patch/fix (e.g. via direct commit or upload) the more there will be. When it comes to getting packages into Debian I think the best solution is to have a person in Debian as a point of contact to help with that. Usually the amount of work is pretty small as the software will have a debian/* dir already with useful stuff in it. But it helps me a lot to have someone doing the Debian uploads, responding to the bugmail etc (even if the bugmail is just forwarded as upstream bugreports :) IMO it is a great opportunity especially for new packagers as they will not have to do a lot of packaging work to get those apps into Debian. This model works very well for me for e.g. gdebi (where Luca Falavigna is really helpful on the Debian side). Is there someone in Debian that you admire for his contributions? There are many people I admire. Probably too many to mention them all. I always find it hard to single out individual people because the project as a whole can be so proud of their achievements. The first name that comes to my mind is Jason Gunthorpe (the original apt author) who I ve never met. The next is Daniel Burrows who I met and was inspired by. David Kalnischkies is doing great work on apt. From contributing his first (small) patch to being able to virtually fix any problem and adding big features like multiarch support in about a year. Sebastian Heinlein for aptdaemon. Christian Perrier has always be one of my heroes because he cares so much about i18n. Christoph Haas for screenshots.debian.net, Michael Bramer for his work on debian translated package descriptions.
Thank you to Michael for the time spent answering my questions. I hope you enjoyed reading his answers as I did. Subscribe to my newsletter to get my monthly summary of the Debian/Ubuntu news and to not miss further interviews. You can also follow along on Identi.ca, Twitter and Facebook.

4 comments Liked this article? Click here. My blog is Flattr-enabled.

31 May 2010

Piotr Galiszewski: Short project update

Hello all.

Last week I was very busy, due to my university duties. On Tuesday I had my first exam in this semester (in Numerical Analysis), which took me more time than I had thought it would be.
From this reason my project is slightly behind schedule, but now I will do my best to catch the plan.

On the other hand, there are also some good news for me. Some time ago, Daniel Burrows migrated aptitude repository from mercurial to GIT. I am git user for nearly year, so I strongly support this move ;) .

Here are some basic thoughts which functionalities good package manager must have (and which approach (from aptitude-gtk, synaptic and adept) I like the most):
All of this will be elaborated in mockups. Am I forgetting something important?

The most important questions is: "What is the most common use case for package manager? Installing packages or upgrading packages?". I am not sure if this two actions require different views, or there should be only new category in install package view.

This is all now. More updates (and mockups) will be available today's midnight (or tomorrow morning). So please stay tuned :)

6 April 2010

Daniel Burrows: aptitude-related projects for Summer of Code 2010

I wasn't planning to be a mentor this year, but it looks like I might have enough time to do a decent job at mentoring after all. So here are a few ideas I have kicking around that would probably be suitable for an SoC coder. I have lots more ideas, but these are the first ones that came to mind. These ideas are deliberately broad: as a student, you are responsible for collaborating with me to figure out the details of your project and put together a workable implementation plan. At the same time, this list is not all-encompassing; feel free to come up with your own idea and explain why I should act as a mentor for it.
  1. Improve the GTK+ interface. The GTK+ interface is only partly complete and I'd like it to be completely complete. A few open ends of interest are noted at aptitude-gtk-status-a-visual-tour, or you can just run the program yourself and see what's missing or ugly.
  2. Work on per-package Wiki pages (the idea is sketched out in my email on the subject). Most likely this can use the existing wiki.debian.org infrastructure.
  3. Create a service for tracking user reviews of Debian packages and implement support in one or more apt clients.
Note that the last two aren't aptitude-specific; they basically amount to working out a spec for clients to access the data in question, and then implementing the spec in one or more clients (possibly including aptitude, apt, synaptic, packages.debian.org, etc).

4 April 2010

Sune Vuorela: A Qt frontend to aptitude as a GSoC project?

I m currently trying to convince (and hopefully succeeding) Daniel Burrows to co-mentor a Qt frontend for aptitude. But for that a student is needed. http://wiki.debian.org/SummerOfCode2010/Aptitude-Qt for first draft of project. http://wiki.debian.org/gsoc for information about GSoC and debian
and #debian-soc on irc.debian.org if you prefer that communication media.

7 March 2010

Daniel Burrows: Installing OpenWRT on a Linksys WRT54GL v1.1

I finally got a few hours free this morning to check an item off my home system administration checklist: upgrading the wireless router's firmware to OpenWRT. There were a couple motivations for this, including the fact that my SoundBridge Radio couldn't maintain a connection to my firefly server using the built-in firmware, and I had read that OpenWRT would work better. Since this is posted on Planet Debian, I should mention why I didn't use DebianWRT. The basic answer is this text at the top of the DebianWRT homepage:
Currently the most common methods used to run Debian on these systems is to install OpenWRT or a similar firmware, add disk space either by USB storage or NFS, create a debian chroot by either running cdebootstrap from inside OpenWRT or debootstrap --foreign on a PC, and running Debian from this chroot. For example, instructions for the WLHDD.
I'm not sure whether this is because DebianWRT is experimental, or because its goal is to use routers as cheap general computers. Either way, it sounds way too complicated and/or fragile for what I'm interested in (i.e., a wireless router with better software). The goal here is to get something that does a better job than the built-in WRT firmware, but doesn't require too much tinkering to get working or to maintain. I have plenty of outlets for my tinkering urges already. Nothing here was exactly difficult, but it was hard to find all the information I needed to get things working. Hopefully documenting the steps I went through here will save someone else some time. Step 1: acquire the firmware This was trickier than you might think. If you click on the download link at the OpenWRT web site, you end up at the top of an FTP server populated with a vast quantity of stuff. Worse, when you find the actual firmware images, you'll quickly discover that there are piles and piles of them divided between sixteen directories, and no guidance as to which one to pick. And picking the wrong one will turn your beautiful router into a doorstop. Luckily, the OpenWRT documentation contains a section called "Getting Started". Unluckily, that section consists of the following text:
1.1 Getting started 1.1.1 Installation 1.1.2 Initial configuration 1.1.3 Failsafe mode
Whoops, someone forgot to execute on a TODO. :-) Undeterred, I consulted the usual fallback reference, Google. It pointed me at several references, some of which were hidden on other parts of the OpenWRT site. Armed with these, I was able to determine that:
  1. My WRT54GL v1.1 probably uses a Broadcom 5352 chipset.
  2. The correct firmware, according to multiple sources, is probably in the kamikaze top-level director, the latest version's subdirectory, and the bcrm-2.4 directory under the version (the link is to 8.09.2, which is current as of March 6, 2010). Apparently the bcrm47xx directory doesn't have wireless support; it helpfully contains a file called NO-BROADCOM-WIRELESS to warn you off, but unhelpfully doesn't include any additional information in that file (like what exactly is broken or that you can find a working firmware in a sibling directory) ... oh well.
  3. The firmware file that you want is openwrt-wrt54g-squashfs.bin, even though it doesn't match the model number of the router. This directory could use a README file explaining which hardware is supported by each of the dozen or so firmware files it contains.
Step 2: install the firmware After Step 1, this was a real relief. I just used the built-in Linksys firmware installer, pointed it at the .bin file, and it went. Step 3: configure the router The installation guide I followed was pretty much silent about what to do after I got the firmware on. Luckily, this is just a software problem, meaning it's much more familiar territory for me.
  1. First things first: I checked that I could still get a DHCP lease. It worked.
  2. Armed with that, I tried telneting to the router. I used the resulting root prompt to set a password on the root account.
  3. I logged out and tried telneting in. Apparently the router is configured to disallow root logins over telnet if you don't have a password. Good for them (although why allow telnet at all?); oops for me.
  4. Luckily, an ssh server was installed by default. I like using keys to log in, so I tracked down the documentation on configuring the server to use public-key authentication; it turns out there's a single global key file named /etc/dropbear/authorized_keys that's exactly like OpenSSH's per-user authorized_keys file. No idea what would happen if I had multiple users, but I won't.
  5. The next obstacle: I didn't have an Internet connection. For some reason, my cable modem didn't want to give the router a DHCP lease. On the off-chance that it was remembering too much, I rebooted it and ran ifup wan. That fixed the problem. I still don't know why.
  6. Not a step, but a useful note: in the process of figuring the above out, I found readlog. It's basically dmesg for syslog files; it shows the most recent lines written to syslog. This is useful because there isn't a real syslog file, due to the fact that there would be no room to store it on the 54GL.
  7. Finally, I had to get wireless working. The documentation is very helpful when it comes to describing the syntax of the wireless configuration. Unfortunately, I read the list of encryption options and missed the section right below where their meanings are explained (although, to be honest, I might not have understood the implications of the explanation without the research I did anyway).
    option encryption none, wep, psk, psk2, wpa, wpa2
    I wanted WPA2 encryption, so I entered wpa2. And nothing worked. After a good hour of trying different options on the client, swapping software components in and out on the router, experimenting with the encryption key syntax, and crawling Google, I finally found my answer. If you just want WPA2 encryption, you must not use wpa2 as the encryption type. Instead, use psk2. It turns out out that wpa2 actually means use WPA2 and also use an external RADIUS server for authentication. psk2 is the system you're familiar with from a typical consumer wireless router.
Step 4: enjoy And with that, it works. Unfortunately, contrary to what I wrote here originally, my Roku still doesn't work. On the other hand, having a real Linux installation is helpful for debugging it. Currently my suspicion is that the router isn't passing multicast packets between the wired and wireless interfaces (broadcast works fine, multicast doesn't). That said, it seems like if I restart Firefly just before I start playing music, I can play reasonably reliably -- as long as I don't stop, because if I do, the Roku forgets that the music server is there. Either way, I've spent about as much time fighting this as I can afford. :-/ One lingering worry I have is security; unlike Debian, which has both a security mailing list and tools to inform me when I need to install a security update, the OpenWRT firmware doesn't seem to have any mechanism for distributing security notices. True, there is an openwrt-security-announce list, but it appears to be entirely unused, as is openwrt-announce. Something to keep an eye on, then. Also (file under note to self), I need to remember to verify that the router isn't exposing services to the outside world. The default iptables configuration is hideously complex; with just a quick glance, it could be setting up a floral shop for all I can tell. I'll need to test this empirically and maybe analyze the rules in more depth.

3 May 2009

Daniel Burrows: aptitude-gtk status, a visual tour

As some of you may know, Arthur Liu did the initial work on a GTK+ interface for aptitude last summer as part of the Google Summer of Code program. Since that program ended, I've been working to complete the GTK+ interface, along with some unrelated but similarly large changes to other parts of aptitude. It's been a while since I've written anything about aptitude-gtk, though, mostly because I set it aside for a few months to improve the dependency solver. While I'll probably be doing more work on the backend over the summer, I also hope to get the GTK+ interface to a point where I can feel comfortable including it in a stable release of aptitude. There are two things that I want to be in place for the GTK+ interface before I release it:
  1. The interface should be functionally complete, meaning that you should be able to do anything you can do in the other interfaces.
  2. The interface should be reasonably straightforward to use and nice to look at. This is a never-ending task, but there are some pieces of the current interface that are particularly ugly or awkward that I'd like to fix before release.
I've gone through the major areas of the current UI and marked them up with notes about things that I think need to be done. This isn't a definitive or exhaustive list, but it should give some idea of where aptitude is and where it's going.
Dashboard Tab annotated dashboard tab The first thing you see when you start aptitude is the dashboard. This is a display that gives you immediate access to some of the most common things you'll want to do: searching for a package, viewing the available upgrades, and upgrading as many packages as aptitude can figure out how to upgrade without removing anything. This is one of the most complete parts of the interface, but there are a few minor aesthetic things that I'd like to take care of. [UPDATE] Michael Johnson writes:
I agree with most of the notes on this screen. However, I'd like to see the Selected Package changelog stay. I often look more than one upgrade back. The summary doesn't show more than what's being upgraded. However, I do like the idea of package description tab. The search box should definately always be available. I found it annoying when I had to click back to a different tab to get it. One thing I definately miss from the console version of aptitude is the list of recommended and suggested packages. I don't let aptitude auto-install recommended packages, but I like to know what they are.
Packages tab annotated packages tab This is the view that's shown after you search for a package by name, keyword, or by other criteria. It's also fairly complete, but there are more things that could be added (mostly enhancements). Related to this is the dialog box for editing which columns are visible: annotated visible columns editor
Dependency solver tab annotated dependency solver tab The dependency solver got a lot of love in the 0.5.2.1 release, and it's now basically functional. There are some minor layout issues to be fixed, and if I have time I'd like to add more ways of controlling the resolver's behavior.
Package information tab annotated package information tab The package information page is probably the weakest part of the GTK+ aptitude interface; it wouldn't be too much of a stretch to say that this tab is holding back a stable release of aptitude-gtk. The big problem I have with it is that I feel that the information tab should open up with an overview of the status of the package. What version is installed, what's available, what archive is it in, who's the maintainer, how big is it, etc. Instead, aptitude-gtk currently shows you a big tree of all the package's dependencies. While that's useful, it should be something the user chooses from. I also find this tab to be visually unappealing, but I'm not entirely sure how to fix that. [UPDATE] Gunnar Wolf writes:
As you have mentioned in your blog post (quote this mail if you so wish) that you want to make the package information screen "sexier": I think it would be natural to integrate screenshots (from http://screenshots.debian.net/) into it. Of course, it would be worth measuring whether it imposes too high a load into that system (maybe it should only get a screenshot if the user explicitly asks for it) - but surely would be a net win.
I think this is a great idea and I'm definitely planning to incorporate this feature. [UPDATE] Michael Johnson writes:
One of the things I really like about aptitude-gtk over console-aptitude is how obvious it makes some things. The version display on this tab is one of those things. Maybe it could be more obvious, I don't know. I agree, the dependency list is not the most important thing. And it's missing the reverse dependencies. I use that more than the dependencies. As far as the actual display of the list, I can't think of anything besides a tree. Although the current implementation does feel quite clunky. What I'd more like to see first is the list of files in the package. But I can see how other people would have different priorties. So it should be really easy to change the default view.
Preferences tab vast gaping hole where the preferences tab should be One of the reasons that it is a stretch to say that the information tab is holding aptitude-gtk back is the preferences tab, which is a huge pile of not-yet-implemented-ness. In fact, it's not even designed, let alone implemented. Releasing a GUI program with no way to alter the settings strikes me as ... silly.
Preview tab annotated The preview tab is a bit weaker than the packages tab, in my opinion, but much stronger than the information tab. Like the information tab, I find it visually unappealing, but I'm not entirely sure how to improve it. I'm currently leaning towards the paned browser idea. [UPDATE] Michael Johnson writes:
I've never been comfortable using this tab. It never made much sense to me. I think the last one or two times I used aptitude-gtk it started making sense, but that seems to be too long a time. I think it's the mass of text at each top level entry. The console version has brief descriptions with more info in the details pane when the entry is selected. Perhaps healthy use of tool tips or an info pane would help with this. The paned browser idea sounds promising. I think it will be easier for new users to follow. And it provides a natural way to provide more information on what each section is about.
That's all folks!

25 April 2009

Daniel Burrows: aptitude 0.5.2 released

I've released version 0.5.2 of the aptitude package manager (release notes). You can download it from Debian's experimental archive, or from the aptitude package page. WARNING: This is an unfinished piece of software. Do not install it if you want a package manager that does everything you need perfectly all the time. It is buggy and incomplete. This update includes major changes to the resolver backend and a new and improved GUI frontend for the resolver: It also includes an assortment of other fixes and improvements. Click through to the release notes to see a list of what changed in this release, with screen-shots.

10 April 2009

Obey Arthur Liu: Google Summer of Code 2009: Debian s Shortlist

Copy of http://lists.debian.org/debian-devel/2009/04/msg00421.html. Hi folks, We have been pretty busy these past few weeks with the whole Google Summer of Code 2009 student application process.
I can say that we have this year a very good set of proposals and I d like to thank all the students and mentors for this. I am going to present to you our shortlist of projects that we would like to be funded and believe we can reasonably manage to get funded. As always, remember that the number of slots is not final yet at this point so we can t promise anything. The first preliminary slot count given today was *10* (same as last year) and we hope to get *2* more (as we did last year). This shortlist is alphabetically ordered because we don t want to reveal the current internal rankings. I am inviting you to debate what you think is cool, what is useful, what is important to Debian, maybe give us pointers to resources or people that could be helpful for the projects. We will try to alter our current rankings to reflect the zeitgeist in Debian, while taking into account the personal information that we have about each student involved. The deadline for any modification is on the 15th, so get everything in by the 14th. The final selected projects will be announced by Google April 20th, ~12 noon PDT / 19:00 UTC. We ll have another announcement then. Three proposals need or may need a mentor, I indicated it. For more information about the projects or mentoring and how to talk to us directly, scroll down past the list. Debian s Shortlist : - Aptitude Package Management History Tracking
- Automatic Debug Packages Creation and Handling
- Debbugs Web UI: Amancay Strikes Back
- Control Files Parsing/Editing Library/Qt4-Debconf Qt4-Perl bindings
- Debian-Installer Support for GNU/kFreeBSD
- KDE/Qt4 Adept 3.0 Package Manager
- Large Scientific Dataset Package Management
- MIPS N32 ABI Port
- MTD Embedded Onboard flash Partitioning and Installation
- On-demand Cloud Computing with Amazon EC2 and Eucalyptus Integration
- Port back update-manager to Debian and all Derivatives
- Debian Autobuilding Infrastructure Rewrite And the details: Aptitude Package Management History Tracking Student: Cristian Mauricio Porras Duarte, Mentor: Daniel Burrows Aptitude currently does not track actions that the user has performed beyond a single session of the program. One of the most frequent requests from users is to find out when they made a change to a package, or why a package was changed; we want to store this information and expose it in the UI in convenient locations. As a side effect, this might also provide some ability to revert past changes. Automatic Debug Packages Creation and Handling Student: Emilio Pozuelo Monfort, Mentor: Marc Brockschmidt This proposal aims at providing debug binary packages for the packages in the Debian archive in an automatic manner, moving them away from the official Debian archive to an special one. This has the benefits of providing thousands of debug packages without any work needed from the developers, for all the architectures, without bloating
the archive. Debbugs Web UI: Amancay Strikes Back Student: Diego Escalante Urrelo, Mentor: Margarita Manterola The Amancay project aims to be a new read/write web frontend to Debian s BTS; allowing DDs and contributors to easily interact with bugs via an intuitive yet powerful interface, enabling new workflows and creating new contribution opportunities like triaging while upholding reporting quality. Control Files Parsing/Editing Library/Qt4-Debconf Qt4-Perl bindings Student: Jonathan Yu, Mentor: (probably) Dominique Dumont see below This project proposes a common library for parsing and manipulating Debian Control files, including control, copyright and changelog. Main ideas include validating and parsing of these files, with both Strict and Quirks modes for the parser. The second idea is a new frontend for Debconf using Qt4 (for which Perl bindings will be written). Debian-Installer Support for GNU/kFreeBSD Student: Luca Favatella, Mentor: Aurelien Jarno GNU/kFreeBSD is currently using a hacked version of the FreeBSD installer combined with crosshurd as its own installer. While this works more or less correctly for standard installations (read: the exact same installation as in the documentation), it does not allow any changes in the installation process except the hard disk partitioning. This project is about porting debian-installer on GNU/kFreeBSD, and to a bigger extent, make debian-installer less Linux dependant. KDE/Qt4 Adept 3.0 Package Manager Student: Mateusz Marek, Mentor: NEEDS MENTOR, see below. Finish Adept 3.0, a fully integrated package manager for Qt4/KDE4. Adept is currently the only viable path to a Debian native package manager on KDE that would support modern features such as tags, indexed search or good conflict resolving. With Aptitude-gtk still in development and only available for GTK+ and (K)PackageKit having fundamental problems, Debian needs this project to stay in control of its package management on KDE after much neglect in the recent years. Large Scientific Dataset Package Management Student: Roy Flemming Hvaara, Mentor: Charles Plessy Large public datasets, like databases for bioinformatics are typically too big and too volatile to fit the traditional source/binary packaging scheme of Debian. There are some programs that are distributed in Debian, like blast and emboss, that can index specialised databases, but Debian lacks a tool to install or update the datasets they need and keep their indexing in sync. MIPS N32 ABI Port Student: Sha Liu, Mentor: Anthony Fok This project first focuses on creating a new MIPS N32 ABI port for Debian. Different from O32 and N64, N32 is an address model which has most 64-bit capabilities but using 32-bit data structures to save space and process time. A second focus will be given on making such a mipsn32el arch fully optimized for the Loongson 2F CPU which gains more and more popularity in subnotebooks/netbooks in many countries. MTD Embedded Onboard flash Partitioning and Installation Student: Per Andersson, Mentor: Wookey Many embedded devices have MTD onboard flash as persistent storage like the Kurobox Pro NAS, the Neo Freerunner, the Sheeva Plug or the OLPC. With MTD flash being so popular and with increases in capacity, support for MTD partition/installation would make Debian even more interesting to a wide range of of devices, making it one step closer to being universal. On-demand Cloud Computing with Amazon EC2 and Eucalyptus Integration Student: David Wendt Jr, Mentor: (probably) Steffen Moeller see below In many academic fields, as well as commercial industries, people use clusters to distribute tasks among multiple machines. Many times this is done by packaging a whole operating system disk image, uploading it onto the cluster, and having the cluster run it in a VM. This project intends to make it easier for Debian to distribute prepared disk images templates like they distribute CD images now, for the users to recreate or customise these templates with Debian packages and for administrators to host such clusters with Debian. Port back update-manager to Debian and all Derivatives Student: Stephan Peijnik, Mentor: Michael Vogt The project would involve taking the distribution-(Ubuntu-)specific update-manager code, analyzing it, and creating a package with just its core functionality, decoupling the distribution-specific parts and thus making the core code extensible by distribution-specific add-ons. This in turn would remove the need of porting update-manager to Debian with every upstream release. An additional optional goal would be replacing the synaptics-backend with a python-apt based one. Debian Autobuilding Infrastructure Rewrite Student: Philipp Kern, Mentor: Luk Claes Rewrite the software that currently runs the Debian autobuilding infrastructure in a way that makes it more maintainable and robust. It will use Python as its programming language and PostgreSQL for the database backend. By harmonizing buildds, many build failures can be prevented and wasteful workload on buildd volunteers can be reduced. On mentoring: Petr Rockai, the original developer of Adept has offered help to anyone willing to adopt Adept. Sune Vuorela has offered help for any Qt4 and KDE related issues. *We really need a mentor here*. The student is quite competent but Google dictates that we provide a mentor to handle student management. Dominique Dumont, although not DD, has signaled interest in mentoring this, although it hasn t been confirmed yet. Sune Vuorela has offered to help co-mentor for the Qt4-Debconf and Qt4-Perl bindings part. Steffen Moeller has signaled interest in mentoring this, although it hasn t been formally confirmed yet. Charles Plessy of the Debian Med team will provide help for use cases related issues. Eric Hammond, developer of the original vmbuilder image creation tool and maintainer of a set of Debian and Ubuntu images will provide help for Amazon EC2 and image creation issues. Chris Grzegorczyk from the Eucalyptus team will provide help for Eucalyptus and Eucalyptus/Debian integration issues. Contacting us: Considering the tight schedule, most stuff happens live on IRC: #debian-soc on irc.debian.org You can also consult our wiki page for some additional information:
<http://wiki.debian.org/SummerOfCode2009> We have a mailing-list at:
<http://lists.alioth.debian.org/mailman/listinfo/soc-coordination> Keep this discussion on debian-devel@lists.debian.org while cc-ing soc-coordination@lists.alioth.debian.org. This thread is for debian-devel primarily.

23 March 2009

Daniel Burrows: Merge of doom

daniel@emurlahn:~/programming/aptitude/head$ hg log -r tip
changeset:   2611:c81f562751d6
tag:         tip
parent:      1665:94b597a663a7
parent:      2610:78100f86756d
user:        Daniel Burrows <dburrows@debian.org>
date:        Mon Mar 23 10:49:47 2009 -0700
summary:     Merge the entire post-lenny branch into head.
daniel@emurlahn:~/programming/aptitude/head$ hg log -p -r tip   diffstat -m   tail --lines 1
 239 files changed, 105227 insertions(+), 16334 deletions(-), 69203 modifications(!)
That's right: active [aptitude /projects/aptitude] development is now taking place in the head branch again, after a hiatus of about a year while I worked in a branch while waiting for Lenny to be released. Lenny is released now, and so the post-lenny and gtk branches have been merged into head and renamed to avoid accidental commits (you can find them under the names post-lenny-branch-is-closed and gtk-branch-is-closed). Future releases will come out of the head branch (http://hg.debian.org/hg/aptitude/head).

Daniel Burrows: Sometimes, tools are useful.

I generally try not to get caught up in writing auxiliary tools for my free software projects. My experience in the past has been that given how little spare time I have each week, a tool that takes more than an hour or two to write quickly ends up becoming a time sink that prevents me from making progress on the main project. But occasionally it's worth the effort. screenshot I spent the last couple weeks putting together a tool to extract information about a dependency search from a log trace of aptitude's resolver. The search is displayed in a log interface that shows both the steps of the search in order, and the tree structure of the search. It took a long time to write, but after working with it for just a few hours, I was able to track down the bugs in the resolver modifications I was trying to write; without a tool like this, it would have taken a lot of painful grepping through log files to get all the information I needed. In other news, the resolver now stores the choices it has made as a single set of choices, instead of using two different sets for the two types of choices it can make. There's no user-visible change (although I did fix another performance problem that affected safe-upgrade), but this will pave the way for adding more types of choices in a cleaner way: in particular, the resolver should get explicit support for package replacements in the near future. The code for this is available in the tools directory of aptitude's experimental Mercurial repository, http://hg.debian.org/hg/aptitude/post-lenny. When I get around to merging it this with the main repository, it will also be available at http://hg.debian.org/hg/aptitude/head.
A brief digression regarding the implementation language The tool is implemented in Haskell, a decision that I'm not sure was the right one. Haskell's data model was perfect for this task: having algebraic datatypes allows me to precisely represent cases where parts of the structure can't be inferred from the log file, or can only be inferred incompletely. And of course there are the other well-known benefits of Haskell (pure, a strong type system, good support for higher-order programming, etc). But all those benefits were wiped out by the time I spent fighting the runtime. The thing is, Haskell is a lazy language. In theory, this means that it never evaluates anything you don't need and that you can work with arbitrarily large (even infinite) values and only allocate the parts you actually use. In practice, this means that you have to pay excruciatingly close attention to every single expression you write, or as soon as you run your program on non-trivial input, it will either run very slowly, eat all your memory, die with a stack overflow, or all of the above. Oh, and did I mention that laziness means that you can't have a runtime debugger, that trace statements are mostly-useless, because your program doesn't run in any predictable order, and that it's impossible to get backtraces when there's a fatal error? I think if I were doing this over again, I'd use O'Caml. I'm less familiar with it and its syntax is clunkier, but I have used it productively in the past and it provides the core stuff I wanted from Haskell (functional programming with algebraic datatypes). If someone could make a language that was pure, had Haskell's nice syntax and type features, but was also strict, and that had decent tool support and development resources, I think I'd be in nerd heaven.

6 March 2009

Daniel Burrows: Help make aptitude faster!

I'm working on aptitude's dependency resolver at the moment. Mostly I'm working on some changes to support better UI, but while I have all the code in my head I'd like to try to fix some of the nagging speed problems it has. I've already managed to clear up some simple but large problems with aptitude safe-upgrade, because I could get them to happen on my computer. The problem is, most of these problems don't happen on my computer. I have a half dozen or so specific ideas that might speed aptitude up, but I don't want to just randomly apply tweaks without some baseline tests that I can use to make sure that I'm actually making things better and not just perturbing the code for no reason. :-) Plus, seeing why aptitude is actually running slowly might help when it comes to finding ways to improve its performance. So, this is where you come in. If you are using lenny and aptitude takes a long time to resolve dependencies on your computer outside of aptitude safe-upgrade, I would like to hear from you. All you need to do is to run this command when aptitude is slow, before you upgrade any packages: 
$ aptitude-create-state-bundle aptitude-state.tar.bz2
Then, find a way to get the resulting large file to me. Email will work if your outgoing server accepts it; send to dburrows@algebraicthunk.net. That will tar up /etc/apt, /var/lib/apt, /var/cache/apt, and a few other directories: run aptitude-create-state-bundle --print-inputs to see exactly what it will include. Thanks! [UPDATE]: PS: please also include a brief description of the problem that you're seeing so I know what to test. Something like, e.g., when I run aptitude full-upgrade it takes thirty minutes to complete. [UPDATE]: PPS: of course you have to run aptitude-create-state-bundle before you actually upgrade any packages! i.e., you must type n at aptitude's command-line prompt or quit the program if you're using the curses interface. Otherwise, your state snapshot will just tell me what you upgraded to, which is only half the information I need (I also need the state of all packages prior to the upgrade).

4 March 2009

Daniel Burrows: Well, that was easier than expected

daniel@emurlahn:~/programming/aptitude/post-lenny$ time /usr/bin/aptitude -sy safe-upgrade   tail --lines 3
real    0m46.016s
user    0m41.639s
sys     0m0.172s
daniel@emurlahn:~/programming/aptitude/post-lenny$ time ./src/aptitude -sy safe-upgrade   tail --lines 3
real    0m5.336s
user    0m4.096s
sys     0m0.136s
More of the same coming, although I doubt I'll be able to top those statistics. And all it took was deleting twelve lines of code. Well, that and rewriting the entire resolver framework so that those twelve lines were no longer necessary, but who's counting? I'll post more details once I have everything in place and I'm sure it works as expected (most likely in a week or two).

21 February 2009

Daniel Burrows: Access control FAIL

Seen at work today while grovelling through some old code: 
class Foo
 
public:
  Foo(); // Do not use -- no body on purpose

12 February 2009

Daniel Burrows: Should we be scared of the future or not?

I recently received an email from Sigma Xi with various news items. One of them mentioned a Hawking quote:
Science fiction is useful both for stimulating the imagination and for diffusing fear of the future. -- Stephen Hawking?
Now, Sigma Xi is generally a pretty reputable organization, but I wondered: is a scientific organization really encouraging fear of the future? Mind, I think a certain degree of concern about the future is healthy, both individually and societally, and there is a long tradition in science fiction of writing novels about terrifying futures. But Sigma Xi is usually upbeat about the wonders of science and technological progress. Besides, there's a real dissonance in that quote between becoming more creating and imaginative, and becoming more afraid of the future. Maybe it was intentional, but it seems very out-of-place. So, I started to wonder whether this was really what Hawking said. Typing a few key words into Google, I looked at where else this quote showed up. According to the NSF, Sigma Xi's quote is correct.
According to renowned physicist Stephen Hawking, "science fiction is useful both for stimulating the imagination and for diffusing fear of the future." Interest in science fiction may affect the way people think about or relate to science...
ERIC, which is some sort of database of internal government reports, agrees:
According to renowned physicist Stephen Hawking, "science fiction is useful both for stimulating the imagination and for diffusing fear of the future." Indeed, several studies suggest that using science fiction movies as a teaching aid can improve both motivation and achievement.
So, maybe Hawking really does encourage being afraid of the future? Well, here was the third and final reference to this quote that I found, from (of all places) an unofficial transcript of a Larry King Live interview:
HAWKING: I think science fiction is useful, both for stimulating the imagination and for defusing fear of the future. But science fact can be even more amazing. Science fiction never suggested anything as strange as black holes.
So, it looks like this quote comes from a live TV interview. I don't know whether the different interpretations come from different transcripts (this is the only one I can find); defused and diffused are similar enough that it would be easy for either one to be reasonable -- and IIRC, he speaks through some sort of speech-generating machine, which would just make it easier to misunderstand him. But given that Hawking is a crazy-science-guy, I suspect that he would rather defuse fear of the future than spread it around (even if it became more diffuse as it was spread), and that the transcript above is what he meant to say. In which case, I wonder about the versions of the quote that I found everywhere else. Didn't anyone do a double-take when they were typing up those papers, Web pages and news items? Of course, there is a third possibility, which I rather like: the quote could be a play on words, a sort of slant pun (a phrase that I just invented, which is the funnier cousin of a slant rhyme). What's particularly cool about this particular ambiguity is that (a) both sentences are sensible, (b) they have related but opposite meanings, and (c) somewhat oddly, I agree with both of them: science fiction is good at defusing unwarranted fear of the future, while also diffusing fear where it's entirely warranted. I doubt Hawking was going for the double meaning, but it's a nice thought.
And for the inevitable people who can't figure out what this blog entry is about, here's a hint: diffuse:
Verb
  1. (transitive) To spread over or through as in air, water, or other matter, especially by fluid motion or passive means.
  2. (intransitive) To be spread over or through as in air, water, or other matter, especially by fluid motion or passive means.
defuse:
Verb
  1. To remove the fuse from a bomb, etc.
  2. To make something less dangerous, tense, or hostile.

8 February 2009

Daniel Burrows: Dear Lazyweb, does anyone understand job control?

I've been banging my head against this all day and it doesn't make any more sense than it did when I started. I have a need to write some code that can manage job control on a terminal. More specifically, I need to run a single process and stuff it into the background at will, so that it gets suspended when it tries to read from the terminal. So, there's a controller process and a subprocess process. 
 controller  ----------> subprocess
               manages
The man-pages and glibc info documentation make this look simple: disable TOSTOP if it's enabled via tcsetattr() so that the subprocess can write to the terminal, then start the subprocess in a new process group (by fork()ing and calling setpgid()). Once it's going, you can put it in the foreground or background by calling tcsetpgrp() to set the terminal's foreground process group to the subprocess (to put it in the foreground), or to the controller process (to put the subprocess in the background). Whenever the subprocess is in the background, it will be sent SIGTTIN if it tries to read from the terminal. I have everything working -- except for the very last sentence of that last paragraph. I can see my processes being put into the right process group, and I can see them going into the foreground and the background. For instance, here the first line is the controller process and the rest of the lines are the process group of the subprocess (this one started a few other processes). 
STAT CMD                           PID  PGID TPGID  PPID   SID
Ss+  ./src/aptitude              26989 26989 26989 26980 26989
S    ./src/aptitude              26991 26991 26989 26989 26989
S    /bin/sh -c /usr/sbin/dpkg-p 27027 26991 26989 26991 26989
S    /usr/bin/perl -w /usr/sbin/ 27028 26991 26989 27027 26989
Z    [dpkg-preconfigu] <defunct> 27034 26991 26989 27028 26989
S    /bin/sh -e /tmp/lynx-cur.co 27039 26991 26989 27028 26989
S    whiptail --backtitle Packag 27043 26991 26989 27028 26989
If I try to move the background process into the foreground, it is marked as the foreground process in this session: 
STAT CMD                           PID  PGID TPGID  PPID   SID
Ss   ./src/aptitude              26989 26989 26991 26980 26989
S+   ./src/aptitude              26991 26991 26991 26989 26989
S+   /bin/sh -c /usr/sbin/dpkg-p 27027 26991 26991 26991 26989
S+   /usr/bin/perl -w /usr/sbin/ 27028 26991 26991 27027 26989
Z+   [dpkg-preconfigu] <defunct> 27034 26991 26991 27028 26989
S+   /bin/sh -e /tmp/lynx-cur.co 27039 26991 26991 27028 26989
S+   whiptail --backtitle Packag 27043 26991 26991 27028 26989
Note that the background process is not suspended, even though it's busy reading from the tty. If I manually suspend the process group with kill -TTIN -26991, it stops as expected: 
STAT CMD                           PID  PGID TPGID  PPID   SID
Ss   ./src/aptitude              26989 26989 26991 26980 26989
T+   ./src/aptitude              26991 26991 26991 26989 26989
T+   /bin/sh -c /usr/sbin/dpkg-p 27027 26991 26991 26991 26989
T+   /usr/bin/perl -w /usr/sbin/ 27028 26991 26991 27027 26989
Z+   [dpkg-preconfigu] <defunct> 27034 26991 26991 27028 26989
T+   /bin/sh -e /tmp/lynx-cur.co 27039 26991 26991 27028 26989
T+   whiptail --backtitle Packag 27043 26991 26991 27028 26989
So the signal isn't being blocked or ignored. I can also run programs in a normal shell, outside my harness (e.g., links &) and watch them auto-suspend, but the same thing doesn't happen when I start them directly under my controller process. It's not even that they're starting as foreground processes: I start them in the background, and they never see a SIGTTIN. Does anyone have a clue what's going on? Hopefully it's as simple as a flag I have to set somewhere... Oh, and for extra fun, this is all happening inside a VTE terminal widget. That shouldn't make a difference (after all, it's what gnome-terminal uses, and SIGTTIN behaved as expected when I tested it there), but who knows, it might be relevant. [UPDATE] I did a little more testing: I was bringing the subprocess into the foreground whenever the terminal was active, so I couldn't try typing into the terminal while the subprocess was in the background. If I do that, it does get SIGTTIN and it suspends. So it looks like select() doesn't count as a read for the purposes of job control. I doubt there's much I can do to get around that. :-(

2 February 2009

Obey Arthur Liu: Debian Summer of Code 08 : Where are they now (part 3/3)

Welcome back for the last part of the reviews. You may want to look at the previous parts : part 1 and part 2. Jigdo-ivory, a JavaScript Jigdo client Presentation Debian CDs and DVDs take up a huge mount of space on download servers. Using jigdo to download those images can significantly reduce the amount of bandwidth and space needed on the central servers. Unfortunately, jigdo currently needs special client software to be downloaded/installed first. Adding support directly into a browser-based application could potentially make a very big difference for first-time users here. Jigdo was created in 2001. It allowed to create ISOs from .debs grabbed from regular mirrors. It eliminated the need to duplicate the entire contents of the package repository into ISO files for each release, or even more importantly, for weekly snapshots of testing/unstable/whatever. You may find the complete proposal from the student here. The original idea originated from the Debian-CD people, who wanted to explore ideas about creating a light web client. The project was mentored by Steve McIntyre, who developed a new version of the Jigdo tools, jigit, which is much more efficient. Student Dustin Rayner was a 5th year senior undergraduate student at the Oklahoma Christian University in Oklahoma City, Oklahoma. I studied Computer Engineering for 3 years as a Computer Engineering student before deciding to pursue a Mathematics and Computer Science degree. Result This project was unsuccessful due to numerous issues. First, because of an inadequate technical preparation of the original proposal. The Debian-CD people were too optimistic with the possibilities of Javascript. In the end, the copying and checksumming part of the Jigdo process were implemented but the checksumming (with a Javascript implementation of md5) was so slow that it was unusable (think 50kb/s on a regular laptop at full CPU charge). The student did the right thing to investigate Java and ActiveX but it was too late unfortunately and he ultimately lacked the experience and knowledge in the relevant technologies. If the proposal is tried again, the student would be requested to have much more experience with Java (and possibly ActiveX). Those would be much more efficient for the task, as they are the most used technologies among on-line anti-virus scanners, which have a workload somewhat similar to Jigdo. I could not find further public involvement of Dustin Rayner within Debian. Aptitude-gtk, usability and GTK+ GUI for the Aptitude package manager Presentation A GTK+ GUI for Aptitude that will work alongside improved current ncurses and command-line interfaces. This will offer an alternative to Synaptic with an interface design geared toward usability and advanced functionality. Debian currently supports multiple non-command-line package managers, the most used being Synaptic and Aptitude. Synaptic uses a GTK+ interface but offers no command-line mode. Aptitude offers a command-line mode but no X interface, although it offers a ncurses interface.
Comparing the interfaces of Synaptic and Aptitude reveal many design differences. Although Synaptic may be more accessible to beginners, Aptitude offers many interface behaviors and functions that are useful to the regular to advanced users : fully hyperlinked tabbed navigation between packages and versions of packages, mostly modeless interface, interactive dependency conflict resolver The proposal was introduced by the student in coordination with Daniel Burrows, the mentor and developer of Aptitude. Student Obey Arthur Liu was a 22 year old french student of Computer Science and Applied Mathematics at Grenoble Institute of Technology - ENSIMAG, in France. Did I mention that he s also yours truly ? If you want to know more, you might be interested in my previous post. Result This project was successful. The interface was mostly done and functional by the end of the summer. Daniel merged the code into the main post-lenny branch. Development is still ongoing and packages are released into Experimental. For further information, just read the rest of my blog. I could find some further public involvements of Obey Arthur Liu within Debian. Doh! Lintian for fuller automated setups Presentation lintian, the Debian package checker, at the moment presents possible problems in three categories: errors, warnings and informational messages. This leads to several problems, most importantly that the severity and certainty of a check can t be expressed separately. In the course of this project, the student should design and implement in lintian an improvement of the current situation, for example by using a two-letter code (one for certainty, one for severity). This project would make lintian errors much more fine-grained and help in maintaining pertinent quantitative analysis of package quality. The project was mentored by Marc Brockschmidt. The project proposal was commonly introduced by the Lintian team. Student UPDATED: Jord Polo Bard s has done a lot of work with translation in Catalan, his native tongue. He can usually be found on #debian-catalan. He also maintains a few packages as a DM. Result This project was successful. The classification was entirely done. Jord also helped with the new lintian.debian.org website. The Lintian team was very satisfied with the revamped errors list and new website. They have an immediate impact on packages quality reporting. Jord is still active within Debian, helping package a few games. Debexpo, a generic web-based package repository Presentation mentors.debian.net is currently a very specialized web-based repository that allows everybody to contribute software packages to Debian without the need to be a Debian Developer (or Debian Maintainer). It has successfully helped simplifying the sponsoring process in the last years. However it needs to be refactored and in the process should be turned into a generic piece of software that can be used for other Debian source/binary package repositories, too. Mentors is a very good initiative to recruit new packages maintainers (and needs your help!) and the software underlying it could be reused for many different purposes (think PPA). The project was mentored by Christoph Haas. The project proposal was commonly introduced by the mentors team. Student Jonny Lamb was a Computer Science student in the United Kingdom. He was already quite involved within Debian, maintaining a lot of significant packages. Result This project was successful. The whole proposal was perfectly executed. Jonny now continues to develop debexpo, with the mailing-lists and commit logs showing interesting activity. Of course, help for debexpo is appreciated to get it into full shape. Jonny has since become a Debian Developer (here is his AM report). Congratulations to him. It s nice to end on a nice note isn t it ? Now that we re done with the individual reports, I m going to write down my recommendations report. Hopefully it will help with next year s Summer of Code.

Next.