Search Results: "tar"

28 October 2007

Neil Williams: cross-building transition complete

The rewritten apt-cross using the code from NorthernCross in libcache-apt-perl has just migrated into testing. Hinted alongside the new dpkg-cross and libdebian-dpkgcross-perl and emdebian-tools, this completes the first part of the transition.
I've been waiting for the migration to upload new versions but there isn't a new transition to start - yet.
Next stage will be to complete the support for the new dpkg-cross in dpkg itself (439979) and migrate the cross-config.$arch files into dpkg-dev. That only then leaves the dpkg-cross script itself, starting the next transition. On course to remove dpkg-cross before Lenny.

Russell Coker: Xen and Security

I have previously posted about the difference between using a chroot and using SE Linux [1]. Theo de Raadt claims that virtualisation does not provide security benefits [2] based on the idea that the Xen hypervisor may have security related bugs. From my understanding of Xen a successful exploit of a Xen system with a Dom0 that is strictly used for running the DomU’s would usually start by gaining local root on one of the DomU instances. From there it is possible to launch an attack on the Xen Dom0. One example of this is the recent Xen exploit (CVE-2007-4993) [3] where hostile data in a grub.conf in a DomU could be used to execute privileged commands in the Dom0. Another possibility would be to gain root access to a DomU and then exploit a bug in the Xen API to take over the hypervisor (I am not aware of an example of this being implemented). A final possibility is available when using QEMU code to provide virtual hardware where an attacker could exploit QEMU bugs, an example of this is CVE-2007-0998 where a local user in a guest VM could read arbitrary files in the host [4] - it’s not clear from the advisory what level of access is required to exploit it (DomU-user, DomU-root, or remote VNC access). VNC is different from other virtual hardware in that the sys-admin of the virtual machine (who might be untrusted) needs to access it. Virtual block devices etc are only accessed by the DomU and Xen manages the back-end. The best reference in regard to these issues seems to be Tavis Ormandy’s paper about hostile virtualised environments [5]. Tavis found some vulnerabilities in the QEMU hardware emulation, and as QEMU code is used for a fully virtualised Xen installation it seems likely that Xen has some vulnerabilities in this regard. I think that it is generally recommended that for best security you don’t run fully virtualised systems. The remote-console type management tools are another potential avenue of attack for virtualised servers in the case where multiple users run virtual machines on the same host (hardware). I don’t think that this is an inherent weakness of virtualisation systems. When security is most important you have one sys-admin running all virtual machines - which incidentally seems to be the case for most implementations of Xen at the moment (although for management not security reasons). In ISP hosting type environments I doubt that a remote console system based on managing Xen DomU’s is going to be inherently any less secure than a typical remote console system for managing multiple discrete computers or blades. I have just scanned the Xen hypervisor source, the file include/asm-x86/hypercall.h has 18 entries for AMD64 and 17 for i386 while include/xen/hypercall.h has 18 entries. So it seems that there are 35 or 36 entry points to call the hypervisor, compared to 296 system calls on the i386 version of Linux (which includes the sys_socketcall system call which expands to many system calls). This seems to be one clear indication that the Linux kernel is inherently more complex (and therefore likely to have a higher incidence of security flaws) than the Xen hypervisor. Theo’s main claim seems to be that Xen is written by people who aren’t OpenBSD developers and who therefore aren’t able to write secure code. While I don’t agree with his strong position I have to note the fact that OpenBSD seems to have a better security history than any other multi-user kernel for which data is available. But consider a system running Xen with Linux in Dom0 and multiple para-virtualised OpenBSD DomU’s. If the Linux Dom0 has OpenSSH as the only service being run then the risk of compromise would be from OpenSSH, IP based bugs in the Linux kernel (either through the IP address used for SSH connections or for routing/bridging to the OpenBSD instances), and from someone who has cracked root on one of the OpenBSD instances and is attacking the hypervisor directly. Given that OpenSSH comes from the OpenBSD project it seems that the above scenario would only add the additional risk of an IP based Linux kernel attack. While a root compromise of an OpenBSD instance (consider that a typical OpenBSD system will run a lot of software that doesn’t come from the OpenBSD project - much of which won’t have a great security history) would only lose that instance unless the attacker can also exploit the hypervisor (which would be a much more difficult task than merely cracking some random daemon running as root that the sys-admin is forced to install). Is the benefit of having only one instance of OpenBSD cracked due to a bad daemon enough to outweigh the risk of a Linux IP stack? I’m sure that the OpenBSD people would consider that a better option would be OpenBSD in the Dom0 and in the DomU. In which case the risk of damage from a root compromise due to one badly written daemon that didn’t come from OpenBSD is limited to a single DomU unless the attacker also compromises the hypervisor. When working as a sys-admin I have been forced by management to install some daemons as root which were great risks to the security of the system, if I had the ability to install them in separate DomU’s I would have been able to significantly improve the security of the system. Another flaw in Theo’s position is that he seems to consider running a virtual machine as the replacement of multiple machines - which would be an obvious decrease in security. However in many cases the situation is that no more or less hardware is purchased, it is just used differently. If instead of a single server running several complex applications you have a Xen server running multiple DomU’s which each have a single application then things become much simpler and more secure. Upgrades can be performed on one DomU at a time which decreases the scope of failure (which often means that you only need one business unit to sign-off on the upgrade) and upgrades can be performed on an LVM snapshot (and rolled back with ease if they don’t succeed). A major problem with computer security is when managers fear problems caused by upgrades and prohibit their staff from applying security fixes. This combined with the fact that on a multiple DomU installation one application can be compromised without immediate loss of the others (which run in different DomU’s and require further effort by the attacker for a Xen compromise) provides a significant security benefit. It would be nice for security if every application could run on separate hardware, but even with blades this is not economically viable - not even for the biggest companies. I have converted several installations from a single overloaded and badly managed server to a Xen installation with multiple DomU’s. In all cases the DomU’s were easier to upgrade (and were upgraded more often) and the different applications and users were more isolated. Finally there is the possibility of using virtualisation to monitor the integrity of the system, Bill Broadley’s presentation from the 2007 IT Security Symposium [6] provides some interesting ideas about what can be done. It seems that having a single OpenBSD DomU running under a hypervisor (maybe Xen audited by the OpenBSD people) with an OpenBSD Dom0 would offer some significant benefits over a single OpenBSD instance.

[1] http://etbe.coker.com.au/2007/08/22/se-linux-vs-chroot/

[2] http://kerneltrap.org/OpenBSD/Virtualization_Security

[3] http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2007-4993

[4] http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2007-0998

[5] http://taviso.decsystem.org/virtsec.pdf

[6] http://shell.cse.ucdavis.edu/~bill/virt/virt.pdf

Marc 'HE' Brockschmidt: Why THEY don't want me to collect all left socks

Yoe seems to believe I see a cabal in Debian. Let me assure you: I don't - and if there would be a cabal, I would be part of it. I've seen most of Debian, played around with NM, the release team, have seen how our buildd network works (or, in the specific case of today: mostly works) and have close contact to people in all other core teams of the project. I still haven't seen a sign of a Cabal.

I believe that there are "clusters" of people who work better with each other, simply because they have known each other for years and developed trust, but that's nothing I would actually mark as problem. I do the same. Many of our core people have too much work to do, mostly because getting a job well done is usually reason to give you more tasks until your performance becomes abysmal. That doesn't mean that they are bad people or do bad things intentionally, but often, they don't allow people to help them - the usual reasoning is "new people need training, training needs time, time is what I don't have => no new people". Obviously, that argumentation is flawed in the long run.

Anyway, on to the more technical details:

The real reason for not adding new architectures is that not one of the people able to do so cares enough to do it. That doesn't sound as nice as "I don't really know why we don't do it", does it?

I haven't asked the security team to support backports, and I won't. volatile isn't under the supervision of the security team and works just fine.

Most porting machines aren't "down", they are "locked down". See the list of Debian machines (which is, BTW, seriously out of date). Hardware failure is something completely different (though we might want to think about using Debian funds to replace broken machines, if needed), but restricting access is a sign of an overloaded team of system administrators.

Dictating how a team should be changed is usually not a good way to do things - but there isn't another. A few of our core teams have failed to solve their problems on their own. Debian's job now isn't to stand by, watching and saying "He, that's bad. Maybe they'll fix the team in the next decade". Also, there are people experienced with the tasks of these core teams who are not part of them - you might want to use their experiences.

With my release team and unofficial buildd network hat on I will tell you that adding one buildd maintainer for those architectures that only have one at the moment would improve the situation a lot. Please don't assume that I only talk about stuff I have no idea of. Sometimes I actually know about things!

I don't think that we need to change our position regarding the GFDL, but I do believe that we should work on our relation to the FSF. Debian is the biggest free-as-in-free-as-in-free-speech distribution on the market, so the FSF should actually be interested in working together with us. Still, that doesn't happen, so we should perhaps start by having a discussion what the problems of each of these organisations is in the eyes of the other.

And, FWIW: I do hate conspiracy theories too. They come to life because people usually want to see the "bigger picture" - even if there is none and all of the weirdness in their environment can be explained by chance, human error and unknown social problems. And as I said - I don't believe there is a group of people making decisions that seem bad for Debian. I just believe a lot of people try to do their best for Debian - but they fail on a regular basis.

Andrew Stribblehill: Ryanair unstrangeness

I wrote a few months ago about how Ryanair had started charging for web check-in despite it being a cost saving for them — and how this was not the Ryanair that I understood. Well, I was proved right last week when they switched surcharges. Now you have to pay if you want to check in at the desk, and you get priority boarding and online check-in for free. If only they would start putting some effort into maintaining the uptime of the web check-in site, I would be happy. That said, it beats Aer Lingus which refuses to let you print out a replacement boarding pass if (for example), Firefox crashes on getting the PDF the first time around.

Wouter Verhelst: No, there is no cabal

I have a theory. There are two kinds of people in the world: those who see conspiracies everywhere, and those who don't. Personally, I'm part of the latter group; and it annoys me endlessly when people in the former group use conspiracy theories to explain what they don't understand—much in the same way that I don't really understand people who use religion to explain what they don't understand. An example of what I mean: Marc 'HE' Brockschmidt wrote about the stuff he'd like to change in Debian, since he feels many things go wrong. Now even though he doesn't mention the word 'Cabal' anywhere in there, it still reads as someone who sees conspiracy theories in Debian. No hard feelings, Marc, but...

It doesn't really help to 'enforce' addition of new architectures. Arguments like "I've heard that foo said bar" don't really help, either; please go find out for sure. If you can't or won't, then please do assume that those people who did care enough to find out know what they're doing.
Adding more official suites, such as volatile or backports, increases the burden on the security team and others. This isn't something that should be done lightly, and so "just" adding a CNAME won't really fix it.
I don't see why the practice of porting machines should be "resurrected". Yeah, sure, some of them are down, but we all know hardware sucks. That doesn't mean there's malicious intent or a change of policy or anything.
I'll agree that it's a bad thing that some people in core teams aren't delegates, especially when they don't do their job. There are many core teams of which the members aren't delegates, such as the security team, the listmasters, the maintainers of our default MTA, the d-i team, the kernel team, the X strike force, and a huge bunch of other teams; if and when they stop doing their jobs, that would be a serious problem.
I also really don't believe it's a good idea to go to a specific team that you don't have any experience with, and start dictating terms.
With my m68k buildd maintainer hat on, I can tell you that adding more buildd maintainers to any architecture is a bad idea. I'll gladly explain the details in more detail later, if there's any interest.
You do know about the draft updates to the FDL, right? We've put forward our position, now it's the FSF's turn; and though they considered updating the GPL to be of higher priority, they didn't ignore our opinion, which one can easily see if they read the discussion drafts.

I could go on, but I guess you got the point. Like I said, conspiracy theories annoy the hell out of me. Marc, it isn't personal, you're just the last in a long list. Oh, and BTW: comments to this post will be moderated away.

27 October 2007

Russell Coker: Senator Online

I’ve been asked for my opinion of senatoronline.org.au which claims to be Australia’s only internet-based political party. The claim may be correct depending on what you consider to be “Internet based“. Here is a copy of their platform from their web site:
Senator On-Line is not aligned to any other political party it is neither Liberal nor Labor. Senator On-Line ( SOL ) is a truly democratic party which will allow everyone on the Australian Electoral roll who has access to the internet to vote on every Bill put to Parliament and have its Senators vote in accordance with a clear majority view. We will be running candidates for the upcoming federal Upper House (Senate) elections. When a SOL senator is elected a web site will be developed which will provide:

Accurate information and balanced argument on each Bill and important issues

The vast majority of those registered on the Australian Electoral roll the chance to have their say by voting on bills and issues facing our country

A tally of all votes which will then count in Parliament

Each person on the Australian Electoral roll will be entitled to one vote and only be allowed to vote once on each bill or issue. SOL senators will have committed in writing to voting in line with the clear majority view of the SOL on-line voters. Senator On-Line will enable broader community involvement in the political process and the shaping of our country. If you like the concept, please register your details and tell others about SOL.
Now at first glance it sounds like a good idea, the Liberal party (which is similar in all the bad ways to the US Republican party) has demonstrated what happens when a party gets away with entirely ignoring the wishes of the voters. But there are three significant problems with the Senator Online idea. The first is the issue of providing “Accurate information and balanced argument on each Bill“. We have seen many media outlets claiming that there is a debate about global warming (the debate ended years ago, the vast majority of scientists have agreed that global warming existed for a long time) and now the same media outlets are claiming that there is a debate about whether it will cause any harm to us in the near future (ignoring all the dams that are running low). One of the aims of the democratic process is that representatives who spend all their time working on politics can fully analyse the issues and also gain access to resources that are not available to the average citizen, thus being able to make better informed decisions. The next problem is that it can degenerate into mob-rule. The idea of having a tabloid TV show being able to directly control votes in the senate is not an exciting one. The final problem is how to ensure that each citizen gets the opportunity to have exactly one vote, solving this requires good security and identity checks involving viewing photo-id. The checks used for voting (merely asserting your name and residential address) might be considered adequate for an election, but are grossly inadequate for online voting where one false registration allows many false votes. I think it would be more interesting to have a populist party started that campaigns for the issues that have the most impact on the majority of citizens. Issues such as the fact that a couple on the median income can’t afford the median house price [1], the lack of job security that is caused by recent industrial relations legislation, the price of health-care, and the fact that any car which is affordable on a single median income (after money is spent on rent and other basic living expenses) will lack basic safety features such as air-bags. While the Green party has decent policies on these issues they have many other policies to consider. A party that solely concentrated on allowing more people to have decent health care, not risk losing their job, own their own home, and drive a safe car would get a lot of interest from people who don’t normally take much notice of politics.

[1] http://etbe.coker.com.au/2007/08/06/housing-prices/

Martin Zobel-Helas: Will Marc become DPL?

Marc wrote:
It’s this time of the year again: The one where I look back at my involvement in the project and decide that I’ve been a member long enough, so I should probably get out. For Debian, there are two ways to do it: Either become MIA or become DPL. […] So, what do you think? Marc, do you care what i think? If yes, here is my opinion: You really should run for DPL. I have the hope that you will be a very good DPL. Side note: Did the campaigning period start already?
No Tags

Marc 'HE' Brockschmidt: "What to do when I've collected all left socks" or "what I want to change in Debian"

It's this time of the year again: The one where I look back at my involvement in the project and decide that I've been a member long enough, so I should probably get out. For Debian, there are two ways to do it: Either become MIA or become DPL. As I have too many real life connections to other DDs, I guess I can't really do the MIA thing, so I should probably get elected as DPL. Whenever I consider this way out, I think about all the issues that make me unhappy about Debian. This time, I've written them down. So, here's a list of things I would like to see and would work on if I had the power to do so:

Enforce addition of new architectures to the main archive. This includes kfreebsd-* and hurd-* - after all, we claim to be the universal OS. I've sometimes heard that our ftp mirrors wouldn't be able to handle the increased archive size - I don't believe that. We had the mirror split to work around this, not everyone is forced to mirror all architectures anymore. If actual Debian hardware should be a problem, we should use the money we collect to get the hardware we need.

Make backports.org official, just like volatile. This doesn't mean kicking out the current team and give the responsibility for bpo to the dysfunctional ftp-team, but simply move the service as it is into Debian. A start for that would be to add backports.debian.org as CNAME for backports.org and then add fitting hints to our release notes.

Resurrect the practice of porting machines. In ancient times, Debian had all kinds of machines available to all developers so that they could log in and debug architecture-specific porting problems. Nowadays, Debian collects hardware, Debian's hardware collects dust and Debian's BTS collects porting bugs. At the moment, most of these machines are not available to the developer body and access is granted to only a handful of persons besides DSA.

Fix the constitution, so that the ftp-team, the account managers, the buildd admins, the system administrators and all other teams that currently don't see themselves as delegates of the DPL can in some way be changed when they are not doing their job. There is currently a GR for that in preparation, see the excellent proposal from Josip Rodin for more information. When that is done, we can turn to actually fixing those teams, starting by giving full permissions to active people that currently have only restricted access (ie Joerg Jaspert should be able to modify the LDAP holding Debian's account data and should also get to act as full ftp-master, not only -assistant). There are also quite a few people experienced with the tasks of the other teams, so we should add some of these people soonish.

[Not really depending on DPL super cow powers:] Work on half-releases, like etch + 1/2. Besides adding a newer kernel for hardware support and updating the installer, we could think about adding backports packages to update desktop stuff. Many people seem to be interested in that.

Besides those things that I really miss in Debian, there are also a few things I would simply like to see done in the near future:

Add more buildd admins.

Resolve our conflicts with the FSF, get them to accept that we don't like the GFDL and then go on with lobbying for free software.

Improve our appearance on conferences by creating common Debian material. This means creating "official" ISO images for DVDs from time to time (just like they are done by people for a specific conference from time to time), creating official information material that can be translated by the l10n teams, discuss guidelines for the people behind the booth (like "Please don't wear a shirt which allows visitors to see what you've eaten in the last week") and give advice for presentations.

Clean up the DD list, so that we stop telling people that there are ~1000 DDs if only ~600 actually work on Debian. Yes, work on that is already on the way.

Create a place (a ML, wiki page, pseudo package) where Debian people can simply drop interesting ideas. Often, busy developers have great ideas for new projects, but lack the time to actually implement them. Usually, the inside knowledge is only needed to conceive the idea, not to implement it, so we should have a place where people with free time at their hands can find great ideas. It shouldn't be needed to actually describe every detail of your idea, it should be more like "Wouldn't it be cool to collect all left socks and then get to world domination by tying them together?". This could also be the basis for a bounty system (and of course for the GSoC people)

dpkg development - symbol-based shlibs look almost ready, but what happened to Wig&Pen alias dpkg-source v2?

At some point, we should discuss our vision of Debian. I guess we won't find a common one, but it would be interesting to see where people want to see Debian in 5 (or 10) years. Should we "just" be a supermarket so that CDDs and forks can be based on us? Should we work on reintegrating all those efforts into Debian proper? Is that even possible?

We should create a team (or, preferably, a DPL delegate) whose sole task is to communicate with the rest of the Debian ecosystem, so that common problems can be noticed, discussed and solved. I have to admit that I have lost track and couldn't say how many distributions are using Debian as their basis nowadays, but there seem to be quite a lot. I wish for a place where I can find such information (or at least a person who I can ask about it).

So, what do you think? Are all of those ideas crap or should we perhaps try to implement them? Have I missed something important?

Lior Kaplan: Status of the Firefox bug triage (and what I learned from triaging bugs)

The Firefox bug triage started a month ago, with the goal of verifying the relevance of bugs reported against the firefox package, which isn’t present any more in unstable. From the 324 bugs I started with, 37 have been closed and 85 were verified. The latter were reassigned to the Iceweasel package and got the necessary version info. 165 bugs are still pending for verification. Additional 37 bugs are forwarded to the Mozilla bugzilla, most of them are open for a long time. Help with forwarding more upstream bug to the bugzilla is always welcome. The status of the triage is available at the firefox bugs’ page. During my recent bug triages (openoffice.org, cvs, firefox), I think we need to add a “be kind to QA people” section along with the “be kind to porters” one. I encountered a few people thought the goal of the triage was to bother them than to get bugs solved (or at least ease the maintainer to attend to them). One even got an angry reply back after complaining too much about getting an extra mail:

I must say that I don’t understand your lake of patience. So you got another mail… we all get dozens of them everyday. I invest time in doing the ugly QA work, and I don’t I deserve this attitude.

But on the other hand, I had a few people thanking for the fact that someone invested time in their reports, and others that were patient regarding a few mistakes I made. One conclusion I have from the triage process - our DDs don’t use versioning info enough. This makes going over reports harder and more time consuming. It also harder for the maintainer to figure out if the bug is still relevant for a new version.

Russell Coker: A Traditional Approach to an IT Career

I have just read Career Development for Geeks [1] by Erik de Castro Lopo [2]. It makes some interesting points about a traditional approach to an IT career. The path I followed for most of my career (after I had a few years experience) was to work as a contractor and happily leave jobs without having anything else lined up. Erik suggests getting Engineers rather than managers to give references, it’s an interesting idea. Engineers can give better references about quality of work, while managers can speak on behalf of their employer (in theory). In practice a sane manager won’t give a bad review for legal reasons so the value of a reference from a manager is probably limited. Of course one problem with reviews is that I have never heard of a recruiting agent or employer actually verifying the ID of a reference. I could be listed on a friend’s CV as a senior manager in a multi-national company (which doesn’t have a major Australian presence) and give a good review and it seems unlikely that the typical recruiter would work it out. For someone with plenty of spare time and no significant assets (no risk of being sued) it could be entertaining to apply for a bunch of positions that they are not qualified for with friends using pre-paid mobile phones to give references. This could be done as a documentary on corporate hiring practices, or to simply try and get the highest paid job possible. Based on observing some former colleagues it seems that little skill is required to get a job and that when people do really badly they get paid for at least a few months. I am constantly amazed when reading reports about so-called “con artists” who commit crimes for what are often small amounts of money. Getting an income significantly greater than average without knowing anything about how to do the work is very common and is never treated as fraud (the classic example was a former colleague who wanted to write his own encryption algorithm but who didn’t even know about binary and therefore couldn’t use operations such as XOR). Erik’s main suggestion for dealing with recruiting agents is to talk about projet management issues (recruiters don’t understand technology). My way of dealing with them has been to assure them that I know it all and tell them to forward my CV to the client. Another suggestion is to learn new skills and diversify your skills. I don’t supoprt this because I believe that the majority of people who read my blog are significantly more skillful than the typical programmer. If an area of technology starts to go out of fashion then it’s the people with the least skills who suffer the most. If you are good at your work and enjoy it then it shouldn’t matter much if people around you are being laid off. Of course to rely on this you have to be working in a reasonably large field. For example if you develop software in a language which has hundreds of programmers then you may be at risk, but if there are tens of thousands of programmers using the language then you only need to be in the most skillful 10% to be assured of employment for the next decade or two. That said there are benefits to moving around, meeting new people, and working on new contracts. One thing you don’t want is to have 10 years of experience which are the same year repeated 10 times! Update: Here is a paper Erik submitted to OSDC on the same topic [3]. Mostly the advice is the same but with more detail and in a better format for reading.

[1] http://www.mega-nerd.com/Career-dev/career-dev.pdf

[2] http://www.mega-nerd.com/erikd/Blog

[3] http://www.mega-nerd.com/erikd/cddg.pdf

Jan Wagner: software depencies

First the good thing: I updated wordpress to 2.3. Now the bad thing: I updated wordpress to 2.3. With 2.3 there where many infrastructure changes. For a complete list, see the Release Announcement. The most extensions breaking change was the implementation of native tag support. Referencing to Not working Plugin List Ultimate Tag Warrior and Extended Live Archives are problematic. Both caused troubles on update, but there is a fix for ELA and one for UTW. By reviewing the informations for writing this articel, I noticed that I’m running fine with the Ultimate Tag Warrior Bugfix Release provided by Christine from the Internet.
My theme K2 was needing also an update, which caused the style to be broken. So waiting for a fix did take little longer, but when reviewing it, I noticed that only a css class name did need an update.
This all teaches me, when using a project with much depencies, it will be a mess when updating, as long as isn’t all provided by one source. It’s some kind of using a project “still in development”.
Update: While we are at “still in development” .. I updated to 2.3.1-RC1 some seconds ago. :)

26 October 2007

Biella Coleman: Should it stay or should it go?

What a week. Most notably, I am still in a state of partial disbelief that my case with Blue Cross Blue Shield is now on a path toward resolution and I don t have do battle with them, or at least in the same vigorous, soul-crushing way I have been engaged in since May (they still have not paid all the bills but at least part of them are resolved). But the very same day that I received the good news about BCBS, I received not so good news from my dermatologist about the black-blue mole that lies right under my left eye. He wants it gone, removed. So my whole fight with BCBS concerned claims over pre-existing conditions related to my moles. My claim was that I never had a problem with them until I had BCBS insurance and that is indeed the case. Soon after I first started having problems, I went on mole patrol and found a suspicious mole on my sister, which turned out to be melanoma. Now whenever I see a dermatologist, they all say, given my family history, given I grew up in the tropics, and given the odd size and very dark color of my mole, that it has to go. In fact the last dermatologist said with the firmest of tones: there is no doubt in my mind that it must be removed. The words stung because, well, I am pretty attached to the mole. I have lived with it for my whole life and it is fundamentally, even if somewhat mysteriously, linked to my sense of self. If it goes, I would feel like a very important part of me would also vanish. On more prosaic grounds, I can t help think that part of the drive to remove such a spot is just part of somewhat unhealthy social trend toward measuring most everything in finer and finder gradations of risk. Given that this mole is on my face and I see my face at least 2x a day in the mirror, I would probably catch anything if it started to change but given our risk society it is unsurprising that every American dermatologist keeps saying it is time to bid adieu to it. Given my background in medical anthropology I can t help conceptualize all this advice within these much larger social trends and then I go back and forth endlessly in my head as to whether I should remove it because it is safe and right and sound or whether I am caving in to somewhat irrational trend toward the risk management of everything. Finally I fear my insurance company again. Though I have a good job and decent insurance, I may not have good enough insurance for what this procedure entails.
Because of the size and location of the mole (large and close to my eye), if removed it must be done by a plastic surgeon. And plastic surgery in the US and plastic surgery in NYC costs a very pretty penny. While I would plan on getting pre-approval for the surgery, I somehow have the feeling that they will not pay the full cost and I am going to be left with another bill. And I am trying to stop the unnecessary onslaught of bills. So should it stay or should it go? I bet that I will lean toward the latter but not without a lot more thoughts, internal cries of resistance, and sense that even if I decide it is the best thing, I will lose a small piece of myself once removed

Robert Collins: 26 Oct 2007

Flying to the US for UDS in Boston... and this time no dreaded AAAA's. Seems the US doesn't hate me quite so much, much more pleasant this time. Still, being fingerprinted for entering a country is rather irritating, back home we only do that to criminals.

Ian Murdock: Fighting the good fight

Jonathan Schwartz: “[L]ater this week, we’re going to use our defensive portfolio to respond to Network Appliance, filing a comprehensive reciprocal suit. As a part of this suit, we are requesting a permanent injunction to remove all of their filer products from the marketplace, and are examining the original NFS license - on which Network Appliance was started. By opting to litigate vs. innovate, they are disrupting their customers and employees across the world. “In addition to seeking the removal of their products from the marketplace, we will be going after sizable monetary damages. And I am committing that Sun will donate half of those proceeds to the leading institutions promoting free software and patent reform (in specific, The Software Freedom Law Center and the Peer to Patent initiative), and to the legal defense of free software innovators. We will continue to fund the aggressive reexamination of spurious patents used against the community (which we’ve been doing behind the scenes on behalf of several open source innovators). Whatever’s left over will fuel a venture fund fostering innovation in the free software community.” Bravo.

25 October 2007

Neil Williams: Emdebian root filesystem installable

Been a bit busy for the last week or so. Apologies to those who were waiting for other things but I've been dedicating my time to Emdebian in preparation for CELF in Linz, Austria which is now less than a week away.
I spent three days as a guest of Wookey and his employer, TobyChurchill, in Cambridge. TCL are sponsoring Emdebian at CELF and also assisted with this work session so that Wookey and I could work together to fix the root filesystem. I'm very grateful to TCL for donating a balloon3 board that I can use to test all the new packages I'm cross building for Emdebian. Various bugs in emdebian-tools have been fixed, allowing the rootfs scripts to be tested and improved.
Other improvements include outline support for machine-specific customisations of the rootfs from within the tools. Support includes per machine-type and per machine-variant combinations and at three points in the rootfs generation process: package selection, post first_stage_install and post_second_stage_install. I hope to create some extra documentation for those who want to go the whole hog and write a new debootstrap suite-script.
The rootfs uses busybox, dpkg, apt and includes support for ntpdate and cdebconf. Current tarballs are about 14Mb to download, installing to about 25Mb. Two main reasons: Emdebian doesn't recognise "Essential:" and Emdebian doesn't install perl.
I have some more testing to do tomorrow but the current Emdebian target repository does have packages sufficient for a usable root filesystem within a chroot. The remaining stage is combining the kernel modules into the rootfs to make a bootable system.
Support for these new features will be in emdebian-tools 0.4.3, due for upload to Emdebian tomorrow, hopefully. All Debian users of emdebian-tools have access to the emdebian repository so the upgrade will be available immediately after the upload.
Watch the debian-embedded list for full details of the new features (as well as the expanded manpages).

Daniel Jacobowitz: An adventure in Unicode PC fonts

I have always had some trouble with my eyes. One bizarre side effect of this is that switching fonts is very unpleasant. I recently got fed up with fighting things to live in the stone age of pre-Unicode fonts and locales and started hunting for a new console font. First I looked for nice new ones already designed for Unicode, with a selection of accented characters and so forth. The nicest I found was Terminus, but I’ve had trouble adjusting to it. So eventually I decided to stick with the font I’d been using - at least for now. I have been using the same font for my consoles since before 1997. This is, or at least was, the default Linux console font back when I did all my work on a VT on Linux/PPC: /usr/share/consolefonts/default8x16.psf.gz. When I started using X routinely I missed it quite a lot. Mostly that was because I used BitchX, and BitchX is written for an IBM437 code page set of line drawing characters. So I opened up that console font in xmbdfed and converted it to a PCF file. If you have Debian’s BitchX package installed, you’ve got a copy of this work in /usr/share/fonts/X11/misc/default8x16.pcf.gz. So I’ve done the same trick again. Here’s the steps I followed:

Open the PCF font file in FontForge.
Choose Add Enconding Name… from the Encoding menu.
Enter IBM437, which is the iconv name for the traditional PC ROM character set.
Choose Force Encoding and select IBM437.
Choose Reencode and select ISO 10646 (BMP).
Choose Generate Fonts… and save it as a BDF file, since PCF wasn’t an option.
Create a fonts.dir file by running mkfontdir.
Add the directory to the font path using xset +fp.
Attempt to start an xterm.

Actually, after all that I was having trouble with the font’s name and eventually getting scary resource errors from X. I opened the font in gbdfed and saved it, which added some attributes and modified the bitmaps. Then I tried again and it worked. And now, I can use a UTF-8 locale and gcc’s error messages come out with foo’ (”angled” quotes) instead of display corruption! The resulting font is uni8×16, if anyone else would like it. I have switched to using it for all my terminals other than the one I run BitchX in; BitchX still needs the IBM437 version in order to display correctly. Of course, as soon as I finished this Paul Brook informed me that Konsole includes basically the same sort of font. I do not know the origins of KDE’s console8×16, but I do know that it’s a slightly different font (the @ has a smaller center, for example). And I know that I’m mighty annoyed it didn’t get added to my font path correctly when I installed konsole ages ago, since I might otherwise have found it!

Matthew Garrett

Linux-hero wrote about how Ubuntu kills your hard drive. The situation is somewhat less clear than you might think from the article, but the basic takeaway message is that Ubuntu doesn't touch your hard drive power management settings by default. In almost all cases, it's more likely to be your BIOS or the firmware on your hard drive.

The script that's executed when you plug or unplug your laptop is /etc/acpi/power.sh. The relevant sections are:

function laptop_mode_enable  
...
    $HDPARM -S $SPINDOWN_TIME /dev/$drive 2>/dev/null
    $HDPARM -B 1 /dev/$drive 2>/dev/null

That is, when the laptop_mode_enable function is called, we set the drive power parameters. Now, by default laptop_mode_enable isn't called:

if [ x$ENABLE_LAPTOP_MODE = xtrue ]; then
    (sleep 5 && laptop_mode_enable)&
fi

because ENABLE_LAPTOP_MODE is false in the default install (check /etc/default/acpi-support). This means that, by default, we do not alter the hard drive power settings. In other words, the APM settings that your drive is using in Ubuntu are the ones that your BIOS programmed into it when the computer started. This is supported by the fact that people see this issue after resuming from suspend. We don't touch the hard drive settings at that point, so the only way it can occur is if your BIOS or drive default to this behaviour.

If you enable laptop mode, then we will enable aggressive power management on the drive and that may lead to some reduction in hard drive lifespan. That's a fairly inevitable consequence of laptop mode, since it only makes sense if the laptop enages in aggressive power management. But, as I said, that's not the default behaviour of Ubuntu.

There's certainly an argument that we should work around BIOSes, but in general our assumption has been that your hardware manufacturer has a better idea what your computer is capable of than we do. If a laptop manufacturer configures your drive to save power at the cost of life expectancy, then that's probably something you should ask your laptop manufacturer about.

24 October 2007

Enrico Zini: axi-query-expand

apt-xapian-index: suggesting new terms for improving the query I've recently posted an introduction of apt-xapian-index, an example of how to query it and a way to add simple result filters to the query. Today I'll show how to use a very interesting feature of Xapian: computing a list of the best terms to use to improve the query. I'll expand apt-query-pkgtype.py to write, after the results, the suggested terms. If you feel like reimplementing my examples in another language, let me know and I'll include it to the post. Let's start with a few definitions taken from the Xapian glossary:

RSet (Relevance Set) The Relevance Set (RSet) is the set of documents which have been marked by the user as relevant. They can be used to suggest terms that the user may want to add to the query (these terms form an ESet), and also to adjust term weights to reorder query results. ESet (Expand Set) The Expand Set (ESet) is a ranked list of terms that could be used to expand the original query. These terms are those which are statistically good differentiators between relevant and non-relevant documents.

What I'm doing now is: after printing the normal result, I build an rset with all the results. For better results one can ask the user to pick the results they like (make a GUI with that: it should rock!), but starting with the top results in the RSet is a good enough default:

# Now, we ask Xapian what are the terms in the index that are most relevant to
# this search.  This can be used to suggest to the user the most useful ways of
# refining the search.
# Select the first 10 documents as the key ones to use to compute relevant
# terms (matches is the mset returned by a normal query)
rset = xapian.RSet()
for m in matches:
    rset.add_document(m[xapian.MSET_DID])

Then I use the RSet to compute the ESet, and display the results to the user:

# This is the "Expansion set" for the search: the 10 most relevant terms
eset = enquire.get_eset(10, rset)
# Print it out.  Note that some terms have a prefix from the database: can we
# filter them out?  Indeed: Xapian allow to give a filter to get_eset.
# Read on...
print
print "Terms that could improve the search:",
print ", ".join(["%s (%.2f%%)" % (res.term, res.weight) for res in eset])

You can also abuse this feature to show what are the tags that are most related to the search results. This allows you to turn a search based on keywords to a search based on semantic attributes, which would be an absolutely stunning feature in a GUI. We can do it thanks to Xapian allowing to specify a filter for the output of get_eset. This filter filters out all the keywords that are not tags, or that were in the list of query terms:

class Filter(xapian.ExpandDecider):
    def __call__(self, term):
        "Return true if we want the term, else false"
        return term[:2] == "XT"

I just rerun get_eset like above, but adding the filter, to show only a suggestion of tags:

# This is the "Expansion set" for the search: the 10 most relevant terms that
# match the filter
eset = enquire.get_eset(10, rset, Filter())
# Print out the resulting tags
print
print "Tags that could improve the search:",
print ", ".join(["%s (%.2f%%)" % (res.term[2:], res.weight) for res in eset])
sys.exit(0)

This is it. You can use the wsvn interface to get to the full source code and the module it uses. Try running it as:

axi-query-expand.py --type=gui image editor
axi-query-expand.py --type=cmdline image editor
axi-query-expand.py --type=game image editor

you will see that as the results change, the suggestions change as well. I'm happy to see that Xapian often suggests tags: this means that tags are useful discriminators in search results, and that all the work done on Debtags has been a good work. Notice that the suggestions change as you change any of the terms, the custom filter, and the tags you put in the search. Try to imagine how such a thing could improve Synaptic. What if I told you that Xapian is fast enough that all of this can happen live while you type? Next in the series: searching for similar packages.

Lucas Nussbaum: Better mentors.debian.net/REVU to improve our sponsorship workflow?

I recently sponsored several uploads, and was asked to sponsor even more uploads, and that got me thinking about our sponsorship workflow. It’s a clear bottleneck in Debian, and discourages many new contributors, which obviously sucks. It’s important to note that the same problems exist in Ubuntu (their equivalent to mentors.debian.net is named REVU). The best way to improve the process would be to have packages of better quality when a DD first look at them. They would be more likely to be uploaded right away, which frees time for other packages. I think that there’s a lot of room for improvement in the current mentors.debian.net implementation. Here is a small list of features I would like to see.

Integration of some QA tests in mentors, as soon as the package is uploaded:
- does the package build cleanly?
- piuparts test?
- lintian/linda checks?
Better list of packages awaiting sponsors, with info including:
- does the package fixes bugs (number of bugs fixed per severity)?
- is that package already in Debian?
- is that package a new upstream version?
- popcon score
- how long has the package been waiting?
This would allow potential sponsors to prioritize requests.
A commenting system, for each package, so comments for rejected packages are not lost, and the next potential sponsor can double-check
A way for sponsors to mark some sponsorees as “friends”, so it’s easy to find all the requests from people I “trust” (for some definitions of “trust” ;)
Maybe, a scoring system, where providing good comments on other’s packages would make you win “karma points”, and improve your classification, which could later be used by sponsors to choose what they are going to sponsor next.

The good thing with this whole list of features is that everybody can help. So, if you are looking for a sponsor and want to help solve this problem, start coding now ;) And if you need me to create nicenameforyourservice.debian.net, just ping me. There’s probably some code to steal from svnbuildstat.debian.net, so contacting its developers would be a good idea.

Steve Kemp: With many a winding turn

GNU Screen rocks, in general, but recently I've been using it a lot for custom applications and have discovered a pair of annoying bugs. If you're not familiar with it then please read this GNU Screen tutorial - it really is worth getting to know! Anyway onto the bugs:

Session names may not start with numbers.
Session names must be globally unique.

The two are related, but I'm not yet sure whether I should report bugs against the Debian package and the code is sufficiently cryptic that I cannot create a fix yet. Taking the bugs in reverse order please try this:

screen -S foo
[detach]
screen -S foo2
[detach]

At this point you should have two screen sessions "foo" and "foo2". You should be able to attach to them by running "screen -R foo" or "screen -R foo2". Lets try that:

skx@vain:~$ screen -R foo
There are several suitable screens on:
        22317.foo       (Detached)
        22342.foo2      (Detached)
Type "screen [-d] -r [pid.]tty.host" to resume one of them.

Even though foo should be sufficient to identify a unique screen, the first one, it doesn't let you attach. Nasty. (Yes, you can attach to it if you use the number/number+name:

screen -R  22317.foo

The second issue is related. Create a screen session with "screen -S 222". Now try to attach to it with "screen -R 222" - instead of attaching it gives you a brand new screen. Ugh.

Next.

Previous.