Search Results: "xam"

23 July 2007

Simon Richter: Package management and components (aka "our users and free software")

Ian, I am not sure that “good package management” is a real selling point for anyone but system administrators. Home users do not care, as there is not that much difference between double-clicking an executable (that pops up a window, asks a few questions and then installs the software) and selecting a check-box in a list and clicking “apply” (followed by window, questions, etc.). In a business setting the users usually do not have permission to install software anyway, so they could not care less how it was installed, the only thing important is that it is there. In my opinion, the single technology that has had the most impact would be COM, the “Component Object Model”, because this is what the ISVs want: an easy way to define a (binary) interface that will allow components from different vendors to work together, ideally in the same virtual address space for maximum efficiency. The registry provides the necessary lookup functionality, so that you can construct an object of a given type by giving its class name or GUID to CoCreateInstance, and you get something back that provides the interfaces that this class implements, which is either the object directly or an RPC proxy. The latter happens, for example, when a multithreaded application wants to call methods in an object that is not threadsafe (COM will then serialize the calls) or when the object is provided by a system service (where we have dbus). The fact that the only thing you can rely on is the defined interface (a set of methods and data objects that map to getter/setter methods) makes it necessary to define the interface first and invest lots of Interfaces become immutable as soon as they are published, incompatible changes need a new interface name (that's where all the interface names ending in …2 or …3 come from). Under Linux, the closest thing is GNOME and their use of CORBA. If you were to write an application that runs within the GNOME framework, and if you wanted third party vendors to be able to plug into it, then you would probably publish an IDL file for it. Lots of “if”s there. Licensing is also an issue here. One thing the GPL fails to address, even in version 3, is the integration of components without linking into the same address space (which ORBit never does IIRC). Technically, the component provider and the component user communicate via sockets using a well-known protocol, so the component user is not bound by the GPL; the only thing that could be argued is that the IDL file defining the interface would be GPLd (although most jurisdictions consider interfaces as not copyrightable and/or permit “reengineering” for interoperability purposes). Some people might consider this “cheating” around the GPL, as any GPL'd program or library providing components could be used freely by any closed source application as long as the source code to the GPL'd portions was made available. Last (but not least) it is a matter of developer motivation. Third party vendors are, for most of us, not our target audience, and properly supporting them is going to be quite a bit of work. I've written about mixing different C libraries in the same executable image before, as a “nice to have” item. For components that should share the same image, like COM does, it is an absolute must, and COM goes to great lengths to make sure this actually works. For example, allocating memory that is to be passed to a COM method must be allocated using midl_user_alloc() from the RTC DLL and freed using midl_user_free() from the same; as this is a system library and compiler vendors are forbidden from bringing their own, everyone uses the same allocator pair (that internally just calls malloc() and free() from the Microsoft C library, but that is an implementation detail you are not supposed to know). On the Linux side, people do not care much about matching allocator functions, because in essence it all ends up in glibc's malloc()/free() anyway. When glib introduced their own allocators and started using them for their own structures, a number of applications started crashing unless you set a certain environment variable that told glib to pass everything through to glibc. Personally, I've spent the last days compiling toolchains in order to provide compiler packages that work “out of the box” on at least Debian etch, lenny and sid, and Ubuntu dapper, edgy, feisty and gutsy. Right now, I feel a lot like shipping statically linked binaries. Where Windows has mostly gotten out of the “DLL hell”, Linux has built its own variant of it:
The Windows DLL hell
is a hot place. Everyone is bringing their own grill lighter they believe does not cause cancer, starts a small fire and begins to roast marshmallows. The problem there is that the floor is covered with a sticky mixture of lighter fluid and everything is burning.
The Linux DLL hell
on the contrary is freezing cold. It has small isolated fires where a few friends are sitting, and they suddenly decide to move on and pour water over the fire and your marshmallows without asking first. Getting into a group is difficult, and if you manage it, all it means is that you are told to pick up your marshmallows before they extinguish the fire.
At this point, I'm going to be comment- and trackback-whoring; instead of answering the obvious question, I'm going to leave it open: Would a COM-style componentization model work in the Linux world, and attract software vendors, provided we can put out stable APIs and support them properly, or is that a red herring?

20 July 2007

Aigars Mahinovs: Almost a master

On Wednesday I had a viva for my Master by Research thesis on “Text classification by quality”. After a 15 minute presentation and almost two hours of questions from two of my examiners, I was told that I will receive my Masters degree (after minor corrections to the thesis). It is a great relief and a great reason to celebrate! In order of celebration I will embrace consumerism and get me two things: World of Warcraft: Burning Crusade expansion and a cat. Next week I will be going around animal shelters and cat owners in Riga in a search for a 1-2 months old black, male kitten. It has been my dream to have a cat again since my last cat died of old age 3 years ago. Unfortunately I lived in dormitories where they did not allow pets or in UK. But now, I can finally get one. I am very exited about that :)

Gerfried Fuchs: msgmerge --previous and poedit

Since quite a while gettext has support for a feature which I was waiting for since ages: --previous. It means that when a string gets marked fuzzy it adds the previous original string as a comment so one is able to see the difference which led to the fuzzy marking. This makes translation of po files much much more comfortable because one can easily find out now what has changed. Let me give you a reallife example:
#: data/core/help.cfg:331
#, fuzzy
#  msgid "race^Mechanical"
msgid "trait^Mechanical"
msgstr "Kriegsgerät"
With out the previous string (in the "# " line) you would propably be puzzled about what has changed because the string gets stripped off the everything before the ^ for displaying it in this case. Though, not all translation tools behave correctly. One of them is unfortunately poedit. It moves the previous string to a wrong place and thus breaks the parsing of the file. A bugreport has been filed about it—but unfortunately it also means that upstream projects that already support (like Wesnoth) it are considering disabling it again because of this. So I wrote quickly a perl script that is able to fix that for you: pofix.pl. It's quite simple, reads from stdin and writes to stdout.
Go and support --previous before your competitors do it and hijack all your translators! ;)

18 July 2007

Bastian Venthur: Improved filtering for rng

I’ve just uploaded a new version of reportbug-ng including an Italian and a Hebrew pony and a cool enhancement to the buglist filter: Now the filter doesn’t try to match the whole string anymore but splits the input by whitespaces and tries to match all words. Now you can even exclude words by prepending a leading minus to it. The Syntax for the filter is: [-]keyword [[-]keyword2] ... Multiple keywords are separated by whitespaces and can have a leading minus. A leading minus means “exclude” the whitespaces represent the logical AND. Some Examples: Please note the the filter does look into the actual bugreports (the fulltext), it currently tries to match against the packagename, the bugnumber, the summary, the status and the severity. The best thing about this new filter? It’s not even slower than the old one, the only delay you’ll notice is when filtering packages with a huge amount of bugs (like wnpp) and even there the filter just needs a fraction of a second. The delay of a few secons comes acutally from the QTable when I remove all the rows not matching the filter. QTable.hideRow(int) and -showRow(int) seem to be very expensive even when I temporary disable all graphic updates for this widget during this operation. If someone knows a trick to circumvent this, please tell me (you may get a pony…). This feature isn’t spectacular, but it should make filtering big buglists a lot easier.

Fabio M. Di Nitto: More LDOM love

I finally decided to publish the few hacks I am using to run Linux on SUN's LDOM.
They are not pretty but they do the job done for me.
I take 0 responsibility for them so use them at your own risk.
Use this HOWTO only if you are dealing to mess with your system. Do *NOT* use for production systems if you do not understand what you are doing Step 1: read all the possible LDOM documentation that SUN made available for you. Step 2: understand that this is all experimental software. Step 3: repeat 1 and 2 a few times. LDOM server in Solaris allows to export 3 kind of block devices (file, partition/slice, entire disk) to the guest and those are seen as virtual disks.
The guest cannot really feel the difference but it is important to understand the way in which the LDOM software plays with them.
I did *not* test the export for partition/slice and my hacks do not know how to cope with it. You have been warned. The problem: LDOM virtual disk daemon mangles partition tables In an attempt to make sure that the partition tables exported to the guests are sane, the daemon ends up trashing valid ones because the check it forces to the devices is too strict and it behaves differently if the device is a file, a partition/slice or an entire disk.
The ldm bind operation is the dangerous one to perform and that can kill your partition table for good.
How ldm bind operation work: - for entire disks: * check device partition table
* fix device partition table if check fails
* lock device
* export device to guest
* store status of device once it is validated and do not check it ever again
- for files: * check device partition table
* fix device partition table if check fails
* export device to guest
Remember that I did *not* test export of a slice and that my hack does not take it into account. You have been warned twice. Limitations of the workaround First of all I assume that you installed LDOM 1.0 in the standard path (/opt/SUNWldm/) otherwise you might need to change the script manually to match an alternate path.
Each guest is using the same kind of devices. The workaround does not yet how to cope with exports of files and partitions to the same guest.
You know how to use dd and some other basic tools.
If you start playing with bind/unbind operations manually and things break, you keep all pieces.
How to This list of operation sounds extremely complex, but it is not and it is required only at the first install time and possibly to update your partition table backups if you decide to change them. The changes to the ldom init script will take to restore the last known to work partion table on each reboot. NOTE: if you fail to follow these instruction and do not use valid configuration entries or backup partitions, LDOM might not start anymore or even core dump. TAKE EXTREME CARE (and learn how to reboot into factory-defaults at SC/ALOM)

1 - get the few files from here. This html file is there too for your convenience.
2 - ldmd_start is the newly modified init script for LDOM. It should be placed in /opt/SUNWldm/bin and made executable. Note that if there is no configuration file or the configuration file is empty nothing will happen and you will be running the exact same script as the original. ldmd_start.orig and .diff are there for your convenience to check that I did not add a root kit ;).
3 - ldom.linux is an example configuration file that you want to place in /etc.
4 - mkdir /.ldom.linux , we will use this directory to store the backup partition tables.
5 - decide what kind of device you want to export.
Follow steps 6 and 7 if you decided to use a full disk as device and skip to 8 if you are using a file 6 - Use the format command from Solaris and make sure to label and format the disk.
7 - Take a copy of the clean partition table.
Example: dd if=/path/to/device of=/.ldom.linux/$guestname-$(basename /path/to/device).format count=1 bs=512
So let say that your guest is called foo1 and you are exporting /dev/rdsk/c1t1d0s2 you will issue the following command:
dd if=/dev/rdsk/c1t1d0s2 of=/.ldom.linux/foo1-c1t1d0s2.format count=1 bs=512
Continue from here if you are exporting a file 8 - Configure your LDOM guest following the LDOM administration guide.
9 - Netboot and install Linux in the guest.
10 - Before rebooting into the installation, halt the guest and unbind it. The unbind is required to unlock the exported device for read.
11 - Take a copy of the partition table that has been created by Linux.
Example: dd if=/path/to/device of=/.ldom.linux/$guestname-$(basename /path/to/device).backup
Following the above example: dd if=/dev/rdsk/c1t1d0s2 of=/.ldom.linux/foo1-c1t1d0s2.backup count=1 bs=512
or
dd if=/path/to/file of=/.ldom.linux/foo1-file.backup count=1 bs=512
12 - Edit the configuration file in /etc that you installed at step 3.
Each line contains:
guest_name disk1 disk2 ...
NOTE: you do not need to add entries for Solaris guests. Only Linux guests are required! Empty lines are skipped, no comments are allowed in the file.
Follow steps 13 to 19 if you are exporting a disk 13 - Restore Solaris partition table:
Example: dd conv=notrunc if=/.ldom.linux/$guest-$(basename /path/to/device).format of=/path/to/device bs=512 count=1
14 - Bind the guest
15 - Unbind the guest again. HINT: this is the most important step here because LDOM disk server now has validated the partition table and it will not check it again.
16 - Restore Linux partition table:
Example: dd conv=notrunc if=/.ldom.linux/$guest-$(basename /path/to/device).backup of=/path/to/device bs=512 count=1
17 - Bind the guest again.
18 - Start the guest.
19 - Enjoy.
Follow steps 20 to 23 if you are using a file 20 - Bind the guest
21 - Restore Linux partition table:
Example: dd conv=notrunc if=/.ldom.linux/$guest-$(basename /path/to/file).backup of=/path/to/file bs=512 count=1
22 - Start the guest.
23 - Enjoy.
This said, the script will allow you to reboot the controller without having to worry about Linux guests and it should ensure that LDOM will always start properly. LDOM stores the status of a guests in the Machine Description and we will use this status to restore the partition tables too. If the guest is in either bind or active status, we will take proper actions to stop/unbind/restore/bind/start. If the guest is in unbind state, we will do nothing because we do not know why is in that state. The script also is not error prone. Failing to stop or unbind the domain, to perform restore partition table operation, might cause LDOM to abort and can require manual fixing but the above information should be enough for any experienced user to recover. Have fun

17 July 2007

Matthew Palmer: The Trials of Browsers

Since I'm a member of the set of people "somebody better suited to explaining it" mentioned in Silvia Pfeiffer's "A Long Story of Logins", I suppose it falls to me to explain what the hell we actually did to make the thing work. A quick recap, for anyone who hasn't read Silvia's article: the Vquence website is a very AJAXy, Web 2.0 style video socialisation site. Since we want the site to be available to all as much as possible, we don't lock any part of the site away behind a "login required" page unless we absolutely have to. An unpleasant side effect of this is that, when it does does come time for you to login, pushing you off to a separate "please login" page is going to be, well, a pest. There's all sorts of state about what the user is up to (it's a web-based video mashup creator, after all) that we need to submit -- but only after the user has logged in. Enter the realm of the Web 2.0 Login. Instead of sending the user to a login page, getting them to login, and then redirecting them back to where they want to go, you just have some sort of popup form that, when submitted to, does a background request and logs the user in behind the scenes. This isn't a new concept by any means -- for example, try voting up or down a link on reddit when you're not logged in. However, most sites that implement this sort of thing (based on our brief informal survey around the office) don't protect the password information in any way -- it goes over the wire as a plaintext HTTP request. We don't like this, so we wanted to send the login form over HTTPS. And there began our problems. The trivial implementation of a Web 2.0 Login page is as you'd expect -- the AJAX stalwart, XmlHttpRequest, POSTing to a handler somewhere that validates the login, sets a session, and sends back some javascript to change the page layout a bit to show you're logged in. We had this working nicely, until we started testing on our staging environment, where HTTPS comes into play... You see, XmlHttpRequest doesn't like making a request to a domain other than the one that the main page comes from -- for security. It also thinks that the same hostname via a different protocol counts as a different domain, for the purposes of this security check. (I can see the theoretical reasoning for it, but it is a bit weird for HTTP to HTTPS transitions) So if you want to POST to https://www.vquence.com/something using XmlHttpRequest, you need to be doing it from a page which is retrieved from somewhere in https://www.vquence.com/ as well. We considered just turning on HTTPS across the board, but that would have all sorts of unpleasant ramifications, so we decided against that. But the problem remained, how do we get the user's credentials to our server in a secure manner with a minimum of unpleasantness? We came up with all sorts of options. This is where the infamy of "bug #93" started -- it's a monster of a bug log. Lots of it is me going "we could do this, but it'd suck, or we could do this, but it would suck". Finally, Silvia (I think) suggested we look at using an iframe. "Eeew" was the general response, but it might at least give us a way to set a part of the page to HTTPS, which we could use as a launching point for our XmlHttpRequest call. While researching how to make this all work (I had, to my great pride, never inflicted an iframe on the world before), I managed to come across a few articles on how you can use a hidden iframe to handle requests instead of using XmlHttpRequest. Basically, the trick is that you do a regular POST on your main page, but set the target of the form to a hidden iframe in your page. A general discussion of the principle is available At the Apple Developer site, and there's also a Rails plugin that implements the concept rather nicely. When I found these, I briefly danced for joy. My quest was over. POST to the iframe, the iframe get some JS back from the server, JS mangles the main page's layout to show the user is now logged in... aaaand profit. Like fun it was. My implementation was short and sweet -- a true work of genius (if I do say so myself). Everything worked like a charm. Until we got it back into staging. Then cross-domain security bit me again. Naturally, if you don't want XmlHttpRequest calls to perform cross-domain, you probably don't want iframes for different sites to be manipulating each other's content either. That's exactly what happened -- the iframe can't manipulate the main page's layout as we need it to. Why I didn't think of this before I charged mightily ahead I don't know. My only defence is that I've never claimed to be a web development guru. Sneaky tricks will defeat any security measure, though. For instance, we control both sites (http and https), so we can send the user back to the unsecured site after login and then that location (still in the hidden iframe) can manipulate the main page layout. Redirect to the rescue! That worked really well in Firefox, even with HTTPS in staging. We were saved. At least until IE came along. Bloody Microsoft, ruining all my cunning plans. IE complains when a HTTPS site redirects (using the Location HTTP header) to a HTTP site. You apparently can't turn this off. I was shocked that Microsoft cared so much about their user's data security -- it's certainly not the first thing I think of when I think "Microsoft". My faith in the world was restored when Silvia pointed me to a workaround for the problem -- if you send the redirect in a <meta http-equiv="refresh"> tag instead of a Location header, IE is more than happy to let it pass. Because obvously a chunk of HTML that says "HTTP equivalence" is completely different to a HTTP header. Sheesh. To snatch defeat from the jaws of victory, after all this brilliant hacking, I managed to make a newbie error with the contents of the http-equiv by setting the content to "0;http://www.vquence.com/blah" instead of "0;url=http://www.vquence.com/blah" (note the url=, because I sure didn't). IE complained by looping infinitely on the source URL, while Firefox just made sense of it and did the right thing. I managed to waste most of an hour of three of us over that little stuff-up. Go me! (Sorry about that guys, by the way) Feeling very chuffed, I went home for the evening. The next morning saw a browser compatibility defect in my queue -- Safari didn't like the new setup. Instead of doing the right thing, clicking on the login button would create a new tab instead of logging in. The fix? Changing the style of the hidden iframe from "display: none" to "width: 1px; height: 1px; border: 0" did the trick. Fark. So much for "hidden" iframes. What's particularly funny is that Apple's own tutorial on hidden iframes says to avoid "display: none" because NS6 doesn't like it, but doesn't mention it's own glaring deficiency in this area. So, what are the lessons learned?

Marc 'Zugschlus' Haber: From the personal Inbox of the exim4 maintainer

The DDs reading this might know the situation: You are subscribed to a gazillion of mailing lists, and spend quite some time answering questions of people using your packages. That’s fine, service to your users. Occasionally, users take great pains in finding out a personal mail address (for example, by googling, and finding the webmasteridiot mail address on my personal web page) to ask their question in private e-mail. This prevents the answers from showing up in mail archives and deprives the public of a possibility to find a solution to this question themselves in the future. Today, I had this happen to me twice. In the morning, I received a question about exim4 in private e-mail. In German. I investigated the user’s problem (which looked interesting enough), found two minor issues in the Debian exim4 packages and upstream exim behaving not as expected. I fixed the two issues in the package and reported the unexpected behavior of exim to the upstream mailing list. I then reported this - in German - back to the person who asked the question. Time from incoming message to outgoing report was like 80 minutes, most of which I spent with this user’s issue. On my personal time. A short while afterwards, I found out that this person has waited a mere eleven minutes before asking the same question on pkg-exim4-users, this time in English. Had I known before, I would have drafted my answer in English as well, to have the answer publicly visible. By doing this double post, the user has deprived the public of my answer. I’m probably going to stop answering questions asked in private mail. I might draft a boilerplate answer along the lines of “I don’t answer questions asked in private mail unless I am being paid to do so. You might be better off by asking one of the numerous mailing lists that are available on the Internet. I read a lot of them, might see your question, and you might get an answer from me. But otoh, there are many more knowledgeable people on these lists, and you’d want them to see your question as well.” Or I just might hit Delete on questions in my private inbox. The “Delete” approach has become more attractive this afternoon after I pulled the following (anonymized) from my personal Inbox:
Great job breaking exim4 on my system with no easy way to fix it.
$ /etc/init.d/exim4 start
Starting MTA:DEBCONFsomethingDEBCONF found in exim configuration. This
is most probably
caused by you upgrading to exim4 4.67-3 or later without accepting the
suggested conffile changes. Please read
/usr/share/doc/exim4-config/NEWS.Debian.gz for 4.67-2 and 4.67-4
2007-07-17 11:31:57 Exim configuration error in line 25 of
/var/lib/exim4/config.autogenerated.tmp:
  malformed macro definition
Invalid new configfile /var/lib/exim4/config.autogenerated.tmp, not
installing
/var/lib/exim4/config.autogenerated.tmp to
/var/lib/exim4/config.autogenerated
I’ve been running exim4 v4.63 for awhile now and only have you bugger
exim4-config it doesn’t work.
I mean, this user has Sorry, but: How stupid can people be? I think I’m going to stop being responsive to users. They’re not worth the blood pressure.

Russell Coker: Testing STONITH

One problem that I have had in configuring Heartbeat clusters is in performing a STONITH that originates outside the Heartbeat system. STONITH was designed for the Heartbeat system to know when a node is not operating correctly (this can either be determined by the node itself or by other nodes in the network) and then force a hardware reset so that the non-functional node will not interfere with another node that is designated to take over the service. However sometimes code that is called by Heartbeat will have more information about the state of the system than Heartbeat can access. For example if I have a service that accesses a filesystem on an external RAID then it’s common for the RAID to track who is accessing it. In some situations the RAID hardware has the ability to “fence” the access (so that when machine B mounts the filesystem machine A can no longer access it). In other situations the RAID may only be capable of informing the system that another machine is registered as the owner of the device. To solve this problem a machine that is to mount such a device must either prohibit the previous owner from accessing the device (which may be impossible or unreasonably difficult) or reset the previous owner. Until recently I had been doing this by writing some code to extract the STONITH configuration from the CIB and call the stonith utility. The problem with this is that there is no requirement that every node be capable of performing a STONITH on every other node, and that even if every node is are designed to be capable of rebooting every other node a partial failure condition may restrict the set of nodes that are capable of performing a STONITH on the target. Currently the recommended way of doing this is via the test program. Below is an example of the command used to reset the node node-1 with a timeout of 20000ms and the result of it being successfully completed. I have suggested that the Heartbeat developers make an official interface for doing this (rather than a test of the API) and I believe that this is being considered. In the mean time the following is the only way of doing it: # /usr/lib/heartbeat/stonithdtest/apitest 1 node-1 20000 0
optype=1, node_name=node-1, result=0, node_list=node-0

16 July 2007

Russell Coker: Documentaries about Gifted Children

On several occasions I have watched part of a TV documentary on gifted children, but I have never been able to watch one completely because every one that I have seen has been offensively wrong. One thing that they always seem to do is say that gifted children have special needs and often claim that they have problems socialising. This sounds quite reasonable, but if that’s the case then why would you make such children perform tricks on TV? Putting the children on TV is a very poor example of journalism and often of parenting - the parents’ desire to boast about their children’s performance (and implicitly their own parenting skills) is apparently more important than protecting the children. I think that the only situation in which gifted children should have their talents demonstrated on TV is if their skill is related to the performing arts. Not that TV coverage is necessarily good for children who have abilities related to performing (a casual scan of the news regarding adults in Hollywood shows the problems that people have dealing with fame), but it’s something that they will be driven to anyway. Also if they are going to demonstrate the intelligence of a child on TV then they really should make sure that they don’t demonstrate a lack of intelligence (on the part of the child as well as the producers). Asking a child to provide a definition of a word is often used as an example of intelligence (when it’s really an example of vocabulary). In one documentary a child defined a philanthropist as “someone who has a lot of money” (according to dict on my system it is “Love to mankind; benevolence toward the whole human family; universal good will; desire and readiness to do good to all men“). In another a child defined a genius as “someone who knows a lot“, while the definition of genius is not clear and there is some disagreement about what it is, most people agree that it’s about ability not knowledge. Being able to recognise when you don’t know something and admit would surely be correlated with intelligence… Telling a child that they are a genius or telling them their IQ seems like a bad idea at the best of times, and doing so in front of a documentary camera crew isn’t the best of times. Children are able to determine how their skills compare to others, there are more than enough attitude problems in schools related to skill comparisons without encouragement from adults. When I was in high school a friend who studied a martial art refused to tell me which belt he wore - he had been taught that such things shouldn’t be discussed outside the dojo to avoid creating a hierarchy based on belt rating in the community. I think that the same thing could be applied to IQ ratings. The term genius is grossly overused generally, I believe that showing greater than average ability in one area is not enough to qualify. I think that the minimum criteria would be to produce dramatic new developments/inventions in one area of research (EG Albert Einstein or Stephen Hawking) or to advance multiple fields of science or art (EG Leonardo da Vinci) would be the minimum criteria. I had people call me a genius when I was young because I won prizes in maths competitions, was good at programming computers, playing chess, etc. The main criteria for achieving such things is to avoid wasting time on sport - in a school run by idiots with a focus on sport this was more a tribute to being stubborn than being smart! The TV documentaries mention many things that gifted children supposedly require, but strangely I don’t recall any of them mentioning the need to meet adults who are significantly smarter than average! You might think that this is the most obvious thing. The Big Brother and Big Sister programs of mentoring children who are at risk of crime and drugs apparently provide significant benefits, something similar for gifted children might do some good, probably more based on meetings of a few intelligent adults with a small group of intelligent children instead of the BB-BS model of one-on-one meetings. Washingtonienne seems like a good example of the need for this. There are some useful print articles however, this article in The Age makes some good points, it’s particularly interesting to note that some schools lie about special programs to support gifted children to attract students (someone should name the schools that do this). But then you get awful ones like this, naming the child in such a situation is irresponsible journalism. The girl in question may be forced to change her name to escape google when she’s older. It’s strange that it’s illegal to name a child who is involved in a court case but it’s not illegal to name them in such an article (let’s hope that the journalist used pseudonyms). Also you might expect an organisation such as Mensa to have someone smart enough to realise that bringing such attention on a 2yo is not in the child’s best interests. Does anyone know of a good TV documentary about gifted children? I guess that it would be extremely difficult to make one without showing the children on TV so that would restrict the film maker to children who’s abilities are related to the performing arts.

15 July 2007

Stefano Zacchiroli: python debfile

DebFile: Python Library for Inspecting .deb A couple of days ago Mike Hommey blogged about "coincidences" in free software development: more than one developers are likely to have the same crazy ideas. It happened to me and Filippo last Friday. While writing the script for maintaining the OCaml status page I faced, unsuccessfully, the need of a library for extracting meta-information and actual content out of .debs of incoming. While I was getting sad for the lack of such a (Python) library Filippo contacted me on Jabber stating something like you know what? ... a cool addition for python-debian would be a library for querying .deb files ... A few days later we have something working and fitting our (disjoint) needs (it's a branch of the python-debian package which includes two brand new modules: arfile.py and debfile.py):
$ bzr branch http://people.debian.org/~zack/bzr/pkg-python-debian/

Here are some examples to wet your appetite: It is already a proposed addition to python-debian, feel free to comment about the API ... before it's too late :)

13 July 2007

Michael Prokop: Find out what users request

One of the coolest features of grml IMO is grml-tips. grml-tips is a collection of tips you can request via a keyword. For example if you want to know how to use grml-x just run ‘grml-tips grml-x’:
mika@meilenschwein ~ % grml-tips grml-x
Start X window system (XFree86 / Xorg / X.org):
% grml-x $WINDOWMANAGER
Usage examples:
% grml-x fluxbox
% grml-x -mode '1024x768' wmii
% grml-x -nosync wm-ng
--
mika@meilenschwein ~ %
So as soon as you invoke ‘grml-x $keyword’ and if the grml-tips collection includes something with $keyword inside you’ll get it. I use that on daily base because all the hard-to-remember command sequences or dangerous commands like mdadm, sfdisk,… are easy to use this way. :-) But how about keywords not being availble in grml-tips collection yet? What about keywords other users request but don’t get a match? Thanks to formorer we have a submit feature within grml-tips since version 0.4.1:
% grml-tips nfs
Sorry, could not find a tip for 'nfs'. :-( 
Do you want to submit the keyword 'nfs' to grml's keyword database?
The grml team will write tips for the most requested and useful keywords.
To use and contribute to this feature you'll need a working networking connection.
No personal data will be transmitted to the database.
Send "nfs" to grml's keyword database? [y N]
Keyword 'nfs' has been submitted to grml's keyword database.
Thanks.
%
Now being the maintainer of grml-tips I can check what users want to get through querying the sqlite database which stores the submitted keywords. Dear paranoid users: by default the keyword is not transfered to the database of course, you have to manually press ‘y’ to submit it. It’s not the default behaviour but an option. And there’s really no personal information stored inside the database, all I can query looks like the following:
$ID nfs grml 1.0-1 Release Codename Sonnenbrand [2007-07-13] 1184261151
Oh and thanks to Nico we also have our 24/7 support slave within the #grml channel:
19:37 <     mika > grml-tips: grml-x
19:37 < grml-tips> matches: http://www.grml.org/tips/index.html#23
And if our support guy does not work as expected:
19:38 <     mika > grml-tips: does-not-match
19:38 < grml-tips> Sorry, could not find a tip for 'does-not-match'.
If you want to submit a tip please mail it to tips (at) grml.org - thank you!
… and once a week I get a cronjob-triggered mail sending me all the keywords without a match. Cool, ey? ;-) All what’s missing now is a bot that writes all the missing tips on its own. 8-) Conclusion: make it easy for your users to submit information you might be interested in - to help your users.

Wouter Verhelst: Traffic Shaping

Erich Schubert blogs about traffic shaping, and shows how you can do some basic traffic shaping using a rather long example. I'm sure that example works, but mine is slightly less complex: tc qdisc add dev eth1 root tbf rate 212kbit burst 1540 latency 50ms This is actually a literal example in the Linux Advanced Router and Traffic Control-HOWTO, and it works for me. Linux has a reasonably good default prioritizing and queueing already, and using the tbf filter doesn't replace them, whereas the detailed sfq example that Erich gave does. I'm doing this on my router (on the outgoing interface), not my laptop, which is also a rather important bit. Of course, it working for me doesn't necessarily mean that it'll work for you too. But you might just want to try it...

Ross Burton: Embedding Binary Blobs With GCC

For a long time I've wanted to know how to embed binary blobs into executables. This would be most useful for files like Glade and and UI Manager definitions, which are required for a given program to work at all but either cannot be embedded as a string literal (Glade) or can be but is annoying (UI Manager). I finally asked the Interweb, and Daniel Jacobowitz replied with some pointers. It turns out that doing this is remarkable simple. First, a caveat. This probably requires GNU ld, which may or may not be a deal breaker for many people. First, create a data file. Let's call it foo.txt, and put some text in it.
Hello, World!
Using ld this can be read in as a plain binary blob, and then written as a standard relocatable ELF object.
ld -r -b binary -o foo.o foo.txt
Now we have a standard ELF object with the data and some useful symbols defined. objdump will show you the contents.
$ objdump -x foo.o 
foo.o:     file format elf32-i386
Sections:
Idx Name          Size      VMA       LMA       File off  Algn
  0 .data         0000000d  00000000  00000000  00000034  2**0
                  CONTENTS, ALLOC, LOAD, DATA
SYMBOL TABLE:
00000000 l    d  .data  00000000 .data
0000000d g       .data  00000000 _binary_foo_txt_end
0000000d g       *ABS*  00000000 _binary_foo_txt_size
00000000 g       .data  00000000 _binary_foo_txt_start
Here we see 13 bytes of data, and a symbol which contains the address of the data. This is all we need to access it from a C program.
#include <stdio.h>
extern char _binary_foo_txt_start[];
int main (void)  
  puts (_binary_foo_txt_start);
  return 0;
 
Now if we compile this and link it against the generated object, we'll have a binary.
$ gcc -o test test.c foo.o
$ ./test
Hello, World!
Hooray! One small problem which alert people should have noticed: the string itself is in the .data section, which is read/write. For my use, I want it to be read-only data in the .rodata section so that it isn't copied for every instance of the application. As far as I know, this isn't possible with ld but objcopy will let us rename sections on the fly.
$ objcopy --rename-section .data=.rodata,alloc,load,readonly,data,contents foo.o foo.o
$ objdump  -h foo.o
...
  0 .rodata       0000000d  00000000  00000000  00000034  2**0
Excellent, problem solved. If you want to download this sample, I have a tarball. Many thanks to Daniel Jacobowitz for pointing out how to achieve this. Update: note that any data embedded in the binary like this won't be terminated with a NULL. This is obvious in hindsight, but due to luck my example still worked. There might be a way of asking objcopy to append a 0 to the end of the data, but if not always remember to use the start and end pointers or size instead of just the start, or append a NULL yourself before converting to an ELF. NP: (), Sigur Rós

12 July 2007

Christian Perrier: Lazyweb mutt

Dear Lazyweb, Can you explain me why, since the last update of mutt in unstable, I no longer have the display of percentages of the mailbox's number of message being read while Mutt is reading a mailbox when I open it? This is kinda annoying on large mailboxes as mutt now just display the number of examined message headers while one has no way to know how long it will need to read the entirre mailbox. That's probably a weird new setting, but which one? I don't feel like reporting a bug about this but this is a minor annoying regression, indeed..:-)

Erich Schubert: Shaping your traffic for improved performance

Sometimes, putting up an arbitrary limit for your network connection can actually improve the performance.A typical setup will look like this: your computer is connected via 54-100 mBit to a router, which has a DSL line of let's say 2 mBit downstream and 192 kBit upstream (and then tons of other links).When you overload one of these connections, performance can go down a lot.The simples case how this can happen is when you upload data. 192 kBit are the weakest link here, and rather easy to fill up.What happens then is that your computer will start sending out data at his full link speed, until the buffers at the router are full and it starts dropping packets. TCP/IP will then adopt and - lacking 'ACK' packets - slow down sending data. It will likely still keep the send-buffer of the router full.Now if the router isn't very smart, it just keeps a first-in-first-out buffer. As long as you have just this single upload, you don't need to care. But if you have other connections, like a download, you do care.For your download to run at full speed, the sender needs to get those ACK packets from you. They're basically a "got your data, send more" kind of message. Really tiny packets, but essential for signaling.A simple trick you can use now is to limit your bandwith at the sending computer already. Since you're going to limit it just to the next link anyway it doesn't mean it will actually be getting much slower. But your network stack is smarter than the router's (probably). It can send out these ACK packets at a higher priority, so you end up with a better download-and-upload performance.If you know your network connection speed, just use it. Otherwise you can still measure it or guess it. I'm always going for 95% of the speed I see. You might want to limit it a bit more if you are sharing the connection with others.Now enough theory, here's what I use:
IFACE=wifi0
# setup ingress filtering
/sbin/tc qdisc add dev $IFACE handle ffff: ingress
# local network is unrestricted
/sbin/tc filter add dev $IFACE parent ffff: protocol ip \
  prio 10 u32 match ip src 192.168.2.0/24 \
  police conform-exceed ok/ok flowid :1
# incoming internet traffic is limited.
/sbin/tc filter add dev $IFACE parent ffff: protocol ip \
  prio 50 u32 match ip src 0.0.0.0/0 police \
  rate 1500kbit  burst 10k drop flowid :2
# outgoing traffic
/sbin/tc qdisc add dev $IFACE root handle 1: htb default 30
# local network is unrestricted
/sbin/tc class add dev $IFACE parent 1: classid 1:10 htb \
  rate 54mbit burst 15k
/sbin/tc qdisc add dev $IFACE parent 1:10 handle 10: sfq perturb 10
/sbin/tc filter add dev $IFACE parent 1:0 protocol ip \
  prio 10 u32 match ip dst 192.168.2.0/24 flowid 1:10
# internet traffic is limited.
/sbin/tc class add dev $IFACE parent 1: classid 1:20 htb \
  rate 160kbit ceil 196kbit burst 15k
/sbin/tc qdisc add dev $IFACE parent 1:20 handle 20: sfq perturb 10
/sbin/tc filter add dev $IFACE parent 1:0 protocol ip \
  prio 50 u32 match ip src 0.0.0.0/0 flowid 1:20
Long example, I know. The main reason is that I'm not limiting traffic to local computers. Furthermore, I'm also limiting my incoming traffic to use up at most 1.5 mbit of the 2 mbit connection, so my housemates will be able to use the internet as well (Debian mirrors send data fast - fast enough to kick them out of any online game they might be playing...).There is one important detail hidden in there: sfq. This stands for "stochastic fair queueing". It tells the network stack to basically send "one packet for each connection in turn". This way, the small packets needed for download or interactive SSH sessions will get out quickly even when doing a larger upload.It works great for me - I'm having a good download rate, using the 160 kBit upload limit completely (I don't want to completely fill up the 192k upload link either) and I'm actually writing this blog entry remotely via SSH. The lag when writing is okay, probably around 300ms. Without shaping, this wouldn't work this well.(And no: just using 'sfq' for your outgoing traffic is not enough, since the key scheduling is happening at the weakest link. So you have to limit your traffic shaping setup to be just below the actual weakest link.)

11 July 2007

Stefano Zacchiroli: uploaders vs maintainers

More Uploaders, Less Maintainers: Infrastructure Issues When yet another discussion about "is collaborative maintenance good or not?" starts on debian- lists, the average lurker can easily identify two arguments / religions:
  1. it is good because duties are shared among co-maintainers
  2. it is bad because no one feels responsible for the co-maintained packages
Of the two religions, I stand for the first one. More generally I share the long term aim of decreasing as much as possible the "ownership" of Debian packages; collaborative maintenance is just a tool to achieve the goal. In the meantime, I believe the second religion is supported also by technical deficiencies in some pieces of our infrastructure. In particular some of our developer tools artificially enforce a distinction among Maintainers and Uploaders where it wouldn't be necessarily needed. In the last few days I thought about some of them: Think about such deficiencies, I'm pretty sure we can spot and fix easily a lot of them in our tools. Update: Lucas: I agree that the two arguments do not necessarily contradict, that's why I (initially) called them arguments. Then I moved to "religions", it was an oversimplification: there are of course pros and cons of collab maint. My main argument holds nevertheless: several perceived cons of collab maint are due to artificial distinctions maintainer/uploader in our tools, thanks for providing yet another example ... I want more! :)

Russell Coker: Questions During Lectures

An issue that causes some discussion and debate is the number and type of questions that may be asked during a lecture. In a previous post giving advice for speakers I suggested that questions can be used as a mechanism for getting a talk back on track if a nervous speaker starts presenting the material too quickly (a common mistake). This mechanism can be used by the speaker if they realise that things aren’t going to plan or by audience members who are experienced speakers and who recognise a problem. Due to this a blanket ban on questions during a talk will only work with experienced speakers who have planned their talk well. There are different styles of presentation favoured by different speakers. Some are determined by the nature of the topic (an example that I have seen cited is topics that are very contentious which would lead to a debate if questions were permitted during the talk), but for computer science I think that questions during the talk can always work well. To a certain extent the fact that code either works or doesn’t limits the scope for debate. Probably the major factor that determines the utility of questions is the size of the audience. If you have an audience of less than 50 people then a conversational approach can work, if you have less than 200 people then a reasonable number of questions can be accepted. But as the audience size increases above 300 the utility of questions approaches zero. If the majority of people who might want to ask questions are unable to do so due to lack of time then the value of allowing any questions diminishes. For the largest audiences there probably isn’t any point in having question time. Another major factor determining which style works best is where the speaker has had experience speaking. Most of my speaking experience is with less formal meetings (such as local LUGs) and in countries with an informal attitude towards such things (Australia, the US, and The Netherlands). A speaker who has primarily spoken for universities such as Cambridge or Oxford (which seem to have a very formal style and questions strictly reserved for the end) or who has come from a country such as Japan(*) where it’s reported that the audience are obliged to show respect for the speaker by being quiet will probably expect questions only at the end and may flatly reject questions during the talk. A speaker who has a background speaking for less formal audiences will expect a certain number of questions during the course of the talk and may plan the timing of their talk with this in mind. When I plan a talk for a one hour slot I plan at most 30 minutes of scheduled talking (IE covering my notes) expecting that there will be 15 minutes of questions along the way and another 15 minutes of questions at the end. Often with such plans my talks run over-time. Of course this means that people who have mostly had experience speaking to smaller and less formal audiences will find it exceedingly difficult to give talks to larger audiences. This may be an incentive for having more formal arrangements for LUG talks to increase the skills of speakers. The next issue is what level of contentious questions is acceptable. I believe that if you have a disagreement with points that the speaker is making (and have some experience in the field in question) and the audience is not particularly large then one hostile question is acceptable - as a speaker it’s reasonable to refuse to take any further questions from an audience member who has asked one hostile question. Another category of question is the challenging question (not to be confused with a hostile question), for example describing in one sentence what your business requirements are and asking how the topic being discussed will apply to that business requirement. One of the most useful questions I have been asked during a SE Linux talk was concerning the issue of backups of file security contexts, it was presented in a challenging way and the answer that I gave was not nearly as good as I could give now (the code base has improved over the last few years in this regard) - but I think that everyone learned something so that validated the question. In smaller groups there may be some heckling when the speaker is a well known member of the group, I don’t think that this is a problem either as long as it only consumes a tiny fraction of the time (maybe 20-30 seconds at the start). For larger audiences or for speakers who don’t know the audience well heckling is generally a bad idea. (*) When speaking in Japan I had a lot of audience interaction. I’m not sure if this is an indication of the Japanese culture changing in this regard, the fact that translation problems forced some interaction, or the audience was showing respect for the Australian culture by asking questions.

MJ Ray: North Somerset Rubbish Targets

One of my examples to explain "the tragedy of targets" used to be councils meeting recycling collection targets by collecting the recyclable waste, then putting it in landfill because that was cheaper than actually recycling it. That's bad because it makes a lot of extra work for everyone, but only meets the collection target in fact, not in spirit. I stopped using that example because I thought it was too incredible and I didn't have a good reference to support it. So imagine my surprise to see a report in the Weston Mercury saying:
"If Veolia does not have enough manpower it will send out a green waste lorry, or a domestic refuse lorry which usually picks up black bag rubbish, to pick up both at the same time. [...] The staff will just take it for granted that green waste is in the green lorries and domestic waste is in the blue lorries."
This is bad on many levels, even worse than my old example where recycling-ready waste was knowingly tipped into landfill. There's green waste contamination and false collection statistics, for starters. The article continues to report that Cllr Carl Francis-Pester, North Somerset Council's executive member for environment, said:
"We are carrying out an immediate investigation into these allegations which we take very seriously indeed. This investigation will include urgent discussions with the contractor to ensure they are fulfilling the contract we have with them in the correct manner."
There doesn't seem to be any information about this investigation on North Somerset Council's web site. And if Veolia have spotted a loop-hole in the contract, then what? Even if they haven't, the council has to watch solely-profit-led contractors like hawks because if you give them a chance to cut costs (and therefore increase profits), they will try it. That is the hidden cost of outsourcing, not covered by the Best Value Indicators and other privatised council jargon. Incredibly, the councillor continues with:
"Thanks to the residents of North Somerset we have increased our recycling rate from 20.5 per cent to 30.8 per cent in the last 12 months."
If the allegations are true, then no, we haven't, because the statistics would be as full of toxic rubbish as the green waste lorries. Have private rubbish collectors been doing this elsewhere?

9 July 2007

Russell Coker: Why Can t Bush be as Honourable as Nixon?

The Libby commutation has demonstrated the contempt that the Bush regime has for the US political process, the opinions of the voters and party members, and the reputation of the US. The Talking Points Memo has an insightful analysis. They suggest that an argument could be made for pardoning Libby by claiming that the case is politically based and bogus, such an argument would be difficult to sustain - particularly given the fact that the prosecutor and the judge were Republican appointees. But a commutation is admitting that the case was valid but that Bush wanted to protect a friend - or someone who might inform on him. In either case it’s a very bad thing. As an example here’s a YouTube video of Keith Olbermann making the case for George Bush to act as decently and honourably as Richard Nixon and resign from office. Paul blogs about why embedding media players is bad for the case of web radio. One thing I notice is that there is no convenient or easy way of extracting an embedded youtube video from a web page a quick test in my favourite browser (Konqueror) showed that the YouTube embedded section didn’t even appear. The Debian package youtube-dl allows you to download the Youtube content in a format that can be played with mplayer, but only if you see the link - which seems to not happen if the web site tries to embed it. Avoiding flash is good if you want security or if you want to use a platform other than i386 Linux or Windows and youtube-dl removes what I consider to be the only compelling reason for Flash.

8 July 2007

Matt Brown: POSIX/NFSv4 ACL Inheritance Problems

I (as root) have a directory hierarchy that I want a particular group to always have write access to. The files and folders inside the hierarchy are owned and manipulated by a wide variety of diffrent users. Essentially I want to delegate ‘root’ access for a portion of the filesystem to a particular group. My first attempt at implementing this was to use the standard POSIX ACLs that are available for almost every filesystem Linux supports. I recursively set an ACL on the top-level directory to give the group write access to all files and directories that currently exist and then I recursively set a default ACL to give the group write access on all the directories. This default ACL should be inherited by any new files that are created ensuring that the group keeps write access to everything. Problem solved? Unfortunately not. The intricacies of complying with POSIX means that ACLs are implemented as an ACL plus a mask. To gain access to a particular file or directory the user or group must match an appropriate ACL granting the access and the mask for that file or directory must also allow the requested permission to be granted. When you add an ACL to a file or directory, the ‘group’ bits of the standard Unix permissions magically switch from controlling group access to controlling the mask portion of the ACL, effectively providing an upper bound on the permissions that an ACL entry can grant. This prevents legacy POSIX applications that do not understand ACLs from unintentionally granting excessive permissions - arguably a good thing. Unfortunately this also makes it very hard to preserve the ACL granting write access to the ‘root’ group which I legitimately intended to have in place on this portion of the filesystem. Newly created files under the hierarchy generally inherit the ACL as intended, as most applications attempt to create files with as many permissions as possible, leaving it up to the umask to remove undesired permissions. However any file that is copied into the hierarchy without the ‘group’ write bit set, or any file that has the ‘group’ write bit removed via chmod will actually remove the write bit from the ACL mask invalidating the ACL and leaving me back at square one! After a bit of Googling I thought that NFSv4 ACLs might be the answer to this problem, as they are marketed as “very similar to Windows ACLs” and I’m sure that I vaugely recall Windows being able to properly inherit ACLs from parent directories. Unfortunately after downloading the NFSv4 ACL patches and trying all the various mount options I cannot find any combination that will offer the functionality I need. The implementation conforms to POSIX, so it still has a mask parameter and the same problems as the standard POSIX ACLs. The only benefit from using NFSv4 ACLs that I can see is that you have more permissions to grant. So once again, I’m back to square one. I’m hoping that there is some fundamental point that I’m missing as this seems like a very common use-case that I would have thought would be well supported. If a command-line example is clearer to you look at:
http://www.mattb.net.nz/blog/dump/acl-inheritance-problems.txt My current solution is to run a cronjob every X minutes to recursively ‘chmod -R g+w /dir’, however that’s far from optimal as it exposes all sorts of race conditions and just seems ugly! Any suggestions or solutions will be gratefully received.

Next.

Previous.