Search Results: "tar"

6 November 2008

Steve Kemp: It's in your nature to destroy yourselves.

Elections I've said this elsewhere, but it bears repeating:
Anybody who expects a nation to turnaround overnight, due to a changing government, hasn't watched/read enough documentaries.
Television
Who is going to make documentaries when David Attenborough dies?
ObFilm: Terminator 2

Mike Hommey: Emptying a deleted file

Yesterday, at work, we had the typical case where df would say there is (almost) no space left on some device, while du doesn’t see as much data present as you would expect from this situation. This happens when you delete a file that another process has opened (and, obviously, not yet closed). In typical UNIX filesystems, files are actually only entries in a directory, pointing (linking) to the real information about the content, the inode. The inode contains the information about how many such links exist on the filesystem, the link count. When you create a hard link (ln without -s), you create another file entry in some directory, linking to the same inode as the “original” file. You also increase the link count for the inode. Likewise, when removing a file, the entry in the directory is removed (though most of the time, really only skipped, but that’s another story), and the link count decreased. When the link count is zero, usually, the inode is marked as deleted. Except when the usage count is not zero. When a process opens a file, the kernel keeps a usage count for the corresponding inode in memory. When some process is reading from a file, it doesn’t really expect it to disappear suddenly. So, as long as the usage count is not null, even when the link count in the inode is zero, the content is kept on the disk and still takes space on the filesystem. On the other hand, since there is no entry left in any directory linking to the inode, the size for this content can’t be added to du’s total. Back to our problem, the origin was that someone had to free some space on a 1GB filesystem, and thought a good idea would be to delete that 860MB log file that nobody cares about. Except that it didn’t really remove it, but he didn’t really check. Later, the “filesystem full” problem came back at someone else, who came to ask me what files from a small list he could remove. But the files were pretty small, and that wouldn’t have freed enough space. That gave me the feeling that we probably were in this typical case I introduced this post with, which du -sk confirmed: 970MB used on the filesystem according to df, but only 110MB worth of data… Just in case you would need to find the pid of the process having the deleted file still opened, or even better, get access to the file itself, you can use the following command:
find -L /proc/*/fd -type f -links 0
(this works on Linux ; remove -L on recent Solaris ; on other OSes, you can find the pid with lsof) Each path this command returns can be opened and its content accessed with a program, such as cat. That will give access to the deleted content. I already adressed how to re-link such a file, which somehow works under Linux, but in my case, all that mattered was to really remove the file, this time. But we didn’t know if it was safe to stop the process still holding the file, nor how to properly restart it. We were left without a possible resolution, but still needed to come up with something before the filesystem gets really full while waiting to be able to deal with the root of the problem. The first crazy idea I had was to attach a debugger to the process, and use it to close the file descriptor and open a new file instead (I think you can find some examples with google). But there was no debugger installed. So, I had this other crazy idea: would truncate() work on these /proc/$pid/fd files? You know what? It does work. So I bought us some time by running:
perl -e 'truncate("/proc/$pid/fd/$fd", 0);'
(somehow, there is no standard executable to do a truncate(), so I always resort to perl) Afterwards, I also verified the same works under Linux (where you wouldn’t really know what it’d do with these files that are symbolic links to somewhere that doesn’t exist). The even simpler following command works, too.
> /proc/$pid/fd/$fd
It doesn’t truncate() but open() with O_WRONLY O_CREAT O_TRUNC, and close() right after (to simplify), which has the same effect. Good to know, isn’t it?

Martin-Éric Racine: Available for a new job

A couple of years ago, I started what has probably been the best job I've held so far, as a Business Development Manager for Artec Group where I focused on marketing and selling the ThinCan LTSP client. During that time, I used a combination of Guerrilla Marketing techniques to spread the word about this cool Estonian embedded platform in its LTSP client configuration and defined an extremely tight market-oriented focus that saw this relatively obscure electronic design start-up rise to the forefront of the LTSP hardware market and also brought the company commendable notoriety in the Coreboot community, thanks to its Programmable LPC Dongle, a low-cost ROM emulator that can be used to bootstrap various embedded hardware and test firmware images. Looking back at close to 3 years with Artec (if we include the stint I did at Linutop), I most fondly remember how I managed to bring the Estonian and Turkish nations closer together in collaborating on Innovation. On my first visit to Turkey, I had noticed that very little is know about Estonia in Turkey, even though both countries have a collaboration that dates back to the 1920's, so my tactic was to introduce Estonia's achievements as a whole, to establish confidence in Artec's offering. It worked: Among other things, my idea to organize an Estonian ICT Conference during the visit of Estonian PM Andrus Ansip in Turkey created a lot of extremely positive buzz that resulted in Estonia being mentioned as a trendy country in a prime time Turkish TV series. During this time, I've also made a lot of contacts among the Estonian and Turkish business communities and, more surprisingly, among the Estonian diplomacy who, much to my amazement, welcomed my ideas for better promoting Estonian interests abroad with great enthusiasm. Come December 2008, my time at Artec will become a page in my employment history. While the Thincan remains one of Artec's key products, current marketing and sales resources are focused on new markets and new technologies that completely fall outside my core expertise, which is why the company decided to let me go. I thus welcome offers from interested parties in Coreboot or LTSP markets. Other refreshing offers from outside the box - such as in business diplomacy - are equally welcome, as I've been pondering a career change for a long time.

Eddy Petrișor: Coordinating localization via git

Note: this is quite long and might not be interesting for people not involved with /in either of: debian i18n, debian l10n, git, shell scripting.


For some time now I have been the de-facto coordinator of the Romanian localization team. During this time I was faced multiple times with problems related to motivation of the team members, setting goals for a release, coordination of the changes inside the team, integration of new translators in the team, loosing team members, my lack of time on some occasions and other problems.

I thought a lot about how to improve on those points and I was never satisfied with the answers I got. What I think was the worst problem in the team was the inability to set clear goals, especially during lenny's development cycle.

During sarge's and etch's development cycles we were in an infancy state and setting as the goal to have a fully translated installation process was really enough to keep people motivated. For sarge we missed by a small margin, but the translations were of poor quality, while for etch we worked more on improving the translations.


But during lenny it was bad, really bad. My free time had been shrinking for a while, starting with etch's release, and we were unable to set clear goal, since for etch we managed to have the installation process fully translated and a few other translations. There was no way for us to reach 100% translations during lenny's development cycle, so setting that as a goal was really unrealistic. Percents by themselves don't mean anything for people, and as long as there's no substance to those numbers, there's no motivation to reach for one arbitrary percent.

I tried to set as a goal translating the packages installed by default in a new installation, but that hit the eternal question How do we know easily which are those packages?. This remained sometimes unanswered or got an unsatisfying answer. Also, there was a goal to have correct diacritics for Romanian in lenny, to have aspell-ro that uses the correct diacritics.

I even got to a point that I, myself, lost my motivation and set myself a personal goal of overrunning the level the translations that the language just above Romanian had in the po-debconf l10n statistics (ranking between languages). This was nice way for me to keep myself motivated, but I had my reserves in making this motivation public out of fear of being misinterpreted, because, by a strange coincidence, that next language was Hungarian, and in Romania's history there was some friction between Hungaria and Romania, while there are still some tensions with the Hungarian minority in Romania, in areas where they represent the local majority.

I managed to reach my personal goal, but this wasn't addressing the big picture.



So, sometime around the start of this year I started thinking about ways to coordinate the Romanian localization team in order to have:
For short, a tool that would allow the team to work more efficiently while having the possibility to set clear goals in order to keep people motivated.



So after some pondering, I thought that creating a repository with the translations and the helper tools that would do the funky sync, checks, stats would be the best way to do that. So I started hacking on that somewhere around July-August and I published the result, but without much publicity, since it is still incomplete.

Some of the technical details are still in a haze, but I have a general idea and I got some basic functionality.


Today I decided that I should announce this semi-officially though my blog, maybe I get some input, ideas, or even contributions (I really should write a TODO).




I give you the Debian L10n Romanian coordination repository.

This is a git repository that has some tools to facilitate translation coordination and the translations that are current in the distro for the team.


Can be cloned with:

git clone git://git.debian.org/git/users/eddyp-guest/debian-ro-repo.git

or, if you're behind a restrictive firewall:

git clone http://git.debian.org/git/users/eddyp-guest/debian-ro-repo.git



Currently the work flow for updating a translation is as follows:
  1. source _bin/polibs (. _bin/polibs)
  2. cd foo
  3. po_refresh
  4. complete the translation
  5. po_rearrange "ro.po"
  6. git add "ro.po" && git commit -m "updated translation for foo"
  7. send the translation ("git format-patch origin" and send the patches by mail, or, alternatively, just "git push")


Features:
Planned features:

Problems:

I was hoping that the release notes for lenny would facilitate from this infrastructure, but unfortunately I was lately in a really inactive period wrt Debian.



Questions, suggestions, ideas are welcome.

Martin Michlmayr: European Commission publishes guidelines on the procurement of FOSS

The Open Source Repository and Observatory (OSOR), a new site sponsored by the European Commission to foster the exchange of FOSS related information and software among European public administrations, recently published guidelines on the procurement of open source software. Public administrations in Europe have to follow public tender procedures and the new guidelines give practical and legal advice on how open source software and related services can be incorporated into the procurement process. Rishab Ghosh, who presented the guidelines at the Open Source World Conference in Malaga, argued that the procurement guidelines were needed because of two reasons. First, they studied recent tenders and found that many explicitly mentioned proprietary applications. 16% of 3615 software tenders explicitly asked for products from top 10 software vendors, such as Microsoft, SAP and Oracle. This practice may be illegal because public tenders generally have to describe functional requirements in a general way instead of specifying specific products. Second, many public administrations don't have any experience with the procurement of FOSS. In fact, they often don't know whether or under which circumstances they are allowed to adopt and ask for FOSS solutions. The guidelines are specifically designed in order to clearly and simply explain how public administrations can acquire open source and they don't assume that a country has adopted a specific policy regarding open source. The guidelines include a long section about open standards, open source and how they relate. Both open standards and open source align very well with the needs of public administrations who have an "obligation to support interoperability, transparency and flexibility, as well as economical use of public funds". The guidelines argue that the exit cost, i.e. the cost incurred in moving to another IT system, is also an important consideration but one that is often neglected. The adoption of a proprietary solution without open standards may limit the future choice, thereby increasing the long-time costs and giving the proprietary vendor an unfair advantage in future tenders. The procurement guidelines describe two ways of acquiring FOSS: it is possible to go the usual route of publishing a tender for the supply of software (possibly with related services). However, in the case of FOSS, it is also possible to download the software directly from the Internet. This is possible because the software is not only free of charge but comes with no contractual obligations. If there were any obligations involved with the download (such as fees, the agreement to an EULA or the requirement to purchase services in the future), software download is not an allowed method. What I like about the guidelines is that they explicitly say that downloading software has to be part of the formal procurement process. You have to think about your requirements, look at various alternatives, and so on, and not just blindly download something from the Internet. When it comes to the procurement of FOSS, the guidelines don't suggest that tenders should explicitly ask for FOSS. Instead, they should describe the functional requirements of the software as well as certain properties. For example, a tender could specify that the public administration as well as third parties must have the right to study, distribute and modify the software. In a sense, the guidelines suggest that tenders should include the principles of the Free Software Definition along with justifications for these requirements. Personally, I think there is a great need for these procurement guidelines. There are many public administrations that don't know how to acquire FOSS and these guidelines offer clear advice. Furthermore, I find the guidelines very balanced. They don't recommend that you should always ask for FOSS but incorporate FOSS principles into tenders where it makes sense. It remains to be seen whether the procurement guidelines will have an impact on the FOSS adoption in Europe, but I surely hope so. (Originally published on FOSSBazaar)

Russell Coker: EC2 and IP Addresses

One of the exciting things about having a cloud computing service is how to talk to the rest of the world. It’s all very well to have a varying number of machines in various locations, but you need constant DNS names at least (and sometimes constant IP addresses) to do most useful things. I have previously described how to start an EC2 instance and login to it - which includes discovering it’s IP address [1]. It would not be difficult (in theory at least) to use nsupdate to change DNS records after an instance is started or terminated. One problem is that there is no way of knowing when an instance is undesirably terminated (IE killed by hardware failure) apart from polling ec2-describe-instances so it seems impossible to remove a DNS name before some other EC2 customer gets a dynamic IP address. So it seems that in most cases you will want a constant IP address (which Amazon calls an Elastic IP address) if you care about this possibility. For the case of an orderly shutdown you could have a script remove the DNS record, wait for the timeout period specified by the DNS server (so that all correctly operating DNS caches have purged the record) and then terminate the instance. One thing that interests me is the possibility of running front-end mail servers on EC2. Mail servers that receive mail from the net can take significant amounts of CPU time and RAM for spam and virus filters. Instead of having the expense of running enough MX servers to sustain the highest possible load even while one of the servers has experienced a hardware failure there is a possibility of running an extra EC2 instance at peak times with the possibility of running a large instance for a peak time when one of the dedicated servers has experienced a problem. The idea of having a mail server die and have someone else’s server take the IP address and receive the mail is too horrible to contemplate, so an Elastic IP address is required. It is quite OK to have a set of mail servers of which not all servers run all the time (this is why the MX record was introduced to the DNS) so having a server run periodically at periods of high load (one of the benefits of the EC2 service) will not require changes to the DNS. I think it’s reasonably important to minimise the amount of changes to the DNS due to the possibility of accidentally breaking it (which is a real catastrophe) and the possibility of servers caching DNS data for longer than they should. The alternative is to change the MX record to not point to the hostname of the server when the instance is terminated. I will be interested to read comments on this issue. The command ec2-allocate-address will allocate a public IP address for your use. Once the address is allocated it will cost $0.01 per hour whenever it is unused. There are also commands ec2-describe-addresses (to list all addresses allocated to you), ec2-release-address (to release an allocated address), ec2-associate-address to associate an IP address with a running instance, and ec2-disassociate-address to remove such an association. The command “ec2-associate-address -i INSTANCE ADDRESS” will associate an IP address with the specified instance (replace INSTANCE with the instance ID - a code starting with “i-” that is returned from ec2-describe-instances. The command “ec2-describe-instances grep ^INSTANCE cut -f2” will give you a list of all instance IDs in use - this is handy if your use of EC2 involves only one active instance at a time (all the EC2 API commands give output in tab-separated lists and can be easily manipulated with grep and cut). Associating an IP address with an instance is documented as taking several minutes, while Amazon provides no guarantees or precise figures as to how long the various operations take it seems that assigning an IP address is one of the slower operations. I expect that is due to the requirement for reconfiguring a firewall device (which services dozens or maybe hundreds of nodes) while creating or terminating an instance is an operation that is limited in scope to a single Xen host. One result that I didn’t expect was that associating an elastic address is that the original address that was assigned to the instance is removed. I had a ssh connection open to an instance when I assigned an elastic address and my connection was broken. It makes sense to remove addresses that aren’t needed (IPv4 addresses are a precious commodity) and further reading of the documentation revealed that this is the documented behavior. One thing I have not yet investigated is whether assigning an IP address from one instance to another is atomic. Taking a few minutes to assign an IP address is usually no big deal, but having an IP address be unusable for a few minutes while in the process of transitioning between servers would be quite inconvenient. It seems that a reasonably common desire would be to have a small instance running and to then transition the IP address to a large (or high-CPU) instance if the load gets high, having this happen without the users noticing would be a good thing.

Share This

5 November 2008

Neil Williams: Componentisation of Debian/Emdebian

OK, I know I've been very quiet on the blog for weeks now but I have my reasons and all will become clear, eventually. I'm principally waiting for the Lenny release before being able to start full discussions about certain ideas that have been consuming a lot of my time recently. I continue to do what I can to get Lenny out of the door because the wait is driving me nuts. (Fixed another RC bug yesterday, yippee!)

Anyway, something I can talk about is Emdebian Grip. Emdebian has always wanted to support "in-between" devices that don't need the full-on cross-build everything approach of the current Emdebian packages for ARM but don't quite have room for a full Debian install either. Until I had at least a basic package set available via the cross-building system, I didn't want to start working on an intermediate system, for reasons of simplicity mainly, but also because I knew that getting a cross-building package set would involve changes to the tools needed for the intermediary set.

Bah, I'm waffling again. For the background, see Emdebian, Squeeze, Grip and Crush.

The good news is that Emdebian Grip, once running, will take far, far less time (for everyone) than the full-on cross-building malarkey of Emdebian Crush. Grip can be 99.9% automated by breaking Debian into discrete package sets or 'components'.
Componentising Debian for Emdebian
Emdebian Grip is about a smaller Debian, nothing more. No changes in functionality, no changes in dependencies; only changes in final installation size and Packages.gz size. It means using the code from dpkg-cross to unpack packages, remove unwanted files and repack the .deb. Yes, yes, this can be done one package at a time at a trivial level. Emdebian Grip seeks to bring a standard method for this operation that will work for all packages on any architecture for any architecture from any suite, creating and maintaining repositories of "gripped" packages for use in devices like my Acer Aspire1 where you just don't need documentation, manpages and the rest. Squeeze is the obvious target release but any release supported by the current version of dpkg could be "gripped" too, like oldstable.

The componentisation comes in at the level of the Packages.gz file. If the device cannot realistically be expected to run CAD software or has no fancy graphics or other power-hungry hardware, why should packages that need such resources clutter up the apt-cache list? This data isn't just slowing down searching for packages you do want, it slows down every installation process (that bit about 'Reading database . . . . . '), it slows down the process of getting the updated cache data in the first place because there is more of it, it wastes space on your device in several different places when the whole point of Emdebian is that disc space is not cheap and it wastes your precious battery power if you need to do simple package management stuff away from the mains. What is needed is a mini-repository, a mini-mirror that just contains the parts of Debian that your machine needs - a "well-trimmed" mirror with sources and all dependencies satisfied, a mirror that can be automatically updated but which doesn't keep getting NEW packages that you won't use. New dependencies will be added, yes. New packages unrelated to what you currently use? Absolutely not. You could even keep a mini-mirror on your home network, updating and "gripping" while you sleep, because a mini-mirror should be a lot, lot smaller than a standard Debian mirror.

I now have a script that will do the work, what I've lacked is the idea on how to "feed" packages to the Gripper. :-)

So, I'm going to be exploring this method:
  1. An arbitrary package set - say the output of 'dpkg -l' from the target machine setup by an expert, or 'dpkg --get-selections' or just the packages from a normal debootstrap - the basic idea is to start with only the packages you absolutely must have for your target device.

  2. A very small wrapper for some debootstrap code that downloads those .debs from the Debian mirrors - debootstrap can cope with choosing the mirror, the architecture, the suite etc. (already written, landing in Emdebian SVN soon).

  3. Parse the list of packages (.debs) and use 'apt-cache showsrc' to convert into a list of source packages. (Script done).

  4. 'apt-get source' to match the source package to the binary. At this point, we have what amounts to a partial mirror but with very well defined limits. There are other ways of doing the same thing so anything that gets you to this point is as equal as the steps above, as long as the .debs exist alongside the .dsc and all associated source package files. What you need is full control over the list of packages at this stage, we'll worry about dependencies later.

  5. Simple scripting parses the .dsc to create suitable debian/files content (done).

  6. dpkg-genchanges regenerates the .changes file - critical for keeping the Gripped packages with the correct source packages in the final Grip repository

  7. Now run 'emgrip' against the .changes file to convert the .debs, update the .changes file with the new, smaller, .deb details and copy everything into a suitable incoming/ directory.

  8. point your repository software (tested with reprepro) at the incoming/ directory and generate your Gripped repository - which only contains the packages you listed at the start, just smaller. reprepro has a neat way of allowing foobar-1.2.3-2 to be included even if foobar-1.2.3-1 does not exist in the repository, as long as foobar_1.2.3.orig.tar.gz exists somewhere that reprepro can find and even though foobar-1.2.3-2.dsc does not mention the .orig.tar.gz. Neat! (Tested and working.)

  9. Run 'edos-debcheck' against the repository and "catch-up" with whatever packages are needed to make the repository as a whole "trimmed" (no unresolved dependencies).

  10. Keep the update script running as a cron task to pull updated packages from incoming.debian.org, run the .changes through emgrip and update your repository.

  11. Clean out the .debs, leave the source packages and restart the process for another architecture. (Crafty!)

  12. Possibly, to support those who can afford to do a small amount of compilation on their Emdebian Grip device, add a second repository that provides the Build-Depends that may be missing from the initial package set - stuff like automake and pkgconfig. Depending on how that works out, it might not be many packages and it may be easier to fold them into the mini-mirror anyway.

  13. Instead of 20,000 packages, I'm expecting something closer to 2,000, per mini-mirror.

  14. The process of "gripping" a package also drops numerous control fields (like Homepage and VCS*), as well as Recommends, Suggests and Enhances as well as truncating the long description at a maximum of four lines - all in the name of reduced package data sizes.

  15. The end result is that each mini-mirror contains a component package set for Emdebian Grip, using Debian packages. Mini-mirrors are still "well-trimmed" (no unresolved dependencies) because you get to choose which packages you include and if a particular package tries to bring in both Gtk and KDE dependencies, you can choose to omit that package and use a different one.

  16. Whether Emdebian Grip will or should support dropping certain binary packages from the repository is a difficult issue - Emdebian Crush is probably the preferred route if you need to start optimising an Emdebian Grip mini-mirror that harshly.

  17. Another wrapper will be needed to clean out cruft (old dependencies, outdated packages etc.) - that one may be harder to achieve by automation.


The first Emdebian Grip repositories will be for ARM, armel and i386, based on an XFCE environment package set.

"Gripping" all the packages in my /var/cache/apt/archives (about 1Gb) took about 30 minutes on a fast laptop that was doing other things at the time - the real packages will be "gripped" on the Emdebian server and for multiple architectures in sequence. It's certainly conceivable that any number of mini-mirrors can exist with any number of package sets (components), all with "gripped" packages. You can choose to "grip" .debs upon download from a normal Debian mirror but that wastes time if there is more than one device, it wastes space in /var/lib/apt/lists and /var/lib/dpkg/ and wastes bandwidth. For portable devices that you want to be using, not updating, I think that a mini-mirror is a worthwhile bonus.

Expect a 20% reduction in installation size (for more, you'll need the cross-building, functionality-modifying Emdebian Crush shebang) but a large decrease in the size of all the apt-cache data and dpkg status data. I'm hoping to push towards 30% reductions but most of these figures are guesswork right now.

And beyond the mini-mirrors? Debian-Installer, of course. I really need to set some time aside to prepare an Emdebian Grip installer using current D-I. Simple things like:

It really, really annoys me that I have to spend ages getting devices like the Acer Aspire1 working properly under Debian and that I have to take particular care during the installer to change the Debian default to something more sane for that kind of device. For Debian to be The Universal Operating System, the installer needs to be as smooth on an NSLU2* as on a standard desktop, a fast server and a cluster. This isn't just an architecture thing, there are big and small devices in all architectures, it needs to be a machine:variant thing. Mini-mirrors of single components are a start and part of the bigger answer is to support machine-specific configuration packages in these mini-mirrors. Packages that only exist in a specific mini-mirror and are specifically designed to setup a host of machine-specific settings, blacklist certain modules and enable others, set sensible settings in /etc/fstab and other places based on results from things like powertop, turning off features for some devices and turning them on for others. The whole point of such packages is that what they do would be insane on what would currently be considered a "typical Debian machine".

*(I'm not talking about an NSLU2 with an external hard drive here, I'm talking about fitting Emdebian Crush into the 6.4Mb of space on the NSLU2 itself so that external hard drives can be hot-plugged, as well as devices like the OpenMoko and iPAQs etc.)

Don't get me wrong, D-I does a fantastic job for the kinds of devices Debian currently supports but there are a host of other devices out there that need customised solutions where a mini-mirror and a machine-specific configuration package can remove huge amounts of hassle from the actual installation process. Emdebian Crush already uses machine:variant specific packages and the current Emdebian packages repository is, itself, a mini-mirror, albeit not well trimmed.

Let's make the most of "Debian Squeeze" - let's squeeze it onto more devices and make it even easier to get Debian onto anything.

Decklin Foster: Hello America, I missed you

As we walked back toward Mass Ave to catch the bus last night, fresh from the euphoria of my friend Josef's election-night party, I thought about how fortunate we are to be here, now. We cheered, we screamed, we sang the national anthem at the top of our lungs, popped champagne, and danced. I'm on some stranger's facebook somewhere, at the edge of a crowd of people smiling for what felt like the first time in years. I knew I'd remember, but I wanted to write something down. I'd been following along on Twitter all night, but I didn't know what to say when the good news finally came. I thought about our friends in California, where Prop 8 looked like it was going to win. Despite how far we've come, it's likely three states are about to write discrimination into their laws or constitutions rather than out of them. So I picked up my phone and just typed in, "charged." Everyone was excited, yes. I haven't felt electrified by the possibility of change like that in a very long time. But we also haven't automatically won change by changing our leaders. We have won a responsibility to make that change happen. "Yes we did", we joked to each other, and I emailed to my parents in the morning. But no one actually meant, "yes we can win an election". That's just the starting line. Now, we are charged with making America better. We are charged with protecting the liberties of everyone. I wanted to remember that too. That's why it matters that we won. Eight years from now, this could be a nation where the whole idea of "banning gay marriage" sounds as antiquated and offensive as segregated schools or not letting women vote. We have the ability to change our culture, nothing more. Mandates don't do that; people do. And yes, we can.

4 November 2008

Andrew Pollock: [life/americania] On being a fly on the wall for the 2008 Presidential Election...

(or Taxation without Representation) Wow, it's finally over, and best and right man won. I'm very excited and happy with the result. It has been very interesting being able to observe the entire process from start to finish. All the primaries and caucuses, the protracted battle between Clinton and Obama in the Democratic primaries, the record breaking money Obama raised, the campaigning, the misinformation. It'll be very interesting to see what the next four years brings. It's an exciting time to be living in America. California is an interesting state. It has "ballot initiatives". Its constitution can (and has been) amended by essentially referendums whenever there's some other excuse for a vote. Anyone can propose a ballot initiative with sufficient backing by petition. Wikipedia has a good writeup of all of California's propositions. Probably the most controversial one this time around is Proposition 8, which seeks to essentially reverse the same-sex marriage right that the Californian Superior Court ruled on earlier this year. This concept of ballot initiatives seems to trickle all the way down to the city level, where they're called Measures. San Francisco has some ridiculous number of them this time around. They use letters, and I think they go all the way to V. There's one to rename a sewage treatment plant after George W Bush, there's one to decriminalise prostitution. A co-worker who lives in San Francisco brought the official voter guide book to work with her today. It's about the size of a phone book and about 1.5 centimetres thick. I don't see how it's possible to make an informed decision on everything you're expected to vote on, when there's that much reading. The observation I've made is that Americans seem to have an inherent distrust of their governments, so that's why so many bits of it are directly elected by the people. Judges are voted for, county sheriffs are voted for. You vote for your school board. It seems like very few positions of any power are appointed by someone. Coming from Australia, where it's really the opposite, I'm not sure what I think is better. I think I like the simplicity of the Australian system. You vote for someone, and you essentially delegate power to and place trust in them.

Russell Coker: Support Gay Marriage in case You Become Gay

A common idea among the less educated people who call themselves “conservative” seems to be that they should oppose tax cuts for themselves and support tax cuts for the rich because they might become rich and they want to prepare for that possibility. The US census data [1] shows that less than 1% of males aged 15+ earn $250K. For females it’s less than 0.2%. On the Wikipedia page about homosexuality [2] it is claimed that 2%-7% of the population are gay (and 12% of Norwegians have at least tried it out). Apparently homosexuality can strike suddenly, you never know when a right-wing politician or preacher will suddenly and unexpectedly be compelled to hire gay whores (as Ted Haggard [3] did) or come out of the closet (as Jim Kolbe [4] did). So it seems that based on percentages you are more likely to become gay than to become rich. So it would be prudent to prepare for that possibility and lobby for gay marriage in case your sexual preference ever changes. But on a serious note, of the people who earn $250K or more (an income level that has been suggested for higher tax rates) there will be a great correlation between the amount of education and the early start to a career. Go to a good university and earn more than the median income in your first job, and you will be well on track to earning $250K. A common misconception is that someone who has not had a great education can still be successful by starting their own company. While there are a few people who have done that, the vast majority of small companies fail in the first few years. Working hard doesn’t guarantee success, for a company to succeed you need to have the right product at the right time - this often depends on factors that you can’t predict (such as the general state of the economy and any new products released by larger companies).

Share This

Ross Burton: Sound Juicer "Old Man Take A Look At My Life" 2.25.1

Sound Juicer "Old Man Take A Look At My Life" 2.25.1 has been released. Tarballs are available on burtonini.com, or from the GNOME FTP servers. Everyone's favourite Frockney did a huge amount of work on this, and I'm still talking to him after he admitted that the master plan is to replace Sound Juicer with Rhythmbox in Fedora!

Russell Coker: Basics of EC2

I have previously written about my work packaging the tools to manage Amazon EC2 [1]. First you need to login and create a certificate (you can upload your own certificate - but this is probably only beneficial if you have two EC2 accounts and want to use the same certificate for both). Download the X509 private key file (named pk-X.pem) and the public key (named cert-X.pem). My Debian package of the EC2 API tools will look for the key files in the ~/.ec2 and /etc/ec2 directories and will take the first one it finds by default. To override the certificate (when using my Debian package) or to just have it work when using the code without my package you set the variables EC2_PRIVATE_KEY and EC2_CERT. This Amazon page describes some of the basics of setting up the client software and RSA keys [2]. I will describe some of the most important things now: The command “ec2-add-keypair gsg-keypair > id_rsa-gsg-keypair” creates a new keypair for logging in to an EC2 instance. The public key goes to amazon and the private key can be used by any ssh client to login as root when you creat an instance. To create an instance with that key you use the “-k gsg-keypair” option, so it seems a requirement to use the same working directory for creating all instances. Note that gsg-keypair could be replaced by any other string, if you are doing something really serious with EC2 you might use one account to create instances that are run by different people with different keys. But for most people I think that a single key is all that is required. Strangely they don’t provide a way of getting access to the public key, you have to create an instance and then copy the /root/.ssh/authorized_keys file for that. This Amazon page describes how to set up sample images [3]. The first thing it describes is the command ec2-describe-images -o self -o amazon which gives a list of all images owned by yourself and all public images owned by Amazon. It’s fairly clear that Amazon doesn’t expect you to use their images. The i386 OS images that they have available are Fedora Core 4 (four configurations with two versions of each) and Fedora 8 (a single configuration with two versions) as well as three other demo images that don’t indicate the version. The AMD64 OS images that they have available are Fedora Core 6 and Fedora Core 8. Obviously if they wanted customers to use their own images (which seems like a really good idea to me) they would provide images of CentOS (or one of the other recompiles of RHEL) and Debian. I have written about why I think that this is a bad idea for security [4], please make sure that you don’t use the ancient Amazon images for anything other than testing! To test choose an i386 image from Amazon’s list, i386 is best for testing because it allows the cheapest instances (currently $0.10 per hour). Before launching an instance allow ssh access to it with the command “ec2-authorize default -p 22“. Note that this command permits access for the entire world. There are options to limit access to certain IP address ranges, but at this stage it’s best to focus on getting something working. Of course you don’t want to actually use your first attempt at creating an instance, I think that setting up an instance to run in a secure and reliable manner would require many attempts and tests. As all the storage of the instance is wiped when it terminates (as we aren’t using S3 yet) and you won’t have any secret data online security doesn’t need to be the highest priority. A sample command to run an instance is “ec2-run-instances ami-2b5fba42 -k gsg-keypair” where ami-2b5fba42 is a public Fedora 8 image available at this moment. This will give output similar to the following: RESERVATION r-281fc441 999999999999 default
INSTANCE i-0c999999 ami-2b5fba42 pending gsg-keypair 0 m1.small 2008-11-04T06:03:09+0000 us-east-1c aki-a71cf9ce ari-a51cf9cc The parameter after the word INSTANCE is the serial number of the instance. The command “ec2-describe-instances i-0c999999” will provide information on the instance, once it is running (which may be a few minutes after you request it) you will see output such as the following: RESERVATION r-281fc441 999999999999 default
INSTANCE i-0c999999 ami-2b5fba42 ec2-10-11-12-13.compute-1.amazonaws.com domU-12-34-56-78-9a-bc.compute-1.internal running gsg-keypair 0 m1.small 2008-11-04T06:03:09+0000 us-east-1c aki-a71cf9ce ari-a51cf9cc The command “ssh -i id_rsa-gsg-keypair root@ec2-10-11-12-13.compute-1.amazonaws.com” will then grant you root access. The part of the name such as 10-11-12-13 is the public IP address. Naturally you won’t see 10.11.12.13, it will instead be public addresses in the Amazon range - I replaced the addresses to avoid driving bots to their site. The name domU-12-34-56-78-9a-bc.compute-1.internal is listed in Amazon’s internal DNS and returns the private IP address (in the 10.0.0.0/8 range) which is used for the instance. The instance has no public IP address, all connections (both inbound and outbound) run through some sort of NAT. This shouldn’t be a problem for HTTP, SMTP, and most protocols that are suitable for running on such a service. But for FTP or UDP based services it might be a problem. The part of the name such as12-34-56-78-9a-bc is the MAC address of the eth0 device. To halt a service you can run shutdown or halt as root in the instance, or run the ec2-terminate-instances command and give it the instance ID that you want to terminate. It seems to me that the best way of terminating an instance would be to run a script that produces a summary of whatver the instance did (you might not want to preserve all the log data, but some summary information would be useful), and give all operations that are in progress time to stop before running halt. A script could run on the management system to launch such an orderly shutdown script on the instance and then uses ec2-terminate-instances if the instance does not terminate quickly enough. In the near future I will document many aspects of using EC2. This will include dynamic configuration of the host, dynamic DNS, and S3 storage among other things.

Share This

3 November 2008

Eddy Petrișor: signing packages heads-up

When trying to sign some Debian packages fails, you might think that temporally changing LANG to find similar errors on the web is a good idea.

But when you see that the expired key is not the default one and you made sure that the GPG agent environment info is correct, while signing still fails, you might start to consider that bad rendering of the "s comma below" character in your name might be worth taking into account.


gpg: skipped "Eddy Petri or ": secret key not available


Also, remembering that LC_MESSAGES=C is the right way to have English messages, while preserving all the rest of the localization info, is also useful.

Eric Warmenhoven

Dear Lazyweb, what’s best dedicated server/virtual private server offering? My basic requirements are: Right now I have Dreamhost, which has been reasonable for what I pay, which is $16/mo. They run Debian and, aside from root, give me everything else that I want. However, I’m starting to get really annoyed at being on shared hosting. Also, they’re moving everyone over to new servers, and when they do, I won’t be able to use procmail filtering anymore. So I’m looking for alternatives. One of my friends had used ServerPronto, which meets all my requirements. However, his experience convinced me that I don’t want to be using ServerPronto. So, what should I use?

Isaac Clerencia: Not again

Every Democrat was pretty sure Al Gore was going to win in 2000. Most polls said John Kerry was going to win in 2004. And yet both times they managed to snatch defeat from the jaws of victory. Just remember how you felt then and try your best so it doesn’t happen again. There is so much at stake … the US and the world can’t afford 4 more years of a Republican US President. Even if the GOP seems to be defeated, shattered and panicking, Republican voters always end up voting for their guy on election day, while Democrats have great problems closing the deal. It has to be different this time. We need Obama to win tomorrow, and we need him to win big. And a Democratic wave to get a filibuster-proof Senate majority and a comfortable victory in the House of Representatives, so they can enact a progressive agenda in the upcoming years and restore some of the damage from the last two Bush terms. We also need a solid ‘no’ to Proposition 8 in California and similar ones in other states, so no excuse to stay at your place just because you are in one of the bluest states. I would like to ask you to do your best tomorrow, but probably Larry Lessig does a better job at it. Once we are past this, the hard work starts, trying to hold every elected representative accountable, draft better candidates for future elections, spot DINOs, …, but for now, let’s focus on tomorrow. I am hoping for the best, and I wish I could do more about it, but it’s up to you Americans. Do the right thing and leave everything on the road, you don’t want to regret it tomorrow.

Enrico Zini: rc-buggy packages sorted by popularity

rc-buggy packages sorted by popularity After doing a round of tag reviews so that we release lenny with the latest tag contributions, I still had a bit of spare time. Since people are urging to fix the last bugs I thought I could contribute a list of RC-buggy packages sorted by popularity: 
#!/usr/bin/python
import urllib2
import sys, re #, gzip
PKGRE = re.compile(r'<strong>Package: </strong><a href="[^"]+" name="([^"]+)">')
POPCONRE = re.compile(r'^(\d+)\s+(\S+)')
PKGMAPRE = re.compile(r'^([^(]+)\(([^)]+)')
COMMASPLIT = re.compile(r'\s*,\s*')
# Allow to download other package views
if len(sys.argv) > 1:
    url = sys.argv[1]
else:
    url = "http://bts.turmzimmer.net/details.php"
def read_packages(url):
    "Download the list of rc-buggy packages"
    fd = urllib2.urlopen(url)
    for line in fd:
        mo = PKGRE.search(line)
        if mo:
            yield mo.group(1)
# FIXME: this is a deprecated data source, but at the moment it is just what
#        is needed
def read_bintosrc(url = "http://qa.debian.org/data/ddpo/results/ddpo_packages"):
    "Download the binary to source package map"
    fd = urllib2.urlopen(url)
    for line in fd:
        mo = PKGMAPRE.search(line)
        if mo:
            src = mo.group(1)
            bin = COMMASPLIT.split(mo.group(2))
            for p in bin:   
                yield p, src
def read_popcon(url = "http://popcon.debian.org/by_inst"):
    "Download popcon information"
    fd = urllib2.urlopen(url)
    # It doesn't work on urllib2 objects because it wants tell()
    #fd = gzip.GzipFile(fileobj=fd, filename=url, mode="rb")
    for line in fd:
        mo = POPCONRE.match(line)
        if mo:
            yield mo.group(2), int(mo.group(1))
print "Reading mapping of binary packages to source packages..."
bin2src = dict(read_bintosrc())
print "Reading popcon scores..."
# We actually want the score of source packages, and for it we use the score of
# its binary package with the highest score
scores = dict()
for pkg, score in read_popcon():
    pkg = bin2src.get(pkg, pkg)
    if pkg not in scores:
        scores[pkg] = score
#scores = dict(read_popcon("http://popcon.debian.org/by_inst.gz"))
print "Reading list of rc-buggy packages..."
# The bug page has a mix of source and binary packages: convert all to source
# packages
packages = sorted(read_packages(url), key=lambda x:scores.get(bin2src.get(x,x), 0))
for p in packages:
    print scores.get(bin2src.get(p,p), 0), p
This can be run on people.debian.org as ~enrico/popzimmer (0 is the rank of packages that for some reason cannot be mapped to any rank): 
$ ~enrico/popzimmer 
Reading mapping of binary packages to source packages...
Reading popcon scores...
Reading list of rc-buggy packages...
0 roxen-fonts-iso8859-2
2 dpkg
11 pam
61 glibc
61 libc6
123 initramfs-tools
149 bind9
150 grub
180 portmap
213 nfs-common
279 libgl1-mesa-dri
298 libsnmp-base
298 snmpd
309 avahi
397 mysql-dfsg-5.0
408 xserver-xorg-input-wacom
409 xserver-xorg-video-vesa
428 docbook-xml
499 openoffice.org-core
567 xine-lib
600 evolution-data-server
669 eog
671 xserver-xorg-video-intel
691 epiphany-browser
691 epiphany-gecko
695 kdelibs4c2a
698 uswsusp
754 guile-1.6
959 gpm
964 cups
990 kdebase
1002 ghostscript
1069 giflib
1076 xulrunner-1.9
1177 libapache2-mod-perl2
1217 vlc-nox
1229 kernel-package
1311 kdeutils-dev
1520 libparted1.8-10
1767 wireshark
1785 selinux-policy-default
1908 dia
1943 htop
1983 boost
2140 cantlr
2145 transmission
2159 emacs22
2312 swfdec-mozilla
2449 eclipse
2449 libswt3.2-gtk-java
2728 lilo
2898 libnss-ldap
3013 java-access-bridge
3051 blender
3099 libtemplate-perl
3499 fglrx-atieventsd
3602 virtualbox-ose
3619 jfsutils
3806 anjuta
3962 kbuild
4076 libgnustep-gui0.14
4222 libwebkit-1.0-1
4332 rosegarden
4412 matplotlib
4412 python-matplotlib
4950 mail-notification-evolution
5195 nut-cgi
5273 sbcl
5538 ruby1.9
5679 kmymoney2
6076 websvn
6147 miro
6264 gnat-4.1
6431 gallery2
6478 gnome-chess
6512 joystick
6862 istanbul
6937 wordpress
7476 luatex
7635 csound
7635 python-csoundac
8257 libpam-mount
8471 wmcpuload
8631 nbsmtp
8642 oxine
8774 axiom
8949 libengine-pkcs11-openssl
9052 clamav-getfiles
9221 open-iscsi
9262 ptex-bin
9701 egroupware-core
10012 debian-cd
10127 libjogl-java
10453 libjson-xs-perl
10698 gcjwebplugin
10736 request-tracker3.6
11294 python-extclass
11518 rkward
11600 alevt
12929 otrs2
13205 bindgraph
13956 icecc-monitor
14223 netmaze
14505 cdebconf
14526 mumble
14687 fnonlinear
14724 recite
14789 jbidwatcher
15766 libnss-ldapd
15812 glpi
15905 lustre-source
15982 win32-loader
16232 ampache
17168 libghc6-missingh-dev
17311 acl2
17830 apertium
18462 emacspeak
18469 glassfish
19257 isight-firmware-tools
19719 libnagios-object-perl
20144 libwx-perl
20465 dns2tcp
21614 python-kinterbasdb-dbg
21690 hf
21902 libcobra-java
22365 tomcat6
22573 libphp-snoopy
23501 coco-java
23926 boxbackup-server
24009 libghc6-wash-dev
26323 libdesktop-notify-perl
26439 opendb
28426 adolc
28795 xorp
29986 libgdata-java
31262 libjboss-serialization-java
31472 pixelpost
31658 cournol
32939 mediamate
34365 gforge-plugin-scmcvs
34659 libgearman-client-async-perl
42233 libhamcrest-java
48076 mahara
55303 mxallowd
85881 roxen-fonts-iso8859-1
So, if you don't know from what package to start fixing bugs, "Begin at the beginning,", the King said, very gravely, "and go on till you come to the end: then stop."

Stefano Zacchiroli: camlbz2 bzip2 ocaml bindings

CamlBZ2: OCaml bindings for libbz2 Since the next building block for proper compression support in Batteries is bzip2 integration, I started from the basic ingredient: OCaml bindings for bzip2. It was not entirely missing: the old GZ project by Olivier Andrieu was in fact offering bzip2 bindings, but was doing that in tandem with gzip bindings (which already have a standard de facto implementation: Camlzip). Hence I contacted the author and together we decided to split away the two parts and releasing CamlBZ2 as a new project, devoted to bzip2 bindings alone. The old code has been ported to latest OCaml versions and a couple of bugs have been squashed in the interim. The project is now hosted on the OCaml forge, has its own homepage, and a Git repo (look ma, it looks like a serious project!). Today I've made the first alpha release: CamlBZ2 0.5.99 is now available. It is intended to be a test release ... so please test it :-) and report any issue you find using the tracker. Next steps for CamlBZ2:

Jonathan McDowell: NM gets longer and longer

If you ask the old timers about how they got started in Debian they'll tell you about sending off an email saying they wanted to get involved and having an account setup the same day. By the time I got round to joining it was during tbm's blitz when NM opened up again and it took only a few days to get through the process. When I first AMed, back in 2002, it took anywhere from a few days to a few months to process an applicant. I've recently started AMing again and so far my NM and I have exchanged 9 mails over the course of 2 weeks and still aren't through Philosophy and Procedures.

This isn't due to any failing on the part of my NM. It's just that there's a lot more "paperwork" to get through these days. There's currently a lot of discussion going on about the idea of introducing new classes of Debian involvement. I think that's something that's definitely worth looking at; it's already in existence for Debian Maintainers vs Debian Developers. However anything new that's introduced really needs to make some effort to be lighter than NM; someone who just wants to handle a few packages is going to get put off by too many hoops and the current NM process is already largely constrained by available AM time rather than anything else. I don't pretend to know what the right solution is, but I think anyone attempting to produce one should be praised for that rather than attacked.

Russell Coker: Christian Principles in an Election Year

The National Council of Churches in the US [1] has produced some advice for Christian voters titled Christian Principles in an Election Year [2]. It starts by saying “Your church, your communion, and the National Council of Churches USA do not endorse any political party or any candidate” (which is in bold in their text) and then lists 10 issues that Christians should consider when voting. Here are the 10 section headings, the full article has a sentence or two explaining each one. I think that most good people (regardless of religion) will agree with all of this - maybe substitute “God” with the name of some other entity that has not been proven to exist or with “humanity”.

  1. War is contrary to the will of God.

  2. God calls us to live in communities shaped by peace and cooperation.

  3. God created us for each other, and thus our security depends on the well being of our global neighbors.

  4. God calls us to be advocates for those who are most vulnerable in our society.

  5. Each human being is created in the image of God and is of infinite worth.

  6. The earth belongs to God and is intrinsically good.

  7. Christians have a biblical mandate to welcome strangers.

  8. Those who follow Christ are called to heal the sick.

  9. Because of the transforming power of God s grace, all humans are called to be in right relationship with each other.

  10. Providing enriched learning environments for all of God s children is a moral imperative.

The Blogger Joy Reid [3] often uses the term “Matthew Christian” to refer to Christians who follow the book of Matthew and act in ways that would be considered to be good by most people regardless of belief. This is in stark contrast to some of the nasty people who call themselves Christian and who promote hatred, inequality, and war - such people give all Christians a bad reputation (see the comment section of any blog post concerning religion for examples). John Goerzen’s post titled Politics and the Church (which references the NCCUSA article [4]) is also worth reading. Interestingly his blog post had a Google advert for “Christian Masturbation” when I viewed it. John also has a good post explaning why he is voting for Obama - based on his Christian beliefs and “Traditional Values” [5].

Share This

Adeodato Sim : Software that rocks

From the “Software that recently rocked my pants” department: I also thank Martin F. Krafft for helpfully updating the encryption howto I followed for my previous laptop to mention that using a single encrypted device and LVM on top of it is (possibly) preferred nowadays. And to Simon McVittie for his idea of a small unencrypted Debian installation for recovery purposes in laptops without an optical drive. P.S.: I can’t believe this shit about blogging every day in November.

Next.

Previous.