Search Results: "xam"

15 April 2008

Lars Wirzenius: Stolen laptop, call home!

The Finnish computer magazine Tietokone has an article on stolen laptops calling home and the police using this to track down thieves. The article reads like a press release from the proprietary software company mentioned in the article, although perhaps that's just because it only mentions that one company. However, the main point is that by having the stolen laptop report its IP number, police can trace it and recover it. In the interest of helping Unix users have similar protection, I provide you with the following crontab snippet:
*/15 * * * * wget -q -O /dev/null http://your.home.page/call-home; true
That page does not need to exist, you just need to be able to access the httpd logs for the web server it accesses to be able to find out the IP number. By using a distinctive URL, you make it easier to grep. Perhaps someone would like to turn that into a Debian package? Add some debconf magic to specify the URL to access, and it should be easily usable. For all the people without a web site of their own, perhaps there would be people interested in operating a server people could use for that. (There's some privacy issues involved, but it's an opt-in thing, obviously, so that lessens them.) Now, to be fair, the proprietary software package mentioned does more than my cron job. For example, they can take control of the system remotely, and wipe it out and make it inoperable. Personally, I don't want that, since it's a huge security risk, but if that kind of thing interests you, you could enhance the cron job a bit. For example, it could download the file and verify that it's GnuPG signed with your key, and if so, execute it as a shell script. One of the risks with this is that if anyone ever happens to get a copy of a file you've made like that, say, your web hosting provider, they can put it there whenever they want to, and boom goes your laptop. I'm sure there are ways around that, but if you want the system, you'll have to solve that yourself.

14 April 2008

Steve Kemp: So helpless against what is coming.

I've made a new release of the chronicle blog compiler. There are a couple of minor changes to the variables exported to the theme templates, as contributed by MJ Ray, and a new spooling system. This works in a simple fashion, and allows you to queue up posts. For example If you write a new entry containing the psuedo-header "Publish: 20th April 2008" and you have a crontab entry containing this:
  chronicle-spooler
    --spool-dir=~/blog/spool/  \
    --live-dir=~/blog/data/  \
    --post-move='cd ~/blog && make upload'
It works as expected. When you call this on the 20th April the file will be moved from ~/blog/spool into ~/blog/data, and your blog will be rebuilt & uploaded. The implementation was different than the original suggestion, but is nice and clean, and can be used to do other things with a little bit of effort. Anyway if you see this entry the spooling system is working! ObQuote: 30 Days of Night.

13 April 2008

Russell Coker: Debian Work and Upstream

Steve Kemp writes about security issues with C programs [1]. It seems obvious that if you are going to do something that is overly tricky (such as anything related to setuid programs) then you should have a good knowledge of what you are doing. Steve goes a little further and suggests that anyone who doesn’t know C should not package a C program. Andrew Pollock is a Debian developer who doesn’t program in C and yet packages some C programs, he explains why he thinks that it’s correct to forward C coding bugs upstream [2]. In this debate I have more sympathy for Steve’s position. Security is one of the most important things in software that we develop, if you can’t develop secure software then IMHO you shouldn’t distribute any software. Although it should be noted that the two posts in question don’t directly conflict. Steve’s main point (your should know the code well to do something relevant to security) does not directly conflict with Andrew’s main point that a non-C coder can maintain the Debian packaging files and forward bug reports related to C code upstream. In the more general case it’s often impossible to know all languages used in a project. It’s not uncommon for a single project to have core code written in C, a configuration file format (which is a programming language), a Makefile (GNU Make is a fairly complex interpreter), Autoconf/Automake (even more complex tools for creating complex Makefiles, and then some Perl or Python to manipulate input or output data. The SE Linux project has code in all those languages plus the M4 macro language. It seems obvious that in a large project the number of people who are capable of understanding all languages which are used is going to be a small sub-set of all Debian Developers. It also seems likely that someone who knows some of the languages really well may be capable of doing a better job than someone who has a passable knowledge of all of them. For the vast majority of my work in Debian I have been a member of the upstream project in question (for Bonnie++, Postal, Portslave, Maildir-bulletin, and logtools I’m the sole upstream developer). Often I start maintaining a package with the intent of doing the minimum needed to keep it maintained in Debian but then end up doing upstream work, for example I briefly had write access to the CVS repository for pppd when I was maintaining it and I had write access to the KDE CVS repository when I was working on KDE packages (which incidentally was before I became an official DD). When I first started working on the SE Linux project I aimed to merely package the code in question for Debian. I was not planning on spending seven years doing upstream work! When I’m working on a project I spend my time working on the things that most need attention. That often means that I get diverted from Debian packaging bugs (which are often minor issues) to work on upstream bugs. In many ways my work practices make me a Debian user and upstream developer who also does Debian packaging rather than a dedicated DD. This of course does have a detrimental affect on the quality of my Debian packaging work - but overall I believe that it’s best for the users! As an upstream developer who is focussed on Debian there are significant advantages in being a DD. I can immediately upload a new package to Unstable without any delay, and there is the potential to take over a related package (either be given the package or NMU) if it’s the best way to get around a roadblock. Another benefit is that in certain limited situations I can speak on behalf of Debian (in terms of packages I “own”). When it comes to upstream development of SE Linux I can make decisions about how things will be done in Debian and work with other upstream developers to ensure that cross-distribution compatibility works reasonably well. It seems to me that a fairly ideal process for managing Debian development would have the people who are capable of the hard coding doing so and some people who haven’t got specific coding skills working on the packaging. I would be very interested in receiving patches related to the Debian packaging of the packages I “own” (in Debian it’s widely regarded that a man’s package is his castle) so that I can spend more of my time on the C coding. I’m not bothered about NMUs (a Non-Maintainer Upload - is when someone else uploads a new version of one of my packages to fix a bug). If anyone would like to NMU one of my packages they are welcome to do so as long as there is a suitable combination of bug urgency, simplicity of the fix, and lack of response from me. If someone wants to start NMUing my packages to fix bugs reported by the latest version of Lintian in Unstable (Lintian is the Debian tool to automatically check for packaging bugs) then that’s OK. Of course there are a variety of other situations that DDs sometimes find themselves in. There have been situations of overtly hostile upstream developers. I try and avoid associating with people who have certain difficulties in interacting with others. If someone is going to change their license and try to back-date the change or create a license that makes it difficult to distribute modified versions of the code then I’m inclined to avoid even using the software, let alone being involved in developing it. This means that the software that I use in any serious way has reasonable people doing upstream development who welcome new people to join their work.

Gunnar Wolf: Some Rails plugins: acts_as_magic_model

This is the third plugin I have been working on for Rails - It's the one that still needs most work to be fully usable (for what I originally envisioned it), but is almost there - I have not touched it (nor the other two I recently presented here, acts_as_catalog and Real FK because real-life has demanded my time on various other fronts.
Anyway, this is IMHO the most interesting of the three plugins I've written so far.
So, following what I did with the other plugins: You can go to acts_as_magic_model's main page, or jump straight to its SVN repo. And the basic description follows:
Rails plugin to allow for extensible models, where inter-model relations are inferred from the database's structure, and not necessarily from explicit declarations in the models.
This means, when one of your models acts_as_magic_model, this plugin will try to infer all the basic relations (has_many, belongs_to and has_and_belongs_to_many) its base table has with other tables in the database, adapting the model correspondingly. Using the plugin In order to use this plugin, you only have to declare it in your model. As an example, say you have the proverbial project-management system, where you have projects (each of which can be of several different types) and people, and a HABTM relation between people and projects. So, you have the following tables (expressed as migrations here):
    create_table :people, :force => true do  t 
      t.column :name, :string
    end
    create_table :project_types, :force => true do  t 
      t.column :name, :string
    end
    create_table :projects, :force => true do  t 
      t.column :name, :string
      t.column :project_type_id, :integer
    end
    create_table :people_projects, :force => true, :id => false do  t 
      t.column :person_id, :integer
      t.column :project_id, :integer
    end
Using acts_as_magic_model, you can skip declaring the relations in your models, like this: (of course, each of the classes still has to be declared in its own file)
    class Person < ActiveRecord::Base
      acts_as_magic_model
    end
    class Project < ActiveRecord::Base
      acts_as_magic_model
    end
    class ProjectType < ActiveRecord::Base
      acts_as_magic_model
    end
And the gaps will be filled in for you - that is, you can go on and ask for Project.find(:first).people, ProjectType.find(:first).projects.size, Person.find(:first).projects.map pr pr.project_type , and whatever you fancy.
In future versions of the plugin, I expect even to get rid to explicitly declare the class declarations.
Of course, you should be warned: Initializing a model using this plugin is quite database-intensive. It will clutter your logs, and it might be unbearable if you use it in development mode. Still, its convenience is very often worth it.
Its main uses should be when prototyping and when designing a system that needs to be flexible to the data models themselves - If you don't expect your system's data structures to be highly malleable, you should probably use acts_as_magic_model only as a prototyping aid.

Gintautas Miliauskas: Perl program 'design'

Acme::Eyedrops allows you to 'design' your Perl programs (if you have any; you'd better not). Basically, given a Perl program and a "shape" file, it produces a perl program that visually resembles the given shape and behaves like the former program. Here's my favorite example:
    ''=~(        '(? '        .(' '         '%')        .('['        ^'-')
    .(' '         '!')        .(' '         ',')        .'"'.        '\\$'
    .'=='        .('['        ^'+')        .(' '         '/')        .('['
    ^'+')        .' '        .(';'        &'=')        .(';'        &'=')
    .';-'        .'-'.        '\\$'        .'=;'        .('['        ^'(')
    .('['        ^'.')        .(' '         '"')        .('!'        ^'+')
   .'_\\ '      .'(\\$'      .';=('.      '\\$= '      ."\ ".(      ' '^'.'
  ).((' ')     '/').').'    .'\\"'.+(    ' '^'[').    (' ' '"')    .(' ' '/'
 ).('['^'/')  .('['^'/').  (' ' ',').(  ' ' ('%')).  '\\".\\"'.(  '['^('(')).
 '\\"'.('['^  '#').'!!--'  .'\\$=.\\"'  .(' '^'[').  (' ' '/').(  ' ' "\&").(
 ' '^"\[").(  ' ' "\"").(  ' ' "\%").(  ' ' "\%").(  '['^(')')).  '\\").\\"'.
 (' '^'[').(  ' ' "\/").(  ' ' "\.").(  ' '^"\[").(  '['^"\/").(  ' ' "\(").(
 ' ' "\%").(  ' '^"\[").(  '['^"\,").(  ' ' "\!").(  ' ' "\,").(  ' ' (',')).
 '\\"\\ '.+(  '['^"\+").(  '['^"\)").(  ' ' "\)").(  ' ' "\.").(  '['^('/')).
 '+_,\\",'.(  ' '^('[')).  ('\\$;!').(  '!'^"\+").(  ' '^"\/").(  ' ' "\!").(
 ' ' "\+").(  ' ' "\%").(  ' '^"\[").(  ' ' "\/").(  ' ' "\.").(  ' ' "\%").(
 ' '^"\[").(  ' ' "\$").(  ' ' "\/").(  '['^"\,").(  ' ' ('.')).  ','.((' ')^
 '[').("\["^  '+').("\ "   '!').("\["^  '(').("\["^  '(').("\ "^  '[').("\ " 
 ')').("\["^  '/').("\ "^  '[').("\ "   '!').("\["^  ')').("\ "   '/').("\["^
 '.').("\ "   '.').("\ "   '$')."\,".(  '!'^('+')).  '\\",_,\\"'  .'!'.("\!"^
 '+').("\!"^  '+').'\\"'.  ('['^',').(  ' ' "\(").(  ' ' "\)").(  ' ' "\,").(
 ' ' ('%')).  '++\\$=" )'  );$:=('.')^  '~';$~='@'   '(';$^=')'^  '[';$/=' ';

12 April 2008

Evan Prodromou: 24 Germinal CCXVI

I'm pretty impressed with the new Geotag Icon Project. The well-known feed icon has become ubiquitous, and it makes sense to have a free-to-use icon for geographically-tagged information. I can see it being very useful for hcard and geo microformats, for example. tags:

Relayed for your review without comment For all those who think Open Content guidebooks like Wikitravel are by definition less reliable than proprietary ones: "Travel writer tells newspaper he plagiarized, dealt drugs". "A Lonely Planet author says he plagiarized or made up portions of the popular travel guidebooks and dealt drugs to supplement poor pay. [...] He didn't travel to Colombia to write the guidebook on the country because 'they didn't pay me enough. I wrote the book in San Francisco. I got the information from a chick I was dating.'" Cough, cough. tags:

Aigars Mahinovs: Debian development basic links

For a Debian package creation seminar:

10 April 2008

Fathi Boudra: Qt 4.4.0 rc1 reached Debian Sid

Qt 4.4.0 rc1 was uploaded yesterday to unstable. It was previously available in experimental. A rework of the packages was done : Qt4 is built on alpha, amd64, i386, ia64, powerpc, s390 and sparc architectures. On other architectures, the build is in progress. It failed to build on mips/mipsel so far. And I confirm: Qt4.4.0 is a buildd killer. It needs 4x the time and double space: Automatic build of qt4-x11_4.4.0~rc1-2 on debian-31.osdl.marist.edu by sbuild/s390 98
[…]
> Build needed 04:19:38, 4733360k disk space vs. Automatic build of qt4-x11_4.3.4-2 on lxdebian.bfinv.de by sbuild/s390 98
[…]
Build needed 01:14:03, 2221884k disk space Enjoy !

9 April 2008

Adrian von Bidder: Mercurial

After my moderately successful attempts at learning git, I have now tackled Mercurial and found it easy to understand and use. I can't quite see where git is better, so I guess I'll stick with hg now. Output so far is a script I call hgdebuild, which I'll probably package when it's a bit less rough around the edges (yes, I know about hg-buildpackage, but I didn't really like it, and it appears to be orphaned at the moment anyway). Also, with Mercurial, I very easily could address the main problem I've always had with distributed revision control systems: With Subversion, it's easy to work on different computers, and backing up my home directory has become less critical (an important aspect on my Laptop). Both of these are not true when suddenly .hg in my working dir is the backing store.
$ cat ~/.hgrc
[hooks]
commit.push = [ -f .hg/dopush ] \
    && ping -w 1 -c 2 -q -i .2 mygitserver.example.com >/dev/null 2>/dev/null \
    && hg push \
      true
(Probably this problem has been solved before. Comments welcome.) Comments: From the person with madduck in his email address: Bzr checkouts, or a similar Git hook.

8 April 2008

Russell Coker: Olympics and Politics

The latest news related to the Beijing Olympic games is that Kevin Rudd (our Prime Minister) has said “It is absolutely clear that there are human rights abuses in Tibet. That’s clear-cut; we need to be upfront and absolutely straight about what’s going on,” - stating the obvious really. If that was all that happened then it wouldn’t be particularly interesting. The paper version of “The Age” quotes George Bush as seconding Kevin Rudd’s statement. I believe that this is quite significant. Bush is well known for being stupid, the fact that he is now following the example of someone else who is doing good things is a very positive thing for the world. When John Howard was our Prime Minister we had an idiot (Howard) following an even bigger idiot (Bush) and the result was not good. Kevan Gosper (an Australian IOC board member) said “They just take their hate out on whatever the issues are at the time, and that hate against the host country is being taken out on our torch“. People who are totally corrupt sometimes seem confused when other people are motivated by moral principles, maybe we have a culture clash between the corrupt IOC board and the attitudes of most people in the rest of the world (I can’t think of any other way to map Kevan’s statement to reality). Mr Rudd has confirmed that Chinese “security guards” (soldiers) will not be permitted to operate in Australia to protect the Olympic torch from protesters, but the “security guards” have been operating in the UK. I read about this on the web site of The Age, but I won’t link to them because they have a lot of broken links with the following explanation - I am not going to link to sites that are so transient in nature (linking to The Age would lower the quality of my blog). Incidentally does anyone know of a news service in Australia that has reliable pages which stay online? We could not find the page you requested. This is often because older content has been removed from our site. In most cases you can still find the item via our archive service, News Store, where you can buy articles for a small fee. It also has the following text which indicates that the most visible problem is probably transient, but the fact that they deliberately break links is unacceptable to me: If you reached this page from a link on our site, please contact the webmaster (choose Technical faults) and tell us the address of the faulty page and the address shown for this page. The MSN article about the meeting between Bush and Rudd also had an interesting quote from Nancy Pelosi (speaker of the house in the US congress) [1]. She said “As I said in India last week where I met with His Holiness the Dalai Lama, if freedom-loving people throughout the world do not speak out against China’s oppression in Tibet, we have lost our moral authority to speak out on behalf of human rights anywhere in the world” and that the IOC made a mistake in awarding the games to China. Pelosi is not known for being left-wing (the US Democratic party is centrist/right by the standards of most democracies) so it’s interesting to see her take a stand on this issue. Erich Schubert has written about this, he points out that the Olympics are not about sports [2]. Well of course the Olympics are about money, drugs, and sports-science! I’m not sure that I agree with Schubert and Pelosi, the Chinese Olympics has focussed a lot of international attention on what China is doing - this has to be good for human rights. The Olympic games are a white elephant, running them costs a huge amount of money and there is no evidence that they actually make money for the host country once the opportunity costs are taken into account. Maybe we should give the Olympics to Zimbabwe or the Sudan next?

6 April 2008

Filip Van Raemdonck: I'm junior now

Learning that the Linux Professional Institute offered a chance for taking exams at a reduced price during this year's FOSDEM, I decided to take a shot at the level 1 exams.<br /> I was fairly occupied in the period before FOSDEM when I took that decision, but given that I've been using Linux heavily for over a decade, and after reviewing the objectives, I figured that I should be able to pass it without specific preparation.<br /> <br /> Yesterday I received an email that indeed I did, so now I am officially a Junior level Linux professional .<br /> I guess I should take a look at getting LPIC-2 sometime.
Continue reading "I'm junior now"

5 April 2008

Steve Kemp: You know how to use candles?

Mutt
Fran ois Marier has recently been posting some interesting entries about the Mutt mail client upon his blog. His tips are pretty basic, but that doesn't make them less useful. So here's my tip of the day: reply_regexp. When you're viewing a mail, and you choose to reply it the subject of that mail is the basis of your message's subject. For example given a message with "Subject: hello" your reply will typically have the subject "Subject: Re: hello". This is real rocket science here, people. Imagine you're using SPAM filtering which tags messages it isn't sure about with a prefix "UNS:". Suddenly things don't look so hot, as you might end up with a mail with the subject:
Subject: UNS: Re: Hello
Reply to that mail, whilst being half-asleep, and what do you get? You get this:
Subject: Re: UNS: Re: Hello
Bad. Ugly. Wrong. The following snippet in my ~/.muttrc file correctly deals with this case:
set reply_regexp="^(((UNS:[ \t]) [rR][eE]:[ \t])*)+"
Neat. Cool. Nice.
Advertising
I've paid for some advertising upon the LWN.net site. (No link, I'm not trying to game things here.) I didn't know what to expect, but I was willing to risk the expenditure as a way of saying thanks to them for their great content. (Because of my Debian Project membership I get a free LWN subscription, as do all developers. see here for details if you're a developer without a subscription.) I've paid $30 and that has given me a months run of my advert, with clickthroughs hovering around 1%. Not horribly bad at all! I probably won't repeat the experiment for the forseeable future, but I'm glad that I did it at least once. If you have something appropriate for a Debian-based audience don't forget you're welcome to advertise upon the Debian Administration website for free - See the Advert FAQ for details.
ObQuote: The Craft Update: jquery was accepted yesterday. Today I uploaded a new version to more closely align it with the javascript-policy.

3 April 2008

Steve Kemp: It's just a puzzle box!

Chronicle
I've made a new release of the chronicle blog compiler, primarily to allow the "Subject:" header to be used for new blog subjects. (That allows new entries to be automatically posted via email, with an appropriate procmail setup. I'll add one as an example shortly.)
RSS Utility
Whilst on the subject of RSS creation (huh?) I've written a tiny utility which will create an RSS feed from a list of text files. It will also create an index.html file to match. To see why this is useful you could view my recent changelog. I think there is a need for a small tool to read files and create feeds from them - like mod_index_rss does, but without messing with Apache. If there is any interest I'd be happy to release the code, as-is it doesn't use a template..
Anonymous Hosting?
Online privacy is important. Mostly when this is discussed it is in the context of client-side anonymity. Looking at it from the other side, though, How do you host a website anonymously? You could register the domain via a proxy, or with bogus details. But if you host the site yourself the IP address may be traced to the hosting provider, and that may be used to trace back to you. So, the alternatives? Well you could use a hosted site such as livejournal / wordpress / googlepages / etc. But pretty surely they'll be able to trace content back to you - and if you don't host it there's a high chance they'll just pull it if you talk about "bad things". (I guess you could use TOR for uploading / your connections there.) So, going back to the question. How can you host something, easily accessible to the world, without risk of your identity/association being discovered?
I'm, obviously, ignoring FreeNet. Two reasons for that:
  • It's slow, has no search-engine goodness, and is unproven.
  • It requires an atypical client. Aunt Milly won't be able to surf Freenet...
I almost think the best way forward would be to write a site which was a proxy for a file-sharing protocol, then link people to items that way. Relying on the swarm to host the files.. The downside is that you'd have to have a convincing argument for when RIAA comes calling, suggesting that you're sharing their stuff too. If it wasn't a general purpose proxy then the deniability is gone, and if it is you're at risk of general copyright infringement claims. Hard problem. Shame.
ObQuote: HellRaiser

David Pashley: InnoDB being silently disabled

Regular viewers will know that I don't think favourably of MySQL. Here is yet another reason. Let's create an InnoDB table:
mysql> CREATE TABLE  User_  (
mysql> ...
mysql> ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
Query OK, 0 rows affected, 1 warning (0.04 sec) 
One warning, but we're running this as part of an import, so we'll fail to spot this and even if we did, we wouldn't be able to get it back out of mysql because SHOW WARNINGS only shows the last command. So let's look at the table we just created:
mysql> show create table User_\G
*************************** 1. row ***************************
       Table: User_
Create Table: CREATE TABLE  User_  (
...
) ENGINE=MyISAM DEFAULT CHARSET=latin1
1 row in set (0.00 sec)
Eh? what's going on? We asked for InnoDB, but have got a MyISAM table. Lets look at the engines available.
mysql> show engines;
+------------+----------+----------------------------------------------------------------+
  Engine       Support    Comment                                                         
+------------+----------+----------------------------------------------------------------+
  MyISAM       DEFAULT    Default engine as of MySQL 3.23 with great performance           
  MEMORY       YES        Hash based, stored in memory, useful for temporary tables        
  InnoDB       DISABLED   Supports transactions, row-level locking, and foreign keys       
  BerkeleyDB   NO         Supports transactions and page-level locking                     
  BLACKHOLE    NO         /dev/null storage engine (anything you write to it disappears)   
  EXAMPLE      NO         Example storage engine                                           
  ARCHIVE      YES        Archive storage engine                                           
  CSV          YES        CSV storage engine                                               
  ndbcluster   DISABLED   Clustered, fault-tolerant, memory-based tables                   
  FEDERATED    YES        Federated MySQL storage engine                                   
  MRG_MYISAM   YES        Collection of identical MyISAM tables                            
  ISAM         NO         Obsolete storage engine                                          
+------------+----------+----------------------------------------------------------------+
12 rows in set (0.00 sec)
Oh, so innodb has been disabled. We can fix that easily by removing skip-innodb from my.cnf.
root@cmsdb01:/var/log# grep skip-innodb /etc/mysql/my.cnf
root@cmsdb01:/var/log#
But hang on a second, that's not in the config file. What's going on? It turns out that the reason InnoDB is disabled is because of the innodb_log_file_size setting not matching the files on disk.
root@cmsdb01:/var/log# grep innodb_log_file_size /etc/mysql/my.cnf
innodb_log_file_size            = 512M
root@cmsdb01:/var/log# ls -lh /var/lib/mysql/ib_logfile*
-rw-rw---- 1 mysql mysql 5.0M 2006-12-19 18:39 /var/lib/mysql/ib_logfile0
-rw-rw---- 1 mysql mysql 5.0M 2006-12-19 18:39 /var/lib/mysql/ib_logfile1
Rumour has it that you can just stop MySQL, delete these log files and start MySQL again. I'm yet to try this as the server in question is in production use. The alternative is to change the innodb_log_file_size setting to match the file. So in summary the problems with MySQL are:
  • Not logging warnings anywhere useful.
  • Converting engine types with a warning rather than throwing an error. This can be fixed by setting sql_mode to include NO_ENGINE_SUBSTITUTION.
  • Starting up and disabling InnoDB when there is a problem rather than failing to start, giving a false impression that everything is working.
MySQL has not impressed me this week.
Read Comments (3)

2 April 2008

David Welton: Google Maps' terrain view now has contour lines

Amongst other things, I have long had a passion for maps. I remember what a nice discovery USGS topo maps were as a kid, and as I developed an interest in cycling, all the cool places I've found over the years by poring over maps. There's also an element of discovery in getting detailed information about a place you've never been. So I'm excited to notice that Google Maps' terrain view now has contour lines to show elevations, rather than just shading. Here's the mountain north of Innsbruck, for example. Or Oregon's North and Middle Sisters Steens Mountain, Little Blitzen and Big Indian gorges Cool!

31 March 2008

Steve Kemp: I don't shut up, I grow up

Recently I took stock of my Javascript programming efforts, with the intention of simplifying things. To date I've put together three sites which make use of Javascript:
debian-administration.org
There isn't very much javascript in use upon this site, and you might be forgiven for thinking there was none at all if you don't have an account, or don't login. There's a tagging system which is starting to creak under the sheer number of different tags, and several back-end parts of the site make use of AJAX calls. Most of the script lives in a single file common.js which I cobbled via a process of trial and error, augmented with a little copy & paste coding. It works. But I knew I could do better ..
ctrl-alt-date
This was my first attempt to make a site be truely dynamic and "pretty". It has succeeded in that respect, although the lack of members makes the site itself essentially a failure. This time round I decided that I'd utterly refuse to write my own Javascript. So instead I used script.aculo.us (damn I hate those ugly URLS). This library made it almost too easy to add flash. I liked it a lot. Having said that though the sheer scope of the library and the way it didn't fit in the way that I coded made it painful to use at times. It works, and it works well. Like it? Yes. Love it no?
mail-scanning.com
This site has a fair amount of javascript upon it, but that isn't obvious unless you're actually a user. The only piece that I can recall is the real-time count of spam caught upon the front-page. (Realtime stats are cool!) Most of the code here is the simple kind, reverting back to the way I worked on the Debian Administration site; we're talking about basic effects such as:
  • show/hide a div
  • make an AJAX request every now and again.
  • Do a bit of auto-completion.
To get more of a feel for whats out there I wrote this initially with my own code, then later migrated it to jQuery. Quite frankly jQuery rocks. The way it works is a little strange at first, but it is so natural after a while. As an example:
// find the div called "foo" - hide it.
$("#foo").hide()
I'm liking this library a lot recently, but only time will tell if I use it more.
In conclusion I filed #473125: ITP jQuery failing to see the existing ITP already present. Once the package makes it through the NEW queue I will update it to follow the skeletal javascript-policy. I disagree about the naming scheme suggested by that policy primarily because we already have packages of several of the "big" javascript libraries, such as scriptalicious, and see no gain in renaming them. But otherwise the policy is sane enough; following it will cause no harm at the very least. ObQuote: Stand By Me

Lars Wirzenius: Progress bars everywhere!

As I get older, and more impatient, I find myself wanting every program to give more or less accurate estimates of the remaining time until it has completed whatever it is doing. This applies to command line utilities as well. In fact, since most of my time is spent on the command line, it applies particularly to command line utilities. The thing I, as a user, want to know is how long I will still have to wait. I don't particularly care whether the program has completed 97.12765% of its job, I want to know whether it will still take seconds, minutes, hours, or longer to complete. Not remaining file counts. Not remaining byte counts. Time. I'm interested in the entire task. If each file takes 0.1 seconds to process, that's fine, but I don't want to estimate myself that with fourteen thousand remaining files it's going to to take around 25 minutes more. It doesn't have to be completely accurate; a reasonably good estimate is fine. It can fluctuate: when downloading over the Internet, transfer speeds sometimes vary. That's fine. Just give me the current estimate using the average recent speed. For various reasons, I often deal with large files, in the multi-gigabyte range. Copying them with cp seems to be pretty effective, but it's annoying to have to set up something like "watch du -sh source.file target.file" to be able to track progress. I could use "rsync --progress", but that only works well for a single file. Otherwise it reports remaining time for each file separately. I realize that rsync has a particularly difficult job for estimating the remaining time, since it is dependent on the contents of the files at either end. But it could try. Because of these, I tend to like Nautilus for copying files. It has a pretty good indicator for remaining time. Unfortunately, that isn't always practical to use, either, since I don't always copy file so that the local host is at least one end of the transfer, and I often really want rsync. One of the nice lesser-known utilities is pv (in Debian and Ubuntu in the package of the same name). It allows one to construct pipelines that estimate current remaining time. For example, I use this to fill a filesystem's empty space with zeroes:
pv -petrb -s "$ guess k" /dev/zero > temp
(where guess is the free space in kilobytes). Sometimes the pipelines need to be somewhat contrived to fit pv in. This can also make them slower than straightforward ones would be. But it's often nicer to get some progress reporting than to optimize for speed, at least for stuff that you have to actually wait for in real time.

30 March 2008

Francois Marier: Debugging concurrency problems: Forcing a process to run on a single CPU

Multi-threading can cause hard-to-diagnose problems and they are especially visible on multi-core CPUs (or multi-CPU systems) where threads actually run concurrently.

Here's a quick way to find out if the segfaults you are experiencing might be due to some concurrency/locking problem: force the application to run on a single CPU using schedtool (part of the schedtool package on Debian/Ubuntu).
schedtool -a 0 -e applicationname
This is just one example of what you can do with the CPU Affinity controls on Linux.

29 March 2008

Mart&iacute;n Ferrari: Six (more) reasons to hate Java

There are many reasons to hate Java, it's like shooting fish in a barrel. But in the last weeks I found some more, half of them aren't even fault of the standard API (the usual culprit), and I want to share my frustration with the rest of the world. It has no unsigned types Say you want to manage binary protocols, files, etc. You'd like to store and retrieve bytes and words in machine format, of course, but you'll have a hard time finding the right value because you can't have a goddamn unsigned byte! So if you want —for example— to read the length of a string which can be bigger than 127 as a single byte, you'll end doing some aberration like this:
byte strl = foo_retrieve_byte();
int real_strl = strl < 0 ? 256 + strl : strl;
You can't inherit constructors Maybe there's something in the OOP dogma that mandates this, but if I want to define a new exception that does nothing, but allows me to differentiate a custom class of errors, I don't want to explicitly write dummy constructors instead of writing an empty class —which is simpler and cleaner— and have the four standard exception constructors for free. So, you'll have to do this:
public class FooException extends BarException  
	public FooException()   super();  
	public FooException(String s)   super(s);  
	public FooException(Throwable t)   super(t);  
	public FooException(String s, Throwable t)   super(s, t);  
 
Instead of this:
public class FooException extends BarException    
You can't declare destructors Say you've coded a nifty jdbc wrapper that automates most of the boring stuff, like automatically freeing the connection to the connection pool when there was an exception thrown, logs errors and maybe queries, handles auto-commit settings, etc. In the simple case, you'd create an instance, do some work, and then close the connection; if there was an SQL exception, you little class will take care of it and re-throw. BUT, if in the middle of your work in the calling code, you have an unrelated exception, you need to catch it or you'll leak connections, because the wrapper class doesn't have a way of handling this, even if the object gets garbage collected! In the end, I had to write tons of times the same stupid code:
try  
	j = JdbcWrapper.new(foo);
	j.doSelect(bar);
	unrelatedObject.unrelatedMethod();
	j.doUpdate(baz);
  finally  
	j.close();
 
You don't have a sane way of controlling the terminal Most of the time, when dealing with web applications, you won't need this. But when you're doing some serious stuff, you might need to write a CLI application (oh, noes!); and then do some stupid thing like asking the console to stop echoing characters so you can ask for a password. You'd think that the smart guys that designed Java would have devised an portable library to handle this trivial task, but no! Guess what I found when looking at a way of doing this? Some idiot at SUN recommending (and many dummies following) a background thread that spits backspaces faster than the user can write! Of course, SUN will tell you that using Strings is insecure for storing passwords, because they cannot be wiped out like byte arrays. But they fail to mention something as basic as not showing the fucking password in the first place, or something as important as wiping the object after using it: you can't lock memory from Java, so you risk that it could end in some slack in the swap file. No, you can't leave stdin/stdout/stderr fscking alone! When dealing with the previous problem, you say fuck portability and just call stty -echo to do the job. Well, you'll find a little surprise! Even if you don't ask for the process' reader/writer streams, the Runtime class will pipe(2) and dup(2) your child process' standard streams. With this excellent approach you cannot have a child process output stuff to the user, nor even errors; you cannot allow the child process access the tty (stty obviously can't do shit with a piped stdin). And there's no fucking way to tell it to stop crippling your standard streams! The only solution I could find was a very ugly shell hack:
Process p = Runtime.getRuntime().exec(
	new String[] "sh", "-c", "stty echo > /dev/tty < /dev/tty 2>&1" 
);
No, you can't handle signals After doing the previous hack, you find that if you interrupt your process while waiting for user input, your console will remain in noecho state. So the obvious thing is to catch SIGINT and friends and call stty again to restore the console. HA! You will find that there's no standard way of doing that, and if you really need it, you have to resort to undocumented com.sun.* classes which aren't even guaranteed to be present in every JVM. My solution was to set up a shutdown hook, which solved this problem, but would not solve other very common problems that are solved usually with signal handlers. I'm going to DebConf8, edition 2008 of the annual Debian 
       	developers meeting Tags: Planet Lugfi, Planet Debian

David Pashley: Bad Decompression Errors in OpenSSL 0.9.8a

Recently, we rolled out a Shibboleth Single Sign On service to protect one of our services. However, we started recieving intermittant login failures, both on our automated monitoring and from customers. Curiously these failures tended to happen mostly in the evening, which isn't a peak time for us. Debugging showed that the authentication worked, but the authorisaton was failing. Shibboleth works as an apache module and daemom that protects a service, which communicates with a webservice that does the authenication processing. The log files were showing an occasional SSL error in this communcation link.
INFO shibtarget.SessionCache [43005] sessionGet: trying to get new attributes 
      for session (ID=_d0cd2f93840bb92050b28fa73d19ce4f)
INFO SAML.SAMLSOAPHTTPBinding [43005] sessionGet: sending SOAP message to 
      https://login.example.com/shibboleth/AA
ERROR SAML.SAMLSOAPHTTPBinding [43005] sessionGet: failed while contacting   
      SAML responder: error:1408F06B:SSL routines:SSL3_GET_RECORD:bad 
      decompression
ERROR shibtarget.SessionCache [43005] sessionGet: caught SAML exception 
      during SAML attribute query: SOAPHTTPBindingProvider::send() failed 
      while contacting SAML responder: error:1408F06B:SSL 
      routines:SSL3_GET_RECORD:bad decompression
ERROR shibtarget.SessionCache [43005] sessionGet: no response obtained
We didn't manage to find any suitable solutions on the internet, so we pulled out the trusty wireshark and started looking to see what was going on. We could see that the client was advertising deflate and null compression, and that the server was responding by asking for deflate compression. However the client would then claim that there was a decompression error in the servers response. This opened a few lines of enquiry. I made sure that both ends of the connection were running the same version of OpenSSL and they were both using 0.9.8a from Ubuntu Dapper. Interestingly 0.9.8a is the first version that had compression support. We found a couple of suggestions including forcing connections to be SSL2, which lacked compression or recompiling openssl without zlib support. As the former was easier, we tried that first by putting
SSLCipherSuite SSLv2:-LOW:-EXPORT:RC4+RSA
in /etc/apache2/mods-enabled/ssl.conf as suggested by Debian bug #338008, and this seemed to work for around an hour. Packet sniffing showed that it was still negotiating SSL3 including deflate compression. Clearly we had to try something else. Rather than recompile OpenSSL without zlib support, I thought I'd try upgrading the version of OpenSSL to something later in case that fixed the decompression bug. the version in Hardy is 0.9.8g, which sadly required recompiling and disabling the Ubuntu change to enable -Bsymbolic-functions during linking. Installing this on the client end didn't fix the problem, however installing it on the server end seemed to fix it. So far it's been running for 24 hours without an error, so fingers crossed that this has fixed it for good.
Read Comments (0)

Next.

Previous.