Assume for a moment that you have 148 hosts logging, via syslog-ng, to a central host. That host is recording all log entries into an MySQL database. Assume that each of these machines is producing a total of 4698816 lines per day. (Crazy random numbers pulled from thin air; globviously). Now the question: How do you process, read, or pay attention to those logs? Here is what we've done so far:

syslog-ng

All the syslog-ng client machines are logging to a central machine, which inserts the records into a database. This database may be queried using the php-syslog-ng script. Unfortunately this search is relatively slow, and also the user-interface is appallingly bad. Allowing only searches, not a view of most recent logs, auto-refreshing via AJAX etc.

rss feeds

To remedy the slowness, and poor usability of the PHP front-end to the database I wrote a quick hack which produces RSS feeds via queries, against that same database, accessed via URIs such as:

• http://example.com/feeds/DriveReady
• http://example.com/feeds/host/host1

The first query returns and RSS feed of log entries containing the given term. The second shows all recent entries from the machine host1. That works nicely for a fixed set of patterns, but the problem with this approach, and that of php-syslog-ng in general, is that it will only show you things that you look for - it won't volunteer trends, patterns, or news.

The fundamental problem is a lack of notion in either system of "recent messages worth reading" (on a global or per-machine basis). To put that into perspective given a logfile from one host containing, say, 3740 lines there are only approximately 814 unique lines if you ignore the date + timestamp. Reducing logentries by that amount (78% decrease) is a significant saving, but even so you wouldn't want to read 22% of our original 4698816 lines of logs as that is still over a million log-entries. I guess we could trim the results down further via a pipe through logcheck or similar, but I can't help thinking that still isn't going to give us enough interesting things to view. To reiterate I would like to see:

• per-machine anomolies.
• global anomolies.

To that end I've been working on something, but I'm not too sure yet if it will go anywhere... In brief you take the logfiles and tokenize, then you record the token frequencies as groups within a given host's prior records. Unique pairings == logs you want to see. (i.e. token frequency analysis on things like "<auth.info> yuling.example.com sshd[28427]: Did not receive identification string from 1.3.3.4" What do other people do? There must be a huge market for this? Even amongst people who don't have more than 20 machines!

Ok, cool! Time to post again: Like these mega interesting random TOP 10 lists people prepare every December and January, some love to speculate if the current year will be or not the year of Linux on desktop.

Please permit me state the obvious and argue that there will be no year of Linux on desktop, like there was never a year for Microsoft Windows or one being prepared for Apple Mac OS X! It is a evolving process, as dumb as it sounds. In 20, 30 years for historic reasons, some will call the 90s the decade of Linux rise and I agree. I believe that some history books (books?) will not omit the Linux impact (including of course some other key open source software) on Internet availability for the masses.

Do people really believe that "the year of the linux" will be the one when it reach a bigger domestic user base than Windows? Great, what about Linux usage through Internet? It's Linux on desktop and on steroids including - get ready - portability and tons of buzzwords I can't even keep up with. Don't get me wrong, I care about Linux usage on the real hardware that end users are using at home and not only through the Internet but come on, let's stop running in circles.

Ola Bini, one of the JRuby hackers, and a very bright guy, posits that there "won't be a next big language": http://ola-bini.blogspot.com/2008/01/language-explorations.html

There might be some that are more popular than others, but the way development will happen will be much more divided into using different languages in the same project, where the different languages are suited for different things. This is the whole Polyglot idea.

I'm dubious, and wonder what he would consider to be the underlying sociological and economic factors driving this change. Programming languages are, in the end, about people in all their weirdness, so to understand where languages are going to go, you have to consider those human factors, as I've attempted to do here. One trend that points to a slow proliferation of languages is of course the lock-in cited in my article. Today's big languages (Java, and on the web, PHP) won't just go away from one day to the next, just as C, Cobol and Fortran have not disappeared with the advent of Java. That process will continue, making it likely that new languages will carve out new territory for themselves rather than exclusively cannibalizing existing installations from older languages. This naturally leads (slowly) to more languages, even if the next generation has a Next Big Language. And why shouldn't there be one? Ola talks about a situation where various languages run and interact on top of a runtime (JVM). Isn't that similar to what we've had with C, though? Perl, Tcl, Python, etc... all run on top of C. Sure, the JVM is a step up from that in some ways, most notably GC, being a bit more consistently cross-platform, and having a wider array of libraries, but in the end, it still comes down to the network effects of being able to read and write a common language, whatever it runs on. Obviously, the network externalities of programming languages are not so strong that they hit a tipping point after which one language crushes all the others, but they are strong enough to consolidate leadership in one or a few languages. Programmatically, Jython, JRuby, (and Hecl?:-) may even find it easier to interact on the same platform, but the humans writing the code will still push for consistency and the minimum set of common tools in order to aid the sharing and review of code. Another way to look at it might be from an organizational point of view. Today's biggest fish in the pond, Google, only allows four languages for their production systems. What would a "no big language future" organization look like? I can't believe they'd welcome a big hodge podge of things. In conclusion, as computers are ever more widely used, it's certain that more languages will be utilized. However, it's also likely that from time to time a few languages, with one in the lead, will emerge as the leaders.

happy newS year So this was Christmas, new year, et al. Too bad Murphy struck again on the first day of my (theoretical) vacation, delivering me a nice flu (not yet completely absorbed). Nevertheless I managed to do something funny in the last few days:

• I remembered that I do have, as all other DDs, a personal lintian page and I have thus spent some time fixing warning ignored since ever (almost all uploads are still in delayed queues however); while I was at it I also decided to "destroy" my lintian page moving my name to Uploaders for packages that still weren't co-maintained. (/me eager to see this implemented, go Russ, go!)
• I have fixed the vcs field stats page to consider as equal Vcs-Svn fields pointing to URLs like svn://svn.debian.org/foo and svn://svn.debian.org/svn/foo which are indeed the same final path on alioth. That's why in the graph you now see an inversion in the relative positions of field with wrong value and correct field, in case you were wondering. Thanks to Sandro Tosi for the hint.
• I have finally managed to leave the 18th century my mail set-up was living in: goodbye pop + fetchmail + mutt, welcome imap4 + offlineimap + mutt. In the former setup I was unable to read my old mails while away from my laptop, now I have decent replicas of my full mailboxes in various places. I'm still a bit unhappy about how can I force a send/receive email cycle: with fetchmail I used to have a demonized fetchmail running in the background and just executing fetchmail I was able to force that. Now, with offineimap, I've an autorefresh-ed offlineimap running inside screen and I need to bring up the screen and then hit 1 to trigger a sync cycle.

... now back to that paper introduction due for December 26th

I was watching 24 the other day and (without giving anything away) Edgar Styles had to "patch a system call" to save the world. After chiding his assistant for giving him the wrong opcode for a "load A, conditional" the following briefly appears on the screen. It's actually a pretty good piece of technobabble. The register names and use of 0x80 suggest whoever wrote this has at least some familiarlity with x86 assembley. It would have been slightly better if they called the "microshell" a "microkernel" and put the system call in eax before calling the interrupt, but if you belive Jack's amazing ability to get cell phone reception then it's only a small issue!

I was watching 24 the other day and (without giving anything away) Edgar Stiles had to "patch a system call" to save the world. After chiding his assistant for giving him the wrong opcode for a "load A, conditional" the following briefly appears on the screen. It's actually a pretty good piece of technobabble. The register names and use of 0x80 suggest whoever wrote this has at least some familiarity with x86 assembly. It would have been slightly better if they called the "microshell" a "microkernel" and put the system call in eax before calling the interrupt, but if you believe Jack's amazing ability to get cell phone reception then it's only a small issue!

Actually, they were out a few weeks ago, but I've been on vacation at the in-laws' near Monselice, Italy. The release announcement is here: http://www.tcl.tk/software/tcltk/8.5.tml It's worth writing about, because it's been a long time in the making, and contains a number of improvements. Particularly noteworthy in my opinion are 'real' hash tables (dicts), an even better sandbox model (Tcl and Java have the best sandboxes that I know of), and a theme engine for Tk that, in the right hands, significantly improves how it looks. In the fickle world of language fads, Tcl doesn't get the respect that it should these days. Like anything, it has problems, but what doesn't, when examined up close? It's ironic that for years it has gotten things right like Unicode support, and OS-level threading, that are currently being worked on in Ruby. Hecl obviously owes a great deal to Tcl, but since it's a new language (more or less necessitated by cell phone limitations), we're free to take a stab at improving things that Tcl doesn't get right. In any case, Tcl is worth a look if you want to play with something that's fundamentally different from a lot of popular languages, and the core team has put a lot of work into the latest release, so give it a whirl (perhaps using tkcon instead of the readline-less tclsh executable) with an open mind, and see if you don't learn something new.

Glen Turner writes about how Internet censorship could hurt science [1]. The ABC has an article about what is planned [2] which includes “Senator Conroy says it will be mandatory for all internet service providers to provide clean feeds, or ISP filtering, to houses and schools“. If Senator Conroy sticks to that plan then section two of Glen’s post (concerning high-speed access to research data) will not be a problem. This is no criticism of Glen for mentioning the issue as some idiot might try to change the plan to filter corporate and university access. Senator Conroy also earns a Godwin point for “If people equate freedom of speech with watching child pornography, then the Rudd-Labor Government is going to disagree“. There is a good discussion paper by Electronic Frontiers Australia about Labour’s plans in this regard before they won the election [3]. One issue they raise is the small number of sites identified as having child-porn. Imposing filters on an entire country to block the 3,236 web pages identified as prohibited by the Australian Communications and Media Authority is not a sensible solution. Another issue that EFA raises is the variety of reasons for content getting an R18+ rating (if that is to be a filtering criteria). One notable issue is “Adult themes” which includes “issues such as suicide, crime, corruption, marital problems, emotional trauma, drug and alcohol dependency, death and serious illness, racism, religious issues“. I doubt that there is much agreement between parents as to the relative significance of those “Adult themes” and which ones their children should be protected from. I also expect that a significant number of parents would like to have information about safe sex (including safe gay sex) and safe drug use available in case their children are interested in such things. I hope that the number of parents who disapprove of homosexuality and drug use so strongly that they are willing to risk their children’s lives is quite small (although the response to the cervical cancer vaccine indicates that it’s sadly larger than expected). Glen has some good points about this in the first section of his post.

• [1] http://www.gdt.id.au/~gdt/blog/strategy/2008-01-02.1024px


• [2] http://www.abc.net.au/news/stories/2007/12/31/2129471.htm


• [3] http://www.efa.org.au/Issues/Censor/mandatoryblocking.html

Share This

I'm just wadeing through Debian-Mailinglists on the search for enhancements and false-positives.

and i just came over debian-security-announce, a lists that sends a helpful message back to submitters that aren't allowed to send out advisories.

That message didn't contain a Precedence-Header so i added one, and wondered which value would be appropriate... I remember to have seen three values: 'junk', 'bulk' and 'list' (the latter should be set for all our Mailinglists).

So i tried the usual ways to find out about some more possible values, but i couldn't find a RfC or another Document that describes correct usage of the Precedence-Header.

So, i set the Header to 'junk' now, but if someone could point me to some documentation i would be thankful.

Update: formorer pointed me to RFC 3834 - Recommendations for Automatic Responses to Electronic Mail which contains all of the things i needed to know. Thank you.

For a while I’ve been reading the Lenovo blog Inside The Box [1], even though I plan to keep my current laptop for a while [2] (and therefore not buy another Thinkpad for a few years) I am interested in the technology for it’s own sake and read the blog. A recent post concerns a new desktop machine billed as “our greenest desktop ever” [3]. The post has some interesting information on recycling plastic etc, and the fact that the machine in question is physically small (a volume of 4.5L and no PCI expansion slots) means that less petro-chemicals are used in manufacture (and some of the resins used are recycled). However the electricity use is 47W when idle!!! On my documents blog I have a post about the power use of computers I own(ed) [4] which includes my current Thinkpad (idles at 23W) and an IBM P3 desktop system which idles at 38W. Both machines in question were manufactured before Lenovo bought Thinkpad and IBM’s desktop PC business (so they technically aren’t Lenovo machines) and they weren’t manufactured with recycled resins. But the claim that the new machine is the greenest ever is at best misguided and could be regarded as deceptive. I think that the machine is quite decent, but it’s obvious that they can do a lot better. There’s no reason that a low-power desktop machine (which uses some laptop technology) should take more than twice the power of what was a high-end laptop a few years ago. Also comparing power use with P3 machines (which are still quite useful now, my IBM P3 desktop runs 24*7 as a server) is quite relevant - and we should keep in mind that before the Pentium was released no system which an individual could afford had anything other than a simple heat-sink to cool it’s CPU. This is largely a failing of Intel and AMD to make power efficient CPUs and chipsets. It’s also unfortunate that asymmetric multi-processing has not been implemented in recent times. A system with a 64bit CPU core of P3 performance as well as some Opteron class cores that could be suspended independently would be very good for power use with correct OS support. For example when reading documents and email my system will spend most of it’s time idling (apart from when I use Firefox which is a CPU hog) and the CPU use will be minimal for scrolling - a P3 performing core would be more than adequate for that task (which comprises a significant portion of my computer use). Then when I launch a CPU intensive task (composing a blog post in Wordpress or compiling) the more powerful CPU cores could start. It would be good if Intel would release a Pentium-M CPU (32bit) with the latest technology (smaller tracks on the silicon means less power use as well as higher clock speeds). A Pentium-M running at 2GHz produced with the latest Intel fabrication technology would probably use significantly less power than the 1.7GHz Pentium-M that is in my Thinkpad. Put that in a desktop machine and you would have all the compute power you need for most tasks other than playing games and running vista and you could get an idle power less than 23W. The new Lenovo machine in question does sound like a nice machine, I wouldn’t mind having one for testing and running demos. But the claims made about it seem poorly justified if you know the history.

• [1] http://lenovoblogs.com/insidethebox/?p=99


• [2] http://etbe.coker.com.au/2007/10/09/a-long-laptop-lifetime/


• [3] http://lenovoblogs.com/insidethebox/?p=99


• [4] http://doc.coker.com.au/environment/computer-power-use/

Share This

The last day of each year sees one of the strangest traditions I've ever noticed. A 40-year-old English-language film of an 80-year-old music hall comedy is played repeatedly on various stations in Germany - and in Austria, Switzerland, Denmark and Australia too. There's not only the film, but edited versions, localised versions, spoofs and other shows based around it too. The film still isn't shown on any English stations, as far as I can tell. I'd never heard of it before getting satellite TV. So, to see what the fuss is about, grab your Dinner for One times (may contain annoying ads if you switch Javascript on) and sit down with the satellite TV (best), or enjoy this description and script from analoguesat, a German-language description or this BBC report which includes this snippet:-

"To the end of his life, Freddie Frinton heartily disliked Germany and the Germans, thanks to his own wartime experience. He refused to allow a German-language version to be made. That is what has led to the extraordinary fact that today the Germans as a nation have embraced a trifling one-act play in English as their all-time favourite entertainment."

Anyway, the last joke is on we English. It's much better than the aging boy bands on our TV at New Year, isn't it? (Updated 2008-01-03 with 1 comment)

I just came back from Taiwan(went there basically for TWDebCamp2007) tonight. I met lots of great people down there and had lots of fun. I'll write about it later. Theoretically speaking, I was supposed to have decent Internet connections everywhere, but in most cases it didn't work or I was away from it anyway(and we had busy schedule!). So now I'm kinda catching up things quickly, especially what happend on the Net lately. Anywise, it would be a good time to look back 2007 and foresee 2008. Personally, these things were significant for me in 2007:

Finished GPLv3

GPLv3 is finally out, and I'm proud I took part in the revision process. Also I could manage to finish my Japanese translation of GPLv3/LGPLv3. Now my translations are being reviewed by lawyers and other legal professionals, so it will be more sophisticated soon.

Establishing MIAU

As I wrote before, we founded a little organization called MIAU, in hope of becoming a Japanese version of EFF. At the copyright war front in Japan, we are partly losing the battle since the Japanese government essentially ignored our 1800+ unique public comments and seems dare to go the wrong direction somehow. However, we are not going to give up easily -- next year, we will sophisticate our tactics and try to get more people involved. Chris Salzberg wrote up a series of great articles on these or related issues in English, and also submitted his story to Slashdot.

Joined Wikimania 2007

This was also a great fun in Taiwan. I gave a talk at there and also met many great people.

Released GNU a2ps 4.14

Well, this is far less significant, but the new release is now out. I hacked up this release in my hotel room just before I left, so may contain stupid bugs. anyway, as usual, enjoy.

So, now we seem to be in 2008 now. Happy New Year or whatever. I'm gonna do the following this year, at least:

Finish my PhD thesis

This will be the highest priority, of course. I'm planning to make it be a comprehensive study on FLOSS from management/organizational theorists' view. Will contain some math. Maybe not.

Finish my book

I'm still writing a book on Open Source licensing. This was supposed to be done within 2007, so it's long overdue now.

Finish translations

I'm still translating Peter Seibel's great book Practical Common Lisp into Japanese. Also, I guess I should update my Japanese translations of www.gnu.org.

Reduce weight

I'm obviously overweight. I'm gonna go to swim, cycling, or even jogging, and do my best not to eat too much. I'm serious!

Anyway, again I'd say Happy New Year to all, and enjoy your life.

Tag this post with del.icio.us!

You may remember that, short before the release, Aj had a HTML page for packages in NEW, living in his ~/ajt/new/ on our ftp-master host. It was based on a patch from Thomas Viehmann and got lost with the ries crash. Today I reimplemented it, putting it directly into dak and the ftp-master codebase and webtree. My work is also based on Thomas patch, but I reworked the whole thing a bit. I’ve written a new dak command, show-new, which is based on process-new, the command I always use when I process NEW packages. Obviously not with all that functionality, it doesn’t need any reject capability, but it still has some nice parts. Like

• it only runs on packages and their versions if no html output had been generated in the past
• Automatically removes html files of packages no longer in NEW
• It runs right after generating the NEW overview page, hourly. So it should be similar up-to-date (or outdated) as that page is.

To not duplicate code too much this also meant that various code blocks had to be moved into daklib/ files. If you want to look at it - the NEW Overview, something that I wrote long ago, has links to the per-package-version files. Just click on the version number. I have plans to enhance it even more, like having the lintian output use colors too. To make that easy I filed a bug against lintian asking for it. For the other changes - well. I think the layout can be enhanced, possibly going a bit in the direction of the removals summary page, but that will take some time, as I need to talk to my CSS-masters for it. And while I was changing dak code I did some more changes to it. One patchset already got applied from James, removing the obsolete non-US support. I have a large number of other changes, but have to redo them, much more using bzr branches, so James can more easily pick single changes out of them. The changelog for those applied changes is pretty impressive with its length, I think…

2007-12-30  Joerg Jaspert  joerg@debian.org>
        * dak/dak.py (init): add show-new. This is based on a patch
        submitted by Thomas Viehmann in Bug #408318, but large parts of
        handling it are rewritten and show-new is done by me.
        * dak/queue_report.py (table_row): Add link to generated html page
        for NEW package.
        * dak/show_new.py: new file, generates html overview for NEW
        packages, similar to what we see with examine-package.
        * config/debian/cron.hourly: Add show-new call
        * config/debian/dak.conf: Add HTMLPath for Show-New
        * dak/examine_package.py (print_copyright): ignore stderr when
        finding copyright file.
        (main): add html option
        (html_escape): new function
        (escape_if_needed): ditto
        (headline): ditto
        (colour_output): ditto
        (print_escaped_text): ditto
        (print_formatted_text): ditto
        - use those functions everywhere where we generate output, as they
        easily know if we want html or not and just DTRT
        (do_lintian): new function
        (check_deb): use it
        (output_deb_info): Use print_escaped_text, not print_formatted_text.
        Also import daklib.queue, determine_new now lives there
        Also add a variable to see if we want html output. Default is
        disabled, show_new enables it for its use.
        Most of html, besides header/footer are in examine_package instead
        of show_new, as it makes it a whole lot easier to deal with it at
        the point the info is generated.
        * dak/process_new.py (determine_new): Moved out of here.
        (check_valid): Moved out of here.
        (get_type): Moved out of here.
        * daklib/queue.py (determine_new): Moved here.
        (check_valid): Moved here.
        (get_type): Moved here.
        * dak/init_db.py (do_section): Remove non-US code
        * dak/make_overrides.py (main): ditto
        * dak/process_new.py (determine_new): ditto
        * daklib/queue.py (Upload.in_override_p),
        (Upload.check_override): ditto
        * daklib/utils.py (extract_component_from_section):,
        (poolify): ditto
        * dak/import_archive.py (update_section): ditto
        * dak/symlink_dists.py (fix_component_section): ditto
        * scripts/debian/mkmaintainers: ditto
        * scripts/debian/update-mirrorlists (masterlist): ditto
        * config/debian-non-US/*: Remove subdir
        * scripts/debian/update-readmenonus: Removed.

In the recent past I did some more dak changes. Just to list them here, they are

• Fixing daks handling of keys that have expired subkeys together with non-expired subkeys (or non-expired “master” key). That created silent rejects (only sent to ftpmaster, not to the maintainer) in the past. Now it works, iff the signature is valid. (We check other reject reasons like unknown or revoked key, badsig and some other things that can go wrong before that).
• Internal for the NEW queue I adjusted the lintian call to also show what lintian warnings/errors maintainers try to override - and to have its display in color. lintian now also runs on the .dsc file, not only on the individual changes, which also helps me to spot some more errors in NEW which I previously may have missed
• The script to look at the packages in NEW also had a bug that disallowed to look at copyright files from packages using bzip2 compression in their binary tarballs. It did use some ar p tar xz pipeline, which I changed to using dpkg-deb -x and —fsys-tarfile.
• I removed a check for a dpkg pre-depends for bzip2 using binaries, as the dpkg version from sarge knows how to deal with them.

(Most patches applied by James. Can’t remember if someone else applied one. If so that would be Aj). And - at the end of all this - I will finally update my dak package again. Im not sure I get it uploaded this year, as I’m away tomorrow, but it will get an update fairly soon! Ha.

Whilst I'm very pleased with my new segmented network setup, and the new machine, I'm extremely annoyed that I cannot get a couple of (graphical) Xen guest desktop guests up and running. The initial idea was that I would setup a 64-bit installation of Etch and then communicate with it via VNC - xen-tools will do the necessary magic if you create your guest with "--role=gdm". Unfortunately it doesn't work. When vncserver attempts to start upon an AMD64 host it dies with a segfault - meaning that I cannot create a scratch desktop environment to play with. All of this works perfectly with a 32-bit guest, and that actually is pretty neat. It lets me create a fully virtualised, restorable, environment for working with flash/java/etc. The bug was filed over three years ago as #276948, but there doesn't appear to be a solution. Also, only on the amd64 guest, I'm seeing errors when I try to start X which mention things like "no such file or directory /dev/tty0". I've no idea whats going on there - though it could be a vt (virtual terminal) thing?. The upshot of all this is that I currenly have fewer guests than I was expecting:

skx@gold:~/blog/data$ xm list
Name                                      ID Mem(MiB) VCPUs State   Time(s)
Domain-0                                   0     3114     2 r-----   1180.6
cfmaster.services.xen                      1      256     1 -b----      1.0
etch32.desktop.xen                         2      256     1 -b----      1.4
etch32.security-build.xen                  3      128     1 -b----      1.4
etch64.security-build.xen                  4      128     1 -b----      1.4
sarge32.security-build.xen                 5      128     1 -b----      1.0

Following my post about SSH jumphost I have another SSH feature to share with you, which IMO isn’t known or used enough. Imagine you have some data on one host which gets regenerated every now and then and you also need this data on other hosts too. There are multiple ways to sync the data to other machines. You can

• make them available via http/ftp and cron a wget run
• use some rsync or ftp daemon on the target host(s) and write data to them (brrrrr)
• simply use limited SSH keys to signal the target host(s) when there is something new to fetch

The problem of the first option is that you need to make the data available via http/ftp, which needs an extra daemon to run. And then you need to cron a script fetching it, which wastes resources if the data is updated infrequently. The problem of the second option is that you need some daemon available which allows you to write to the host. Depending on what you chose that might be with passwords transferred unencrypted. Or you aren’t able to set it up yourself, as you are not the admin of the machine. So, lets take the third solution. Its safe to assume that SSH is available on the hosts you login to (don’t tell me you still use telnet/rlogin on remote machines), so lets use SSH features. I assume that usage of basic SSH public key authentication is known, if not you first want to google for it and learn about that. Using SSH keys to trigger actions on remote hosts is basically the same as a login to the machine and manually running a command - except you don’t need to do it yourself, a script is doing it. Which leads to one big difference for a “trigger” key compared to a “login” key: You do not set a passphrase for that SSH key! As a consequence of that you use such a key only for exactly one purpose, nothing else. Generate a new key for another trigger. Never, ever, use such a passphrase-less SSH key for normal logins to other machines… Having generated a passphrase-less key on host a (let it have IP 1.2.3.4) we copy the .pub file over to the target host b and put it into the authorized_keys file (usually in ~/.ssh/). The difference to normal login keys is the large line of options we put in front of it:

command="rsync --server -vlHogDtprz --delete --delete-after --ignore-errors . /org/backup/something",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,from="1.2.3.4" ssh-key-here

or

command="/var/mirror/ftpsync &",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty,from="1.2.3.4" ssh-key-here

Both commands allow to use the SSH key from host a with its IP 1.2.3.4 and will run the command thats defined here. No matter what command or options are are specified on the commandline on host a, host b will only do the above. There wont be any port, X11 or agent forwarding and due to the forced command there also wont ever open a shell. With the first command we trigger the remote hosts rsync to write the data we send into that path, using the given options. Of course that means we send data using rsync from our host. The SSH connection stays available for the whole time the command runs. The second command instead runs the given binary, a shell script in this case, in background. As we also forbid tty allocation the SSH connection is closed right after the script is started. To avoid DOSing the remote machine such a script should have some kind of lockfile check and abort if a second copy is already running. SSH also allows to use such limited SSH keys for the root user, while disallowing normal root logins: Change PermitRootLogin in sshd_config from no to forced-commands-only and you wont be able to login to the remote host as root but you can trigger actions that should be run as root. If you want to know what else you can do using options specified in the authorized_keys file then read its man-page, it has some details how you can use it to limit port forward requests, use more than just one IP in the from= parameter, … Some users of the above way of automating tasks on remote machines are

• The Debian mirror network. About all tier-1/2 mirrors are pushed using the described technique, making updates to Debian available as soon as possible.
• da-backup, the Debian Admin Backup tool, used by Debian and DebConf to backup their hosts to central backup servers.
• packages.debian.org updates his data way faster than in the past, thanks to this.
• Possibly many many more, its a useful feature.

While looking at outstanding bugs against my Debian packages, I noticed that an RC bug had crept in against Planner. The bug itself isn't new. In fact, Ubuntu had experienced it already and simply decided to build their package without SQL support to resolve the issue. Planner is a fairly well-matured piece of software, with very few bugs remaining. It remains quite usable in everyday use. However, it has fallen behind in terms of keeping its code up-to-date against the latest versions of some optional dependencies. Tonight, I was just sitting here pondering what to do and it occurred to me that I'm simply no longer interested in maintaining the package. I have done a fairly good job of overhauling the build scripts and of upgrading them for the Python transition but, at this stage, I feel that I've accomplished as much as I could do. Besides, I no longer have the time to maintain so many packages, so I have decided to focus on maintaining those I actually care for and Planner isn't one of them. The package already is assigned to Debian's GNOME team, so no need to actually orphan it, since anyone from the team can pick it up. If you're a member of the GNOME team and feel like taking over, perhaps even submitting patches to upstream to close a few more bugs, go right ahead and jump in.

I’ve just read an interesting article in the New Yorker titled “Twilight of the Books” [1]. It started with some depressing statistics about literacy. One comment that I couldn’t figure out concerned “the panic that takes hold of humanists when the decline of reading is discussed“. The decline in literacy concerns me because I want to live in a society where everyone can contribute as equals and where all voters have the possibility of becoming well informed about the issues that affect their vote. If literacy declines far enough then fascism seems to be the inevitable result. I’m not sure whether that makes me a humanist according to the author. Below in the second part of this post (not included in the feed) I include my dictionary definition of the word, #4 is the definition most commonly used today (which is a reasonable match for my beliefs). There is some interesting information regarding basic research on the mental abilities of literate and illiterate people which suggests that people who an read are better at handling abstract concepts and hypothetical situations. The article notes that spending time on the net increases academic scores for children. It also makes the interesting claim that visiting pr0n sites increases grades, I find that difficult to believe. I wonder whether there is a difference between readers and writers in this regard. Does the typical serious net user who averages writing more than 3650 messages a year for fun (in addition to whatever email may need to be written for work) plus some blog posts and writing other web pages have different mental abilities to someone who just reads what others write? Does blogging (and other writing on your choice of topic) change your brain? What about programmers, does spending 8 hours a day writing formal rules for hypothetical mathematical situations (which is essentially what programming is about) change your brain? Are programmers going to have greater abilities for managing abstract concepts and hypothetical situations than people who have not had such experience? NB It will be difficult to determine this as people who have such skills will find programming easier and more fun so therefore a career in programming computers is self-selecting for people with such abilities. Another interesting fact is the results of research into the comprehension of a PowerPoint presentation. People who read it without an audio-visual accompanying it learned more and rated it as more interesting. Based on this my plan is that in future when attending conferences I will try and read papers which are presented BEFORE attending the lectures. Unfortunately most conferences which have papers presented seem based on the idea that you give a lecture about a topic as a teaser for your paper and assume that the audience knows nothing about it (conferences which don’t have papers outnumber the ones that do). Most Linux conferences seem based on the idea that almost no-one reads papers at all, they attend lectures and visit web sites and IRC channels afterwards. NB this comment is based on talking to people who attend conferences and my observations on attending conferences - it is not a criticism of any conferences merely an observation on cultural attitudes. Based on this information it seems reasonable to provide copies of lecture notes to the audience before the lecture. For example for a typical talk that I give to a LUG I could blog the lecture notes before the meeting and email the LUG mailing-list with the URL, people who really want to learn would hear me talk AFTER reading the notes and hopefully get the best of both forms of learning. I had been considering producing some YouTube [2] videos with instructions on how to use SE Linux. But it seems that might not be such a good idea (at least not a good enough idea to justify the amount of effort I would have to spend). The author seems to misunderstand YouTube, it is NOT about making the Internet resemble TV (where millions of mindless drones watch content created for the lowest common denominator in the audience). The AtGoogleTalks on YouTube [3] and www.ted.com are examples of high-quality free content (comparable to the best TV documentaries - the type of content which is cited as increasing the academic results of children) and the culture of creation and discussion must be a good thing. Most TV is bad because it’s one-way communication and the quality of the content is low. YouTube videos are widely discussed in email, blogs, and other written forums. The creation of the best (and often most viewed) videos would surely involve some writing. I wonder whether the benefits attributed to reading can be obtained without reading fiction. The nature of work is changing to require an increasing amount of reading and writing. The most menial work requires reading instruction manuals. Most of the best paid jobs involve a lot of reading and writing. If reading fiction isn’t required then it seems that there is little risk in the immediate future.

• [1] http://www.newyorker.com/arts/critics/atlarge/2007/12/24/071224crat_atlarge_crain?currentPage=all


• [2] http://www.youtube.com


• [3] http://au.youtube.com/profile?user=AtGoogleTalks

(more…)

Share This

Tomorrow, all being well, I should receive a new desktop computer. I ordered it upon Boxing Day so the turnaround is pretty good. Right now I have two desktop machines:

yours.my.flat

This machine is occaisionally used by my partner, but primarily exists to remotely backup my various servers. It runs rsnapshot to do a full rsync of four machines every six hours.

vain.my.flat

This is my day-to-day desktop machine. It is going to be replaced, and this will be rotated into the backup machine. The machine yours will sit in a closet for a couple of months then be donated to somebody local.

As with all the machines I've bought recently the new one is coming from Novatech - (Another company whos website has a suprising hostname.) The interesting parts of the spec are :

• AMD X2 4200 processor
• 2 x 500Gb SATA drives
• 4Gb memory

The biggest difference to me will be the significantly increased disk space which means that I'll be able to remove the external 120Gb USB hard drive I currently house my MP3 collection upon. By retiring vain and moving my current desktop machine to be the "random guest machine" and backup host I'll also have a lot more space for storing backups upon. (Full database backups every six hours; I'm paranoid...) Having 4Gb of memory will also mean that I can run more than 2 or 3 Xen guests at the same time. My plan is to setup at least the following guests:

• sarge.security-build.xen
• etch32.security-build.xen
• etch64.security-build.xen
• etch32.desktop.xen
• etch64.desktop.xen

The desktop machines will run GDM / X11 and will be available for logins via xdmcp - so that I can have a fully seperate working environment which is pristine and easily regenerated. Giving 5 xen instances 256Mb of memory each will leave me with close to 3Gb of disk for the host and that is three times what I hav enow! All in all I'm looking forward to the new machine enormously a bargain at ~ £290.

I just made time to catch up on processing requests for access to LWN.net under the Debian group subscription sponsored by HP. If you think you were waiting for me, and didn't get some sort of email reply articulating what was wrong or saying you were added, please re-send your request. If you're a DD and don't know what this is about, you can find details about LWN and how to get added to the Debian group subscription in this message in the Debian mailing list archives. For what it's worth, there are now 494 DD's subscribed to LWN as part of this group subscription.