Is there an existing system which will allow me to query Apache logfiles via an SQL string? (Without importing into a database first). I've found the perl library SQL::YASL - but that has a couple of omissions which mean it isn't ideal for my task:

• It doesn't understand DISTINCT
• It doesn't understand COUNT
• It doesn't understand SUM

Still it did allow me to write a simple shell which works nicely for simple cases:

SQL>LOAD /home/skx/hg/engaging/logs/access.log;
SQL>select path,size from requests where size > 10000;
path size 
/css/default.css 13813 
/js/prototype.js 71261 
/js/effects.js 37872 
/js/dragdrop.js 30645 
/js/controls.js 28980 
/js/slider.js 10403 
/view/messages 15447 
/view/messages 15447 
/recent/messages 25378 

It does mandate the use of a "WHERE" clause, but that was easily fixed with "WHERE 1=1". If I could just have support for count I could do near realtime interesting things... Then again maybe I should just log directly and not worry about it. I certainly don't want to create my own SQL engine .. it just seems that Perl doesn't have a suitable library already made which is a bit of a shocker!

Thanks for the comments/mails on my previous post. I've now started the migration properly. All code currently held upon cvsrepository.org will be moved to http://repository.steve.org.uk/ - a nice system-agnostic name. So far I've only setup the pages for two project (my dotfiles and ~/bin) but I'm happy with the process, and the naming scheme appears sane to me now. Using sub-domains I need only install one CGI which is a big win, and using a simple set of templates + mod_rewrite all projects get the same look and feel. I've still got to tweak the stylesheet a little, but otherwise I'm happy with things. I can host both public and private repositories, and that all works magically. Of course the huge win is no longer needing to run a pserver for CVS! All future checkouts will be via HTTP, and the rare few people with commit access will be able to do so over SSH. Once things have been migrated I'll cause the cvsrepository names to redirect to the new locations, and keep that up until the domain expires as a transition period. (I think +1 year :)) The other job for the weekend is releasing a fixed security update for Samba, to fix the regressions.... That is in hand, but the buildds are being slow. If they're not all done I'll release it anyway today/tomorrow. I'm annoyed that I handled the timing so badly, but I hope that people don't hate me for it.

This was informally confirmed by someone at Google: Given this layout code:

<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
  android:orientation="vertical"
  android:layout_width="fill_parent"
  android:layout_height="fill_parent"
  id="@+id/ll">
</LinearLayout>

And this Java code:

public void onCreate(Bundle icicle)
 
    super.onCreate(icicle);
    setContentView(R.layout.main);
    LinearLayout ll = (LinearLayout)findViewById(R.id.ll);
    RadioGroup rg = new RadioGroup(this);
    rg.setLayoutParams(new LinearLayout.LayoutParams(
            LinearLayout.LayoutParams.WRAP_CONTENT,
            LinearLayout.LayoutParams.WRAP_CONTENT));
    ll.addView(rg);
    RadioGroup.LayoutParams rlp = new RadioGroup.LayoutParams(
        RadioGroup.LayoutParams.WRAP_CONTENT,
        RadioGroup.LayoutParams.WRAP_CONTENT);
    RadioButton foo = new RadioButton(this);
    foo.setText("foo");
    foo.setLayoutParams(rlp);
    rg.addView(foo);
    RadioButton bar = new RadioButton(this);
    bar.setText("bar");
    bar.setLayoutParams(rlp);
    rg.addView(bar);
    RadioButton baz = new RadioButton(this);
    baz.setText("baz");
    baz.setLayoutParams(rlp);
    rg.addView(baz);
 

The resulting radio buttons fail to be mutually exclusive. This is a bit frustrating, as Hecl doesn't utilize the XML layout system. Or rather, you can use that if you want, or you can create everything in Hecl. The individual at Google says the problem will be fixed in a future SDK release, so hopefully things will work in the near future. I also hope it's open sourced soonish, so that problems like this can be fixed without requiring the intervention of "the hand of Google". Of course, in this market, there's also the issue of things being shipped on hardware, where open source or not doesn't matter much if the bug is out there, because you simply can't fix it on millions of devices. Hopefully the product that ships will be very solid.

After recently intending to drop the Planet Debian search and recieving complaints that it was/is still useful it looks like there is a good solution. The code will be made live and official upon the planet debian in the near future. The DSA team promptly installed the SQLite3 package for me, and I've ported the code to work with it. Once Apache us updated to allow me to execute CGI scripts it'll be moved over, and I'll export the current data to the new database. In other news I'm going to file an ITP bug against asql as I find myself using it more and more...

My tool to query apache logfiles via SQL seems suprisingly popular. Just as a recap the process goes like this:

• Start the shell.
• A temporary SQLite database is created.
• You load any number of apache logfiles into it.
• Then queries may be executed against those records until you exit.
• The temporary database is dropped.

Now it is possible to save and load the SQLite database, so that you don't need to reparse the apache logs each time, that gives a nice speed increase for non-changing files. By tonight I'll have aliases working for queries so you can bookmark them:

alias refers SELECT distinct(referer) FROM logs

Then in the future the 'refers' command will be available and will run the named query. Neat. Now that I'm comfortable with SQL queries it just seems so natural, easy, and right to query logfiles this way. I guess that makes me strange.

Is there an existing system which will allow me to query Apache logfiles via an SQL string? (Without importing into a database first). I've found the perl library SQL::YASL - but that has a couple of omissions which mean it isn't ideal for my task:

• It doesn't understand DISTINCT
• It doesn't understand COUNT
• It doesn't understand SUM

Still it did allow me to write a simple shell which works nicely for simple cases:

SQL>LOAD /home/skx/hg/engaging/logs/access.log;
SQL>select path,size from requests where size > 10000;
path size 
/css/default.css 13813 
/js/prototype.js 71261 
/js/effects.js 37872 
/js/dragdrop.js 30645 
/js/controls.js 28980 
/js/slider.js 10403 
/view/messages 15447 
/view/messages 15447 
/recent/messages 25378 

It does mandate the use of a "WHERE" clause, but that was easily fixed with "WHERE 1=1". If I could just have support for count I could do near realtime interesting things... Then again maybe I should just log directly and not worry about it. I certainly don't want to create my own SQL engine .. it just seems that Perl doesn't have a suitable library already made which is a bit of a shocker!

Thanks for the comments/mails on my previous post. I've now started the migration properly. All code currently held upon cvsrepository.org will be moved to http://repository.steve.org.uk/ - a nice system-agnostic name. So far I've only setup the pages for two project (my dotfiles and ~/bin) but I'm happy with the process, and the naming scheme appears sane to me now. Using sub-domains I need only install one CGI which is a big win, and using a simple set of templates + mod_rewrite all projects get the same look and feel. I've still got to tweak the stylesheet a little, but otherwise I'm happy with things. I can host both public and private repositories, and that all works magically. Of course the huge win is no longer needing to run a pserver for CVS! All future checkouts will be via HTTP, and the rare few people with commit access will be able to do so over SSH. Once things have been migrated I'll cause the cvsrepository names to redirect to the new locations, and keep that up until the domain expires as a transition period. (I think +1 year :)) The other job for the weekend is releasing a fixed security update for Samba, to fix the regressions.... That is in hand, but the buildds are being slow. If they're not all done I'll release it anyway today/tomorrow. I'm annoyed that I handled the timing so badly, but I hope that people don't hate me for it.

With four plane tickets already in our hands, it's official. Clara, Sabri, Joni and I will be flying to Marrakech as 2007 languishes, spending the first hours of 2008 in the city, and then start our way to the South during the following week. We still haven't settled on a planned route, but travelling with these friends is mostly a synonym of adventure and fun. The desert, the snow, the villages, the mosques, the souks, the spices, the hammams... I can't wait!

A lot has happened after the Summer of Code has ended, and I haven't been able to work on bug-triage after that, in part due to "Real Life", in part due to laziness. Anyway, I've finally got some time to work a bit on it today.

First, I've taken care of Debian's 452918, where the reporter said that using "pending" on bug status meaning open might cause confusion with the pending tag. Actually, pending is the word used internally by debbugs to mean open bugs, and its also what is returned by the debbugs' SOAP interface (bugs tagged pending get the status pending-fixed instead). Anyway, I've just mapped the internal/SOAP debbugs status to the ones used by pkgreport.cgi, that should avoid misunderstandings.

While at it, I also took some time to add a feature I've missed while using bug-triage myself: When browsing upstream bugs in search of a match for a Debian bug, it's necessary to read the upstream bug log; now double-clicking on a upstream bug on bug-triage will open it's log on a browser (the same way that selecting a Debian bug and clicking "Go to" in the toolbar works).

It's only small changes by now, and I'm not sure if I'll be able to dedicate myself much more to this in the (near) future, but I'll try my best.

* Flickr pics says whole story. Foss.in == Fun! I learned a lot while managing Debian/Ubuntu stall. Next year we will have more better stall and representation from community. Debian Tshirts were given to 3 nice people (Thanks - Prasad, Amod and Anivar) who maintain stall during my mysterious gaps. Sorry to Cydork - yeah, size does matter! * Thanks to team at foss.in, You people rocks (always)! * Finally, back /home (as usual, flight was delayed). Kavin is crawling much intelligent way and demands things (he want to chew everything..). So, time to alert!

Some adventures on my way back from FOSS.in at Bangalore. I was planned to fly back through Mumbai, with a quick connection (05:30-07:00). However, the Paris flight got delayed "because of technical problems" and we finally took off at 11:00. However, while flying the deserts of Eastern Iran (so after 2 hours flight), some very bad vibrations were happening and, apparently, one engine of the 747 was shut down by the crew. Then we gently turn 180 degrees and were told that....we were going back to Mumbai..:-( So here am I, currently in Mumbai airport, in the VIP lounge (I didn't want to go to a random hotel with pay-per-use wireless), coping with my mail et keeping in touch with $HOME through an awfully jerky connection. FOSS.in was huge fun but it seems that coming back from it will not be fun at all. Well, that's how air transportation is, right?

Some time ago, I had explained why I thought would never use network-manager, especially because of the gnome-keyring password which should be typed in at every new session. The last evolution-data-server upload introduced a gnome-keyring binding which forced me to inquire again on how to avoid this annoying gnome-keyring password. This led to the libpam-gnome-keyring which allows the keyring to be unlocked automatically at login from the display manager. This is great! I am now using network-manager on my laptop! I am still puzzled about the fact that there is no network connection until someone logs in, but I guess I will live with this, especially with a desktop machine.

Chris Messina wrote an interesting post on his blog about his OpenID Shitlist, Hitlist and Wishlist for 2008. (I'm going to briefly try to remember if using the word "shit" is cause for alarm on Planet Debian, and then I'm going to ignore the problem and move on.) I think it's a good post and it really identifies some of the key players that are helping and failing in the OpenID space. I got a shout-out for Wikitravel and for the MediaWiki OpenID extension, which yay for me. I think Chris was probably a little unfair about Wikimedia's lack of support. The site is moving very slowly towards a unified login for all its wikis, which is a really difficult task to make happen. OpenID is slated for installation some time after that, so it's going to be a while. But I hope that it will be useful for Wikipedians. My main impression from this blog post, however, is that OpenID's day in the sun might be over. That's not a bad thing -- usually the hard work gets done after the cameras are turned off. Zealots like myself tend to think that if only the world knew about our latest-greatest idea or technology, they'd fall right into line and adopt it immediately. But my experience has been that things work a lot slower than that, and I'd guess that OpenID is probably going to have a similar adoption curve. In thinking about opening up the Social Web, it's probably good to compare the opening of the Web Web in the mid-1990s. At the time, most Americans who were "online" were subscribed to a proprietary network like AOL, Compuserve, Delphi or MSN. (I'm not sure how things were in the rest of the world... Similar? Worse?) Many of these services either walled their users from the Internet entirely, or made it a one-way interaction. AOLers could read stuff from the rest of the Web, for example, but others couldn't see what AOL folks were doing. That system looked impenetrable; after all, AOL was paying millions of dollars to make great content and manage forums and the like, and why would they give that away to Internet leeches for free? But eventually it came apart because the Web became more interesting than the on-line services. (I personally think the watershed was the Ate my balls meme, but others might differ. I just kind of picture hundreds of thousands of AOLers calling up technical support asking to see the Mr. T pictures, and that's what finally broke things down.) Today, on-line service "walled gardens" are a negligible part of the on-line world -- those that still exist are more or less big ISPs. I think that the parallel with the Social Web is probably pretty apt. Today, Google, Facebook and Digg are reluctant to open up their identity silos to a distributed social Web. Their main asset today is their collection of identities and the relationships between them -- why give away the crown jewels? My guess is that it's not going to be until a mature, standardized and distributed Social Web exists outside the walls of the big social-networking players that we'll see them really willing to sit down at the table and come to the party. Of course, by that time, it'll be a moot point. But that just means there's a nice big fat market for somebody smart to take advantage of. tags: socialweb openid walledgardens

While I'm still running awesome as my window manager currently and though I do like it, I will switch back to ion on the next login. "Some XRandR support" is not yet enough... it crashes when I add or remove a screen.

Besides that, the only other thing missing is that I have not yet found out how to use

docker

sensibly. Hints welcome.

So overall: looks very promising, but not yet a feature complete replacement for me. But good things will come to those who wait :-)

I’m very surprised about the popularity of an old post of mine, regarding my experiences with CompUSA. It continues to collect horror story comments, the last one coming less than three weeks ago. While any company has its detractors (especially any company dealing directly with the public), it seems odd to me that people continue to be motivated enough to post to my blog, of all places, their tales of woe. For me, life has been very CompUSA-less of late. Indianapolis now has a Fry’s, one of only two east of the Mississippi as of this writing, and for someone in the relatively tech-starved Midwest, it is a godsend. (People from the west coast: please stifle your laughter as best you can.) And evidently enough of these horror stories have been passed around that they felt the need to close over half their stores in February. The Indy store was spared that time, but not for long.

The electronics retailer decided to finish what it had started earlier this year, announcing that it would sell or close the remainder of its stores in the US after the holiday season. The company, controlled by Mexican retail management company Grupo Sanborns since 1999, has been sold to Gordon Brothers Group, a restructuring firm that will be responsible for selling off the remainder of its assets.

In an abstract sense, less competition in the electronic retail business isn’t ever good. But it’s arguable that we’ve never had so much competition in the electronic retail business if you count the Internet stores that have sprung up all over. And I’m certainly happy to see an outfit that will slander people for profit go belly-up.

GCC 4.3 now prints an error message when C++ code contains duplicate function parameter names in function prototypes. You'd think that's a pretty obvious problem, but we still have about 15 packages in the archive that contain such code. Anyway, for code like this:

class foo  
    foo(int w, int w);
 ;

GCC will as of 4.3 show the following error message:

t.cc:2: error: multiple parameters named 'w'

Today, we released the first beta of LSB 3.2. If all goes well, this will hopefully be the only beta. We’ve been working on 3.2 for a while, and we’re really excited about it. We’ve added quite a few interfaces, based on feedback from application vendors and others. There are whole new sections: printing support, Perl and Python, FreeType, Qt 4, and trial use support (our new name for “optional”) for Xrender, Xft, and the ALSA API. Betas can only be as good as the people participating; more feedback means a better standard. So please go check out the beta. Look at the whole thing, or just parts you’re interested in. Read the spec, or check out the tests, or try building your favorite open-source app with our SDK. We’re hoping for a release before Christmas, but that depends on the feedback we get, of course. And we’d rather know about that really big issue we forgot about and delay the beta than find out after the release. So get cracking!

After a lot of work, the DebConf7 Final Report has been released. The report is intended for a large audience, and includes impressions and facts from the conference. Whether you were there or not, we hope you will find the report an interesting read! The report is available as a PDF. Many thanks to those who assisted in the production of this report, and to all those who were there at DebConf7! See you in Argentina, August 2008!

Yesterday, after a lot of time thinking on it, I uploaded all my perl packages to the pkg-perl's SVN, now I can dedicate a lot more of time to xscreensaver, gkrellm and libtorrent/rtorrent :-). I'm pretty sure that in january I will be able (with enough time) to work in some packages of the pkg-perl group.

... was a lot easier than expected. Just not very well documented.First of all, you need the appropriate utilities. Debian users can aptitude install libsmbios-bin Next identify your system. It will look something like this

$ sudo modprobe dcdbas
$ sudo getSystemId
Libsmbios:    0.13.10
System ID:    0x01D8
Service Tag:  ...REMOVED...
Express Service Code: ...although my warrany is over...
Product Name: MXC061
BIOS Version: A10
Vendor:       Dell Inc.
Is Dell:      1

The information you need is the "System ID".Now you need to get the so-called HDR file for your bios. This can either be extracted from their EXE file using wine (with -dump-hdr or so), or you can find it on the linux.dell.com server. This page contains a huge list, and there are tons of dirs like system_bios_ven_0x1028_dev_0x01d8_version_a10. 0x1028 apparently is "Dell". The second hex number is your System ID. The last number (A10 here) is the BIOS revision. Pick the appropriate directory. There should be a bios.hdr file in there.You can verify if the file is appropriate for your system:

$ sudo dellBiosUpdate -f bios.hdr -t

And do the update by calling

$ sudo modprobe dell_rbu
$ sudo dellBiosUpdate -f bios.hdr -u

When rebooting the next time, your screen might be garbled for a few seconds. At least it was for me. I was scared I might have trashed my system, but then it rebooted and had the new BIOS. So just give it some time (Fortunately I've done enough BIOS updates to know to just wait. I've even done a 'blind' video BIOS update on a Nvidia TNT. The first update had trashed the card, but I was able to redo the flash process without seeing anything on the screen, and guess what, the card worked again!)In case you're wondering how this works: as I understand it, the dell_rbu driver will reserve memory for the BIOS update. Being a kernel module, it can just lock the memory in place until the next reboot. It will store that address in CMOS for the Bios and set the update flag. On reboot, the current Bios will check if that the stored image is still intact (I bet they do some checksumming here!) and then load that into the BIOS flash. That way, you don't need to boot into a low-level system such as Dos or Dos-Mode anymore to do an update.