Search Results: "bap"

20 November 2010

Petter Reinholdtsen: Lenny->Squeeze upgrades, apt vs aptitude with the Gnome and KDE desktop

I'm still running upgrade testing of the Lenny Gnome and KDE Desktop, but have not had time to spend on reporting the status. Here is a short update based on a test I ran 20101118. I still do not know what a correct migration should look like, so I report any differences between apt and aptitude and hope someone else can see if anything should be changed. This is for Gnome: Installed using apt-get, missing with aptitude
apache2.2-bin aptdaemon at-spi baobab binfmt-support browser-plugin-gnash cheese-common cli-common cpp-4.3 cups-pk-helper dmz-cursor-theme empathy empathy-common finger freedesktop-sound-theme freeglut3 gconf-defaults-service gdm-themes gedit-plugins geoclue geoclue-hostip geoclue-localnet geoclue-manual geoclue-yahoo gnash gnash-common gnome gnome-backgrounds gnome-cards-data gnome-codec-install gnome-core gnome-desktop-environment gnome-disk-utility gnome-screenshot gnome-search-tool gnome-session-canberra gnome-spell gnome-system-log gnome-themes-extras gnome-themes-more gnome-user-share gs-common gstreamer0.10-fluendo-mp3 gstreamer0.10-tools gtk2-engines gtk2-engines-pixbuf gtk2-engines-smooth hal-info hamster-applet libapache2-mod-dnssd libapr1 libaprutil1 libaprutil1-dbd-sqlite3 libaprutil1-ldap libart2.0-cil libatspi1.0-0 libboost-date-time1.42.0 libboost-python1.42.0 libboost-thread1.42.0 libchamplain-0.4-0 libchamplain-gtk-0.4-0 libcheese-gtk18 libclutter-gtk-0.10-0 libcryptui0 libcupsys2 libdiscid0 libeel2-data libelf1 libepc-1.0-2 libepc-common libepc-ui-1.0-2 libfreerdp-plugins-standard libfreerdp0 libgail-common libgconf2.0-cil libgdata-common libgdata7 libgdl-1-common libgdu-gtk0 libgee2 libgeoclue0 libgexiv2-0 libgif4 libglade2.0-cil libglib2.0-cil libgmime2.4-cil libgnome-vfs2.0-cil libgnome2.24-cil libgnomepanel2.24-cil libgnomeprint2.2-data libgnomeprintui2.2-common libgnomevfs2-bin libgpod-common libgpod4 libgtk2.0-cil libgtkglext1 libgtksourceview-common libgtksourceview2.0-common libmono-addins-gui0.2-cil libmono-addins0.2-cil libmono-cairo2.0-cil libmono-corlib2.0-cil libmono-i18n-west2.0-cil libmono-posix2.0-cil libmono-security2.0-cil libmono-sharpzip2.84-cil libmono-system2.0-cil libmtp8 libmusicbrainz3-6 libndesk-dbus-glib1.0-cil libndesk-dbus1.0-cil libopal3.6.8 libpolkit-gtk-1-0 libpt-1.10.10-plugins-alsa libpt-1.10.10-plugins-v4l libpt2.6.7 libpython2.6 librpm1 librpmio1 libsdl1.2debian libservlet2.4-java libsrtp0 libssh-4 libtelepathy-farsight0 libtelepathy-glib0 libtidy-0.99-0 libxalan2-java libxerces2-java media-player-info mesa-utils mono-2.0-gac mono-gac mono-runtime nautilus-sendto nautilus-sendto-empathy openoffice.org-writer2latex openssl-blacklist p7zip p7zip-full pkg-config python-4suite-xml python-aptdaemon python-aptdaemon-gtk python-axiom python-beautifulsoup python-bugbuddy python-clientform python-coherence python-configobj python-crypto python-cupshelpers python-cupsutils python-eggtrayicon python-elementtree python-epsilon python-evolution python-feedparser python-gdata python-gdbm python-gst0.10 python-gtkglext1 python-gtkmozembed python-gtksourceview2 python-httplib2 python-louie python-mako python-markupsafe python-mechanize python-nevow python-notify python-opengl python-openssl python-pam python-pkg-resources python-pyasn1 python-pysqlite2 python-rdflib python-serial python-tagpy python-twisted-bin python-twisted-conch python-twisted-core python-twisted-web python-utidylib python-webkit python-xdg python-zope.interface remmina remmina-plugin-data remmina-plugin-rdp remmina-plugin-vnc rhythmbox-plugin-cdrecorder rhythmbox-plugins rpm-common rpm2cpio seahorse-plugins shotwell software-center svgalibg1 system-config-printer-udev telepathy-gabble telepathy-mission-control-5 telepathy-salut tomboy totem totem-coherence totem-mozilla totem-plugins transmission-common xdg-user-dirs xdg-user-dirs-gtk xserver-xephyr zip
Installed using apt-get, removed with aptitude
arj bluez-utils cheese dhcdbd djvulibre-desktop ekiga eog epiphany-extensions epiphany-gecko evolution-exchange fast-user-switch-applet file-roller gcalctool gconf-editor gdm gedit gedit-common gnome-app-install gnome-games gnome-games-data gnome-nettool gnome-system-tools gnome-themes gnome-utils gnome-vfs-obexftp gnome-volume-manager gnuchess gucharmap guile-1.8-libs hal libavahi-compat-libdnssd1 libavahi-core5 libavahi-ui0 libbind9-50 libbluetooth2 libcamel1.2-11 libcdio7 libcucul0 libcurl3 libdirectfb-1.0-0 libdmx1 libdvdread3 libedata-cal1.2-6 libedataserver1.2-9 libeel2-2.20 libepc-1.0-1 libepc-ui-1.0-1 libexchange-storage1.2-3 libfaad0 libgadu3 libgalago3 libgd2-noxpm libgda3-3 libgda3-common libggz2 libggzcore9 libggzmod4 libgksu1.2-0 libgksuui1.0-1 libgmyth0 libgnome-desktop-2 libgnome-pilot2 libgnomecups1.0-1 libgnomeprint2.2-0 libgnomeprintui2.2-0 libgpod3 libgraphviz4 libgtk-vnc-1.0-0 libgtkhtml2-0 libgtksourceview1.0-0 libgtksourceview2.0-0 libgucharmap6 libhesiod0 libicu38 libisccc50 libisccfg50 libiw29 libjaxp1.3-java-gcj libkpathsea4 liblircclient0 libltdl3 liblwres50 libmagick++10 libmagick10 libmalaga7 libmozjs1d libmpfr1ldbl libmtp7 libmysqlclient15off libnautilus-burn4 libneon27 libnm-glib0 libnm-util0 libopal-2.2 libosp5 libparted1.8-10 libpisock9 libpisync1 libpoppler-glib3 libpoppler3 libpt-1.10.10 libraw1394-8 libsdl1.2debian-alsa libsensors3 libsexy2 libsmbios2 libsoup2.2-8 libspeexdsp1 libssh2-1 libsuitesparse-3.1.0 libsvga1 libswfdec-0.6-90 libtalloc1 libtotem-plparser10 libtrackerclient0 libvoikko1 libxalan2-java-gcj libxerces2-java-gcj libxklavier12 libxtrap6 libxxf86misc1 libzephyr3 mysql-common rhythmbox seahorse sound-juicer swfdec-gnome system-config-printer totem-common totem-gstreamer transmission-gtk vinagre vino w3c-dtd-xhtml wodim
Installed using aptitude, missing with apt-get
gstreamer0.10-gnomevfs
Installed using aptitude, removed with apt-get
[nothing]
This is for KDE: Installed using apt-get, missing with aptitude
autopoint bomber bovo cantor cantor-backend-kalgebra cpp-4.3 dcoprss edict espeak espeak-data eyesapplet fifteenapplet finger gettext ghostscript-x git gnome-audio gnugo granatier gs-common gstreamer0.10-pulseaudio indi kaddressbook-plugins kalgebra kalzium-data kanjidic kapman kate-plugins kblocks kbreakout kbstate kde-icons-mono kdeaccessibility kdeaddons-kfile-plugins kdeadmin-kfile-plugins kdeartwork-misc kdeartwork-theme-window kdeedu kdeedu-data kdeedu-kvtml-data kdegames kdegames-card-data kdegames-mahjongg-data kdegraphics-kfile-plugins kdelirc kdemultimedia-kfile-plugins kdenetwork-kfile-plugins kdepim-kfile-plugins kdepim-kio-plugins kdessh kdetoys kdewebdev kdiamond kdnssd kfilereplace kfourinline kgeography-data kigo killbots kiriki klettres-data kmoon kmrml knewsticker-scripts kollision kpf krosspython ksirk ksmserver ksquares kstars-data ksudoku kubrick kweather libasound2-plugins libboost-python1.42.0 libcfitsio3 libconvert-binhex-perl libcrypt-ssleay-perl libdb4.6++ libdjvulibre-text libdotconf1.0 liberror-perl libespeak1 libfinance-quote-perl libgail-common libgsl0ldbl libhtml-parser-perl libhtml-tableextract-perl libhtml-tagset-perl libhtml-tree-perl libio-stringy-perl libkdeedu4 libkdegames5 libkiten4 libkpathsea5 libkrossui4 libmailtools-perl libmime-tools-perl libnews-nntpclient-perl libopenbabel3 libportaudio2 libpulse-browse0 libservlet2.4-java libspeechd2 libtiff-tools libtimedate-perl libunistring0 liburi-perl libwww-perl libxalan2-java libxerces2-java lirc luatex marble networkstatus noatun-plugins openoffice.org-writer2latex palapeli palapeli-data parley parley-data poster psutils pulseaudio pulseaudio-esound-compat pulseaudio-module-x11 pulseaudio-utils quanta-data rocs rsync speech-dispatcher step svgalibg1 texlive-binaries texlive-luatex ttf-sazanami-gothic
Installed using apt-get, removed with aptitude
amor artsbuilder atlantik atlantikdesigner blinken bluez-utils cvs dhcdbd djvulibre-desktop imlib-base imlib11 kalzium kanagram kandy kasteroids katomic kbackgammon kbattleship kblackbox kbounce kbruch kcron kdat kdemultimedia-kappfinder-data kdeprint kdict kdvi kedit keduca kenolaba kfax kfaxview kfouleggs kgeography kghostview kgoldrunner khangman khexedit kiconedit kig kimagemapeditor kitchensync kiten kjumpingcube klatin klettres klickety klines klinkstatus kmag kmahjongg kmailcvt kmenuedit kmid kmilo kmines kmousetool kmouth kmplot knetwalk kodo kolf kommander konquest kooka kpager kpat kpdf kpercentage kpilot kpoker kpovmodeler krec kregexpeditor kreversi ksame ksayit kshisen ksig ksim ksirc ksirtet ksmiletris ksnake ksokoban kspaceduel kstars ksvg ksysv kteatime ktip ktnef ktouch ktron kttsd ktuberling kturtle ktux kuickshow kverbos kview kviewshell kvoctrain kwifimanager kwin kwin4 kwordquiz kworldclock kxsldbg libakode2 libarts1-akode libarts1-audiofile libarts1-mpeglib libarts1-xine libavahi-compat-libdnssd1 libavahi-core5 libavc1394-0 libbind9-50 libbluetooth2 libboost-python1.34.1 libcucul0 libcurl3 libcvsservice0 libdirectfb-1.0-0 libdjvulibre21 libdvdread3 libfaad0 libfreebob0 libgd2-noxpm libgraphviz4 libgsmme1c2a libgtkhtml2-0 libicu38 libiec61883-0 libindex0 libisccc50 libisccfg50 libiw29 libjaxp1.3-java-gcj libk3b3 libkcal2b libkcddb1 libkdeedu3 libkdegames1 libkdepim1a libkgantt0 libkleopatra1 libkmime2 libkpathsea4 libkpimexchange1 libkpimidentities1 libkscan1 libksieve0 libktnef1 liblockdev1 libltdl3 liblwres50 libmagick10 libmimelib1c2a libmodplug0c2 libmozjs1d libmpcdec3 libmpfr1ldbl libneon27 libnm-util0 libopensync0 libpisock9 libpoppler-glib3 libpoppler-qt2 libpoppler3 libraw1394-8 librss1 libsensors3 libsmbios2 libssh2-1 libsuitesparse-3.1.0 libswfdec-0.6-90 libtalloc1 libxalan2-java-gcj libxerces2-java-gcj libxtrap6 lskat mpeglib network-manager-kde noatun pmount tex-common texlive-base texlive-common texlive-doc-base texlive-fonts-recommended tidy ttf-dustin ttf-kochi-gothic ttf-sjfonts
Installed using aptitude, missing with apt-get
dolphin kde-core kde-plasma-desktop kde-standard kde-window-manager kdeartwork kdebase kdebase-apps kdebase-workspace kdebase-workspace-bin kdebase-workspace-data kdeutils kscreensaver kscreensaver-xsavers libgle3 libkonq5 libkonq5-templates libnetpbm10 netpbm plasma-widget-folderview plasma-widget-networkmanagement xscreensaver-data-extra xscreensaver-gl xscreensaver-gl-extra xscreensaver-screensaver-bsod
Installed using aptitude, removed with apt-get
kdebase-bin konq-plugins konqueror

16 November 2010

Andrew Pollock: [tech] The knock-on effects of buying a new TV

I'm considering replacing our 5 year old 32" TV with a 40" Sony Internet TV, largely "just because" it has Google TV. That and I have spousal approval to get a new, bigger TV (it just has to be wall mounted). The dilemma is MythTV. It's currently still running on the Dell Dimension 3100 that I bought years ago, and connected via VGA to the TV. The Sony Internet TV is HDMI or nothing, so at the very least, I need to get a DVI-capable video card in my MythTV box. The more or less natural choice would be an NVIDIA card, despite them having a closed-source driver. At least it's well supported under Linux. The problem is, as far as I can determine, NVIDIA cards require at least a 300 watt power supply, and the Dimension 3100 has only a 230 watt power supply. So now I'm looking at having to replace the MythTV box. I'm currently running the front end and the back end on the same box, so it'd be a fairly simple case of splitting the back end from the front end, and just getting something new to be the front end. So now I'm in the market for something small, low-power, has a HDMI (or DisplayPort) interface, and enough grunt for HD playback. I'm wondering if the Boxee Box is the solution. Interface-wise, it has HDMI, so I'm good there, and I believe the marketing claims it can do HD playback. I just don't know how hackable it is. I think it's running Linux, so in theory this should be doable, so long as there isn't some weird hardware in it with a binary-only driver. For the price, it's probably worth just getting one to see. The annoying thing is, I have nothing in the house that is DVI or HDMI capable, so I have no way of testing this out prior to buying the TV.

8 November 2010

Simon McVittie: Debian activities

Most of my recent Debian work has been the usual pkg-telepathy stuff (mainly in experimental while we're frozen), and hacking on the Quake III engine. Having started working on OpenArena DFSG-freeness as random bug-squashing a year ago, I've accidentally ended up taking over the package. The situation for Debian 6.0 (squeeze) looks something like this: The plan for Quake III in Debian 7.0 (wheezy) has already started in experimental: I haven't been doing as much QA work as the RCBW crowd, but here's some halfway-recent bug squashing in the hope that it motivates others: I also started looking at the series of bugs regarding Flash not compiled from source, but I mostly got distracted by all the webapps having a contrib/ directory containing a million embedded code copies...

21 October 2010

Ian Wienand: Logging POST requests with Apache

After getting a flood of spam, I became suspicious that there was an exploit in my blog software allowing easy robo-posts. Despite a code audit I couldn't see anything, and thus wanted to log the incoming POST requests before any local processing at all. It took me a while to figure out how to do this, hopefully this helps someone else. Firstly install libapache-mod-security, then the magic incarnation is 
SecRuleEngine On
SecAuditEngine on
SecAuditLog /var/log/apache2/website-audit.log
SecRequestBodyAccess on
SecAuditLogParts ABIFHZ
SecDefaultAction "nolog,noauditlog,allow,phase:2"
SecRule REQUEST_METHOD "^POST$" "chain,allow,phase:2"
SecRule REQUEST_URI ".*" "auditlog"
So, to break it down a little, the default action says to do nothing during phase 2 (when the body is available for inspection); the allow means that we're indicating that nothing further will happen in any of the remaining phases, so the module can shortcut through them. The two SecRules work together -- the first says that any POST requests should be tested by the next rule (i.e. the chained rule), which in this case says that any request should be sent to the audit log. After that, the similar allow/phase argument again says that nothing further is going to happen in any of the subsequent phases mod_security can work on. As per the parts between A and Z, we'll log the headers, the request body, the final response and trailer. So, as it turns out, there is no exploit; it seems most likely there is an actual human behind the spam that gets through, because every time they take a guess it is correct. So I guess I'll take a glass-half-full kind of approach and rather than being annoyed at removing the spam, I'll just convince myself that I made a small donation from some spam overlord to one of their poor minions!

19 October 2010

Torsten Landschoff: Fun with my Sony PRS-600 eBook reader

I finally bought myself an eBook reader. To be precise, I got the Sony PRS-600 Reader Touch Edition. Two reasons: First of all, the display is only really great with a powerful light source. The reflection is a bit of a nuisance but I can live with it. What I don t like is the PDF support of the device. Viewing a normal page, the font faces are so tiny that it is not readable in any way. The zoom feature works around that, but you can t switch pages from there. Instead you have to unzoom, go to the next page and zoom again. Crappy! There is also the reflow option, which does not mix with source code figures in my books. I am glad that most of my electronic books are from The Pragmatic Bookshelf, which means that I can download them in Epub as well, all without DRM. Thanks, guys! For the PDFs I would like to carry, I googled a while and stumbled across this blog post. The example to work with source documents worked fine for The Little Book of Semaphores at least. But even with BRISS I did not (yet) manage to crop books I bought from APress. Most likely I will try doing it via a bit of python scripting and the poppler bindings for python. I also bought my first DRM content: Star Trek Vanguard: Harbinger. I really enjoyed the book, but the download was a PITA. After buying, you only get a link to a ASCM file, which is not useful by itself. One needs Adobe Digital Editions to download it. So I installed it into a virtual machine which has XP installed. After that, I was able to read it on the PC, which wasn t the goal of the exercise. To put it on the reader, one needs to authorize it from the same Windows software and upload the file. All that stuff took almost an hour and of course the digital edition of the book is more expensive than getting it from amazon germany. But at least I did not have to wait for delivery. Having finished the book I wanted to buy the second part, but of course it is only available for canadian residents on ebooks.com. I researched a number of ebook sites which mostly sell it only in the US. Funny enough, the first hits when searching where pirated versions of the book in full text. Why do they make it so hard to get stuff legally? *sigh*

6 October 2010

Jamie McClelland: Benchmarking PHP

What's the best way to run PHP in Debian in a shared environment? Years ago May First/People Link switched from the old reliable modphp to suPHP because modphp ran all php scripts as the same user, meaning (among other things) that any site could access the database configurations of any other site. suPHP was great, however, after several years we've decided to replace it because: We first turned to fcgid. We tried it out on a couple individual web sites (fcgid can run alongside suPHP), and, after a week without much problems, we implemented it on an entire server. Out of the box, on a shared server with several dozen web sites, the most notable resource utilization difference between fcgid and suPHP is the increase in memory usage. Unlike suPHP which dies after each requests, fcgid gains a performance boost by sticking around (with either 2 or 3 processes living on) to server future requests. If you have several dozen web sites that aren't high traffic sites, it means they may each have up to the three processes staking out memory that would otherwise be free with suPHP. We addressed the memory problem by adding the following to /etc/apache2/mods-available/fcgid.conf: 
DefaultMinClassProcessCount 0
IdleTimeout 60
IdleScanInterval 60
These directions tell fcgid to kill processes that have been idle for more than 60 seconds. That means we still get the performance gain from a site getting more than one page view per minute, but otherwise we re-claim the memory. Before switching our remaining servers we decided to do more extensive testing to ensure we would be getting the performance and resource gains we were hoping. Also, we decided to consider mpm-itk, which is a variation of mpm-prefork that runs each virtual host with a per-vhost configured user and group (like suexec, but for the entire virtual host). Since it's a variation of mpm-prefork, it's safe to run it with the more resource friendly mod_php. We used the apache benchmarking tool ab. And we tested on a virtual machine (running on my laptop) with 1GB of RAM. The test web site we used was a copy of the May First/People Link site (we used the home page for the test), which is running Drupal 6. It's a relatively simple Drupal site, however, we turned Drupal caching off to better simulate PHP processing usage likely to be found on our members' sites. We ran tests against the following configurations: The tests were run with: We originally tried, for the last test, 100 requests/25 concurrent but it caused ab to time out on the mpm-worker/fcgid/suexec configuration. We then lowered it to 75/15. Still timed out (load on our test server went way up). The tests paused 120 seconds between testing each environment to give the server load a chance to settle down. And we paused 30 seconds between each test (not enough time for the fcgid processes to be killed, but enough time for all the processes to hopefully complete their requests). The tests are scripted and the full results are available. You can also see the various vhost configurations (test.mouse, test.mouse.itk, test.mouse.suexec.fcgid), and the fcgid.conf. To summarize the report ... if you are running one web site on a server, there's little argument for using anything other than mod_php and mpm-prefork - the performance of this combination far exceeds any other combination. I'm scratching my head over the the vastly different conclusions reached by 2bits. They didn't publish their tests so I'm not sure exactly how to compare. Maybe they have a lot more static content? Or - maybe there's a mistake in our tests? Open to ideas. suPHP seems to be left in the dust compared with either fcgid/suexec or mpm-itk. Given that it's also less flexible, there's not much going for it. And lastly... fcgid/suexec and mpm-itk seem comparable. That was a surprise. We were expecting mpm-itk to perform better because it's running mod_php. Given the similarity in performance, fcgid/mpm-worker seems like a much more well-tested and well-supported approach for a shared server. One side note... originally, out of laziness, we tested against the default Drupal install.php page. Less static content and much less PHP processing. In these tests, with low loads, the results provide different numbers, but the comparisons between environment were similar ... except with the last test. With the 70 requests/10 concurrent test, fcgid/suexec dramatically out-performed everyone (by a factor or more than 2). That's probably because no new cgi processes had to be spawned and the effect of mod_php using APC was reduced because there was very little php code to process.

2 September 2010

Russell Coker: Raw Satire Usually Fails on the Internet

Sarcasm and satire usually don t work on the Internet. One cause of this is the lack of out of band signalling via facial expression or tone of voice. Another issue is the fact that in real life people usually know something about the person who they listen to while on the Internet it s most common to read articles without knowing much about the author. So the reader can t use I know that the author isn t an asshole as a starting point to determine whether a message should be interpreted literally. This is really nothing new. The standard in printed communication for a long time has been to use Emoticons (Wikipedia) to indicate emotion and other interpretation that might not be deduced from a direct reading of the text. The Wikipedia page cites examples of emoticon use dating back to 1857 although the combinations of characters used for different emotions has changed significantly many times. The common uses that we now know on the Internet date back to 1982. In my experience the symbol :-# is commonly used to note sarcasm or satire. Unfortunately it seems that none of the Internet search engines allow searching for such strings so I couldn t find an early example of this being used. While I haven t found a reference describing this practice, I regularly receive messages annotated with it and find that people generally understand what I mean when I use it in my own email. But that is usually applied to a sentence or two. For a larger section of text a pseudo-HTML tag such as </satire> can be used to signal the end of satire. It seems that a matching start tag is optional as recognising the start of satire is a lot easier once the reader knows that some of the content is satirical. In spoken English a phrase such as but seriously may be used for the same purpose, but such a subtle signal may be missed on the Internet particularly by readers who don t use English as their first language. Another way of signaling a non-literal interpretation is by using Scare Quotes the deliberate usage of quotation symbols to indicate that the writer disagrees with the content that is written. That is common for the case of referencing a phrase or sentence that you disagree with, but doesn t work for a larger section of text. A final option is to make the satire or sarcasm so extreme that no-one can possibly mistake it for being literal. This is not always possible, Poe s Law holds that Without a winking smiley or other blatant display of humor, it is impossible to create a parody of Fundamentalism that SOMEONE won t mistake for the real thing [1]. I think that Poe was understating the case, it is impossible to create a parody of religion that most people won t mistake for the real thing without signals or context. For an example read LandOverBaptist.org and Chick.com, of course if you know those sites then you will know whether they are satirical or serious but I expect that most readers of my blog won t invest enough effort into either of those religious sites to determine whether they are serious or satire. But satire and sarcasm without signals or a reputation usually fails. One example of success is The Onion which is a long running and well known satirical news site [2]. But even The Onion it is regularly mistaken for being serious the number of occasions when people forward me Onion articles for amusement are vastly outnumbered by the number of occasions when I see people taking it seriously. Even when material is known to be satirical it can still fail grossly. An example is the Chaser s satire of the Make A Wish Foundation [3]. Even material that is well known to be satirical seems to fail when it attacks bad targets or attacks in a bad way. One difficulty is in satirising bigoted people, to effectively satirise them without attacking the minority groups that they dislike can be a difficult challenge. Finally, when you write some satire and members of your audience don t recognise it you should consider the possibility that you failed to do it properly. If you can t get a hit rate close to 100% for people with the same background as you then it s probably a serious failure.

30 August 2010

Tim Retout: Hacking

Here in the UK we've had a bank holiday weekend. Usually I would have gone to Cambridge for the Debian BBQ, but this year I joined forces with Thomas Adam for some SmoothWall Express on Debian hacking. There are several challenges involved in moving the SWE3 code from its native distribution to Debian; this weekend we worked around some of the permissions problems. On SWE3, the web server and most of the service daemons run as the user 'nobody'. This means that the web server can write out configuration files as the same user as everything else; it can also read the system log files. When the web interface needs to run a privileged action (like setting firewall rules), it sends a command to 'smoothd', which is a daemon running as root. (Admin ssh access is always directly as the root user.) However, web servers in Debian tend to run as user 'www-data', which does not have permission to read log files. Similarly, writing out configuration files as that user would mean that any cgi script (not just ones in the swe3 package) could modify them. I would prefer to run the swe3 cgi scripts as a separate user, and grant this user permission to view logs etc. This debian-webapps thread makes it sound very easy, but if you want to do that with cgi scripts rather than fastcgi, I think you have to run a separate web server for each user. On Sunday afternoon, in a dramatic display of corner-cutting, I gave up on that approach and added www-data to the 'adm' and 'proxy' groups in the postinst. Thomas heroically patched all the cgis to call the "config writers" via smoothd, although I'm wondering whether some careful use of the chgrp command in the postinst might be better than running that code as root. Another hack: in order to actually start a firewall, we needed to know which network card is the "RED" interface, in SWE3 terms; i.e. which one is meant to be the public-facing network device. It's also nice to know which one is "GREEN". So two debconf questions and some hardcoded magic numbers later, we have a basic firewall init script. Lovely. Oh, and at some point I removed the htaccess file, so any user on your network can mess with your firewall. Should probably fix that. Today I started some awful scripts which use Simple-CDD to build an iso containing all the packages we want. We are going to need to extend the networking configuration in the installer to set up multiple network cards. Then we need to figure out a nicer way of assigning IP addresses to devices; unlike on the proprietary version of the product, there seems to be no web configuration of network settings in Express. I've not figured all of this out yet. So, in summary: we are deliberately trading some technical debt in order to quickly produce an initial release that might interest people. (But please note the disclaimer of warranty in the README file in that directory.) And in other news, I've been working at Smoothwall Ltd. for just over one year. Hmm. So this is what they call 'experience'.

29 August 2010

Steve Kemp: Proxies and Robots

I don't like repeating myself, but I'm very tempted to past my mini-review of the Roomba Vacuum Cleaner robot into this blog. Instead I will practise restraint and summerise: 250. Worth. Every. Penny. In more Debian-friendly news I've been fighting HTTP proxies today. I've noticed a lot of visitors to the various websites I host are logged as 127.0.0.1 - which is an irritation. My personal machine looks like this: 
Internet -> Apache listening on *:80 -> thttpd on 127.0.0.1:xxxx
(This has been documented previously - primarily it is a security restriction. It means I can run per-UID web-servers.) I had previous added a patch to thttpd to honour the X-Forwarded-For: header - so that it would receive the correct remote address passed on from Apache. However the fact that so many visitors are logged as coming from 127.0.0.1 meant it wasn't working 100% correctly, and I wanted to understand why. Today I used ngrep to capture the incoming headers and the source of the problem became apparent: 
skx:~# ngrep  -d lo  X-For ' port 1007'
..
T 127.0.0.1:41886 -> 127.0.0.1:1007 [AP]
  GET /about/ HTTP/1.1..Host: images.steve.org.uk..If-Modified-Since: Mon, 07
   Jun 2010 15:24:33 GMT..User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-U
  S; rv:1.9.1.10) Gecko/20100701 Iceweasel/3.5.10 (like Firefox/3.5.10)..Acce
  pt: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8..Accept
  -Language: en-us,en;q=0.5..Accept-Encoding: gzip,deflate..Accept-Charset: I
  SO-8859-1,utf-8;q=0.7,*;q=0.7..Referer: http://images.steve.org.uk/2009/11/
  20/img_0471.html..X-Forwarded-For: 127.0.0.1, 11.22.33.123..Cache-Control:
  max-age=0..X-Forwarded-Host: images.steve.org.uk..X-Forwarded-Server: image
  s.steve.org.uk..Connection: Keep-Alive....
I bolded the important input; just in case that didn't jump out it was:
X-Forwarded-For: 127.0.0.1, 11.22.33.123
My patch to thttpd was making it read the first address, rather than the second - which meant that requests were being logged as coming from 127.0.0.1 and avoiding my efforts to track sources. Now I understand the problem - The X-Forwarded-Host header is being tweaked by a proxy server, such as Squid, upstream of my server. For the moment I've updated the thttpd patch to read: 
        else if ( strncasecmp( buf, "X-Forwarded-For:", 16 ) == 0 )
            char *tmp = NULL;
            /* Jump to the header-value  */
            cp = &buf[16];
            cp += strspn( cp, " \t" );
            /*
             * If the first change is a 127.0.0.1, then we'll
             * jump over it.  Cope with Squid, et al.
             */
            if (  ( tmp = strstr( cp, "127.0.0.1, " ) ) != NULL )
              cp = tmp + strlen( "127.0.0.1, " );
            /* Parse the IP */
           inet_aton( cp, &(hc->client_addr.sa_in.sin_addr) );
That's not perfect, but the alternative would be: Or something equally hacky and security-by-obscurity-alike. Really I just want a simple way of always getting the correct remote IP. Shouldn't be so hard, should it? *pout*. ObQuote: "You don't mess with fate, Peanut. People die when they are meant to die. There's no discussion. There's no negotiation. When life's done, it's done." - Dead Like Me.

25 August 2010

Christian Perrier: [life nolife] Debconf 10 was...

...awesome. OK, I'm writing this while I'm still in USA, but there are so many things to say about these weeks that I can't write them in only one blog post. And, still, this one will be quite long as it will talk about hacking, running and sightseeing...:) Let's start about hacking: after all, this is the first reason for being there in US, isn't it? I cam to DebConf with a very long TODO list and, for the first time in seven DebConfs, I'm pretty happy with what I achieved from it: As one can see, a lot of planned work happened while I still could maintain the usual flow of recurrent work with localization (Smith reviews, l10n NMUs). Some asked me why I didn't propose l10n sessions this year. Indeed, I wasn't feeling I could sustain animating them and I had no clear idea about which topic I could bring to be discussed. Last year, these sessions slightly killed my free time and I wanted to keep some this year for "impromptu" things. I didn't attend many talks, sorry for the speakers. The most I attended were during Debian Day, which I found highlyinteresting and motivating, just like Eben Moglen's talk. Marga's talk was also one I wanted to attend, though I regreted that things went mostly out of control during the talk (too many comments from the audience to allow Marga pushing her important points). As usual, I invested a big part of my time in "social" activities, the most proeminent being of course the Cheese and Wine party, which turned ut to be a great success. The help of my son Jean-Baptiste and the tremendous support of Michelle Lynn Hall helped a lot, though I still regret that we screwed about accessibility. I also ran a lot..:-)..that may be counted as social activities as I organized several group runs. The one I'm proud of has been participating to a local race, namely the Van Cortland Track Club Summer Series of cross-country running, in Bronx. We went there with no less than 10 DebConf participants and 1 kilt (hey, Luca!). All of us completed the race (that had 170 runners for 5 kilometers) and No l K the even finished 17th scratch and 2nd in his age/gender category. Besides that, we had a great run/sightseeing to Georges Washington Bridge (that links New Jersey and Harlem and offers an unusual view of Manhattan "from behind"). All this with a 17km run. We also ran several times in Central Park, and No l and me happened to go to Coney Island for the Day Trip by doing half of the trip by running (all around Manhattan and over the Broolyn Bridge), for about 20km. Then we "showered" in the Atlantic Ocean....:). At the end of DebConf, I think that I had my record broken with 112km run in 10 days and only one day *without* running. What about sightseeing? Well, this blog post is too long and we reach the end of Interstate-90, close to Albany, so that will be for an upcoming blog post. Aug 25th update: back home, so now I can publish this blog post...

2 August 2010

Joachim Breitner: Protecting static content selectively by OpenID

When I created my on-line photo album, we had the year 2001 and there was nothing wrong with publishing ones photos in the Internet. It has been almost ten years now, and things have changed. Privacy is an issue, especially when it also affects your friends and family. The problem
So I needed to find some protection. These were my requirements:
  1. I only want to protect parts of the site, as there are some pictures that I intentionally share with the public. This selection should be possible down to the individual image.
  2. The solution should work without any dynamic server component besides the web server itself. This rules out any self-written CGI scripts as well.
  3. One password, given to all my friends etc., is not sufficient as it might leak to unintended audience.
  4. I do not want my visitors to have to register and remember yet another username and password just for my site. I also do not want to manage this user database.
  5. I do not want to have do large changes to the file structure of my photo album.
This rules out most common options, e.g. protection by a .htpasswd file. Requirements 3 and 4 point to a solution based on OpenID. With OpenID, my visitors can authenticate against a service they already use (Google, Yahoo, etc.), relieving me from the burden of maintaining a user database and them from having to remember a password. There is a mod_auth_openid module for the Apache webserver, and it is even distributed with Debian in the libapache2-mod-auth-openid package. So requirement 2 is fulfilled. The tricky part is: How do we achieve OpenID protection for some images, and not for others. The solution I first played around with selectively enabling or disabling mod_auth_openid based on <FileMatch> directives in the Apache configuration, but it was not elegant and would not scale well. I have more than 20.000 pictures to manage, and have already selected over 5000 pictures to be shown without protection. My solution is based on a partial copy of the whole directory tree that contains all public files. To save disk space, these are just symbolic links to the real file in the protected location. Some mod_rewrite magic then takes care of giving the user the impression that all files are in the same location. I set up a small example of my solution, which has this directory structure: 
.:
drwxr-xr-x 2 root root 4096  2. Aug 12:03 images
lrwxrwxrwx 1 root root   18  1. Aug 12:05 index.html -> private/index.html
lrwxrwxrwx 1 root root   18  1. Aug 12:05 login.html -> private/login.html
drwxr-xr-x 3 root root 4096  2. Aug 12:03 private
./images:
lrwxrwxrwx 1 root root 33  2. Aug 12:00 pleaselogin.png -> ../private/images/pleaselogin.png
lrwxrwxrwx 1 root root 28  2. Aug 12:03 public.png -> ../private/images/public.png
./private:
drwxr-xr-x 2 root root 4096  2. Aug 12:00 images
-rw-r--r-- 1 root root  267  2. Aug 12:03 index.html
-rw-r--r-- 1 root root   94  2. Aug 12:01 loggedin.html
-rw-r--r-- 1 root root 2091  2. Aug 12:03 login.html
-rw-r--r-- 1 root root   10 18. Nov 2009  protected.html
./private/images:
-rw-r--r-- 1 root root 4074  2. Aug 11:58 pleaselogin.png
-rw-r--r-- 1 root root 2670  2. Aug 11:58 private.png
-rw-r--r-- 1 root root 2043  2. Aug 11:58 public.png
As you can see, real files only reside in private/, outside of that, only symbolic links exist. The apache configuration protects the private directory and blends it into the main directory: 
   <directory /var/www/nomeata.de/openid-test>
        RewriteEngine On
        # Abuse the login page as an error image
        RewriteCond % QUERY_STRING  \.(png jpg)
        RewriteRule ^login.html$ /openid-test/images/pleaselogin.png
        # Ship private files, if they exist, unless public files exist
        RewriteCond  $1 !^private
        RewriteCond  /var/www/nomeata.de/openid-test/$1 !-f
        RewriteCond  /var/www/nomeata.de/openid-test/private/$1 -f
        RewriteRule  ^(.+)$ /openid-test/private/$1
   </directory>
   <directory /var/www/nomeata.de/openid-test/private>
        AuthOpenIDEnabled        On
        AuthOpenIDDBLocation     /var/lib/apache2/mod_auth_openid/mod_auth_openid.db
        AuthOpenIDLoginPage      /openid-test/login.html
        AuthOpenIDTrustRoot      http://nomeata.de
        AuthOpenIDCookiePath     /
        AuthOpenIDCookieLifespan 2592000
    </directory>
A special trick handles the login page for protected images: If the login page is requested and the referrer indicates that the user tried to access a .png or .jpg file, apache will instead ship an image containing an error message. For my photo album I have a small Perl script that, given a directory with a private/ directory therein and a list of rules in form of glob patterns, will symlink matching files and remove symlinks that are not allowed any more.
What s next? As you can see, this does not actually protect the content. It only requires the user to authenticate, then everything is visible. To select which OpenIDs are allowed to access which code, some bugs will have to be fixed in mod_auth_openid first. There was little activity there recently, I hope that the project is not dead.

30 July 2010

Lucas Nussbaum: Debcamp+Debconf again!

As I did since 2007, I will be attending Debconf again this year. I have been quite busy recently, and am lagging behind in some of my Debian work, so I plan to use Debconf to get stuff done, not just socialize and attend talks. Here is my TODO list, basically by order of willingness to do the work . It s probably too much for 9 days, so don t hesitate to talk to me if you want to help with some items.

29 July 2010

Christian Perrier: First day at DebConf10

Yesterday late evening, Jean-Baptiste and I arrived in NYC after about a 16 hours trip. Amazingly, he mentioned to me that we entered a train in the station close to our home....and then never went outside until we exited the 1 line subway at 116th street on Broadway Ave.: It's indeed quite typical to notice that, these days, you can travel 1/4 of the planet, using 9 different trnsportation systems and never go outside. We made a quick appearance in the Hacklab yesterday, enough to say "hi" to a dozen of people around and mostlysee how things are organized. Then bed, which is probably the best way to avoid jetlag. This morning, I of course woke up quite early (Debian grandfathers, blah blah) and of course I went out running..:-) The initial plan was to do a short enough run in Central Park. However, I managed to get partly lost and, indeed, after entering the Park in the NW corner, I managed to read Columbus circle...in the SW corner after wandering around the large alleys. So, my only choice was then to come back through the westernmost part of the Park back to Columbia..and thus ended up with a 13km run (for running junkies, the pace was 5'31"/km (8'52"/mile for local runners). And now, I'm in the hacklab with JB (did I already say that I'm proud of this? Yes, dozens of times...OK...). And they're waiting for me to go to lunch....so, end of this blog post..:)

19 July 2010

Christian Perrier: Taking care of my pet geneweb package

This week-end, I didn't work a lot on Debian stuff (mostly running and gardeing...) but still managed to finally take care of my pet package, the one I maintain since 2001, namely geneweb. The package suffered from a very longstanding RC bug that made it unusable on amd64 machines. I had to call for help Ocaml gurus and R mi Vanicat kindly provided a working patch. So, now, geneweb can be released and I won't have to ask for my pet package to be removed from testing. I even switched it to dh7 and "minimal" debian/rules file (something I planned to do at DC10 but finally turned out to be easy to do. That will be my last week-end of activity in Debian until DebConf10 as I'll be away for the upcoming week-end. So, for those of you who attend DC10, see you in NYC. I'll arrive on Wed 28th evening for part of Debcamp and I'll have the great pleasure to introduce my son Jean-Baptiste to the Debian world.

10 July 2010

Enrico Zini: Searching OSM nodes in Spatialite

Searching OSM nodes in Spatialite Third step of my SoTM10 pet project: finding the POIs. I put together a query to find all nodes with a given tag inside a bounding box, and also a query to find all the tag values for a given tag name inside a bounding box. The result is this simple POI search engine: 
#
# poisearch - simple geographical POI search engine
#
# Copyright (C) 2010  Enrico Zini <enrico@enricozini.org>
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
#
from pysqlite2 import dbapi2 as sqlite
class PoiDB(object):
    def __init__(self):
        self.db = sqlite.connect("pois.db")
        self.db.enable_load_extension(True)
        self.db.execute("SELECT load_extension('libspatialite.so')")
        self.oldsearch = []
        self.bbox = None
    def set_bbox(self, xmin, xmax, ymin, ymax):
        '''Set bbox for searches'''
        self.bbox = (xmin, xmax, ymin, ymax)
    def tagid(self, name, val):
        '''Get the database ID for a tag'''
        c = self.db.cursor()
        c.execute("SELECT id FROM tag WHERE name=? AND value=?", (name, val))
        res = None
        for row in c:
            res = row[0]
        return res
    def tagnames(self):
        '''Get all tag names'''
        c = self.db.cursor()
        c.execute("SELECT DISTINCT name FROM tag ORDER BY name")
        for row in c:
            yield row[0]
    def tagvalues(self, name, use_bbox=False):
        '''
        Get all tag values for a given tag name,
        optionally in the current bounding box
        '''
        c = self.db.cursor()
        if self.bbox is None or not use_bbox:
            c.execute("SELECT DISTINCT value FROM tag WHERE name=? ORDER BY value", (name,))
        else:
            c.execute("SELECT DISTINCT tag.value FROM poi, poitag, tag"
                      " WHERE poi.rowid IN (SELECT pkid FROM idx_poi_geom WHERE ("
                      "       xmin >= ? AND xmax <= ? AND ymin >= ? AND ymax <= ?) )"
                      "   AND poitag.tag = tag.id AND poitag.poi = poi.id"
                      "   AND tag.name=?",
                      self.bbox + (name,))
        for row in c:
            yield row[0]
    def search(self, name, val):
        '''Get all name:val tags in the current bounding box'''
        # First resolve the tagid
        tagid = self.tagid(name, val)
        if tagid is None: return
        c = self.db.cursor()
        c.execute("SELECT poi.name, poi.data, X(poi.geom), Y(poi.geom) FROM poi, poitag"
                  " WHERE poi.rowid IN (SELECT pkid FROM idx_poi_geom WHERE ("
                  "       xmin >= ? AND xmax <= ? AND ymin >= ? AND ymax <= ?) )"
                  "   AND poitag.tag = ? AND poitag.poi = poi.id",
                  self.bbox + (tagid,))
        self.oldsearch = []
        for row in c:
            self.oldsearch.append(row)
            yield row[0], simplejson.loads(row[1]), row[2], row[3]
    def count(self, name, val):
        '''Count all name:val tags in the current bounding box'''
        # First resolve the tagid
        tagid = self.tagid(name, val)
        if tagid is None: return
        c = self.db.cursor()
        c.execute("SELECT COUNT(*) FROM poi, poitag"
                  " WHERE poi.rowid IN (SELECT pkid FROM idx_poi_geom WHERE ("
                  "       xmin >= ? AND xmax <= ? AND ymin >= ? AND ymax <= ?) )"
                  "   AND poitag.tag = ? AND poitag.poi = poi.id",
                  self.bbox + (tagid,))
        for row in c:
            return row[0]
    def replay(self):
        for row in self.oldsearch:
            yield row[0], simplejson.loads(row[1]), row[2], row[3]
Problem 3 solved: now on to the next step, building a user interface for it.

Enrico Zini: Importing OSM nodes into Spatialite

Importing OSM nodes into Spatialite Second step of my SoTM10 pet project: creating a searchable database with the points. What a fantastic opportunity to learn Spatialite. Learning Spatialite is easy. For example, you can use the two tutorials with catchy titles that assume your best wish in life is to create databases out of shapefiles using a pre-built, i386-only executable GUI binary downloaded over an insecure HTTP connection. To be fair, the second of those tutorials is called "An almost Idiot's Guide", thus expliciting the requirement of being an almost idiot in order to happily acquire and run software in that way. Alternatively, you can use A quick tutorial to SpatiaLite which is so quick it has examples that lead you to write SQL queries that trigger all sorts of vague exceptions at insert time. But at least it brought me a long way forward, at which point I could just cross reference things with PostGIS documentation to find out the right way of doing things. So, here's the importer script, which will probably become my reference example for how to get started with Spatialite, and how to use Spatialite from Python: 
#!/usr/bin/python
#
# poiimport - import nodes from OSM into a spatialite DB
#
# Copyright (C) 2010  Enrico Zini <enrico@enricozini.org>
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
#
import xml.sax
import xml.sax.handler
from pysqlite2 import dbapi2 as sqlite
import simplejson
import sys
import os
class OSMPOIReader(xml.sax.handler.ContentHandler):
    '''
    Filter SAX events in a OSM XML file to keep only nodes with names
    '''
    def __init__(self, consumer):
        self.consumer = consumer
    def startElement(self, name, attrs):
        if name == "node":
            self.attrs = attrs
            self.tags = dict()
        elif name == "tag":
            self.tags[attrs["k"]] = attrs["v"]
    def endElement(self, name):
        if name == "node":
            lat = float(self.attrs["lat"])
            lon = float(self.attrs["lon"])
            id = int(self.attrs["id"])
            #dt = parse(self.attrs["timestamp"])
            uid = self.attrs.get("uid", None)
            uid = int(uid) if uid is not None else None
            user = self.attrs.get("user", None)
            self.consumer(lat, lon, id, self.tags, user=user, uid=uid)
class Importer(object):
    '''
    Create the spatialite database and populate it
    '''
    TAG_WHITELIST = set(["amenity", "shop", "tourism", "place"])
    def __init__(self, filename):
        self.db = sqlite.connect(filename)
        self.db.enable_load_extension(True)
        self.db.execute("SELECT load_extension('libspatialite.so')")
        self.db.execute("SELECT InitSpatialMetaData()")
        self.db.execute("INSERT INTO spatial_ref_sys (srid, auth_name, auth_srid,"
                        " ref_sys_name, proj4text) VALUES (4326, 'epsg', 4326,"
                        " 'WGS 84', '+proj=longlat +ellps=WGS84 +datum=WGS84"
                        " +no_defs')")
        self.db.execute("CREATE TABLE poi (id int not null unique primary key,"
                        " name char, data text)")
        self.db.execute("SELECT AddGeometryColumn('poi', 'geom', 4326, 'POINT', 2)")
        self.db.execute("SELECT CreateSpatialIndex('poi', 'geom')")
        self.db.execute("CREATE TABLE tag (id integer primary key autoincrement,"
                        " name char, value char)")
        self.db.execute("CREATE UNIQUE INDEX tagidx ON tag (name, value)")
        self.db.execute("CREATE TABLE poitag (poi int not null, tag int not null)")
        self.db.execute("CREATE UNIQUE INDEX poitagidx ON poitag (poi, tag)")
        self.tagid_cache = dict()
    def tagid(self, k, v):
        key = (k, v)
        res = self.tagid_cache.get(key, None)
        if res is None:
            c = self.db.cursor()
            c.execute("SELECT id FROM tag WHERE name=? AND value=?", key)
            for row in c:
                self.tagid_cache[key] = row[0]
                return row[0]
            self.db.execute("INSERT INTO tag (id, name, value) VALUES (NULL, ?, ?)", key)
            c.execute("SELECT last_insert_rowid()")
            for row in c:
                res = row[0]
            self.tagid_cache[key] = res
        return res
    def __call__(self, lat, lon, id, tags, user=None, uid=None):
        # Acquire tag IDs
        tagids = []
        for k, v in tags.iteritems():
            if k not in self.TAG_WHITELIST: continue
            for val in v.split(";"):
                tagids.append(self.tagid(k, val))
        # Skip elements that don't have the tags we want
        if not tagids: return
        geom = "POINT(%f %f)" % (lon, lat)
        self.db.execute("INSERT INTO poi (id, geom, name, data)"
                        "     VALUES (?, GeomFromText(?, 4326), ?, ?)",
                (id, geom, tags["name"], simplejson.dumps(tags)))
        for tid in tagids:
            self.db.execute("INSERT INTO poitag (poi, tag) VALUES (?, ?)", (id, tid))
    def done(self):
        self.db.commit()
# Get the output file name
filename = sys.argv[1]
# Ensure we start from scratch
if os.path.exists(filename):
    print >>sys.stderr, filename, "already exists"
    sys.exit(1)
# Import
parser = xml.sax.make_parser()
importer = Importer(filename)
handler = OSMPOIReader(importer)
parser.setContentHandler(handler)
parser.parse(sys.stdin)
importer.done()
Let's run it: 
$ ./poiimport pois.db < pois.osm 
SpatiaLite version ..: 2.4.0    Supported Extensions:
        - 'VirtualShape'        [direct Shapefile access]
        - 'VirtualDbf'          [direct Dbf access]
        - 'VirtualText'         [direct CSV/TXT access]
        - 'VirtualNetwork'      [Dijkstra shortest path]
        - 'RTree'               [Spatial Index - R*Tree]
        - 'MbrCache'            [Spatial Index - MBR cache]
        - 'VirtualFDO'          [FDO-OGR interoperability]
        - 'SpatiaLite'          [Spatial SQL - OGC]
PROJ.4 Rel. 4.7.1, 23 September 2009
GEOS version 3.2.0-CAPI-1.6.0
$ ls -l --si pois*
-rw-r--r-- 1 enrico enrico 17M Jul  9 23:44 pois.db
-rw-r--r-- 1 enrico enrico 37M Jul  9 16:20 pois.osm
$ spatialite pois.db
SpatiaLite version ..: 2.4.0    Supported Extensions:
        - 'VirtualShape'        [direct Shapefile access]
        - 'VirtualDbf'          [direct DBF access]
        - 'VirtualText'         [direct CSV/TXT access]
        - 'VirtualNetwork'      [Dijkstra shortest path]
        - 'RTree'               [Spatial Index - R*Tree]
        - 'MbrCache'            [Spatial Index - MBR cache]
        - 'VirtualFDO'          [FDO-OGR interoperability]
        - 'SpatiaLite'          [Spatial SQL - OGC]
PROJ.4 version ......: Rel. 4.7.1, 23 September 2009
GEOS version ........: 3.2.0-CAPI-1.6.0
SQLite version ......: 3.6.23.1
Enter ".help" for instructions
spatialite> select id from tag where name="amenity" and value="fountain";
24
spatialite> SELECT poi.name, poi.data, X(poi.geom), Y(poi.geom) FROM poi, poitag WHERE poi.rowid IN (SELECT pkid FROM idx_poi_geom WHERE (xmin >= 2.56 AND xmax <= 2.90 AND ymin >= 41.84 AND ymax <= 42.00) ) AND poitag.tag = 24 AND poitag.poi = poi.id;
Font Picant de la Cellera "amenity": "fountain", "name": "Font Picant de la Cellera" 2.616045 41.952449
Font de Can Pla "amenity": "fountain", "name": "Font de Can Pla" 2.622354 41.974724
Font de Can Ribes "amenity": "fountain", "name": "Font de Can Ribes" 2.62311 41.979193
It's impressive: I've got all sort of useful information for the whole of Spain in just 17Mb! Let's put it to practice: I'm thirsty, is there any water fountain nearby? 
spatialite> SELECT count(1) FROM poi, poitag WHERE poi.rowid IN (SELECT pkid FROM idx_poi_geom WHERE (xmin >= 2.80 AND xmax <= 2.85 AND ymin >= 41.97 AND ymax <= 42.00) ) AND poitag.tag = 24 AND poitag.poi = poi.id;
0
Ouch! No water fountains mapped in Girona... yet. Problem 2 solved: now on to the next step, trying to show the results in some usable way.

19 May 2010

John Goerzen: Time to learn a new language

I have something of an informal goal of learning a new programming language every few years. It s not so much a goal as it is something of a discomfort. There are so many programming languages out there, with so many niches and approaches to problems, that I get uncomfortable with my lack of knowledge of some of them after awhile. This tends to happen every few years. The last major language I learned was Haskell, which I started working with in 2004. I still enjoy Haskell and don t see anything displacing it as my primary day-to-day workhorse. Yet there are some languages that I d like to learn. I have an interest in cross-platform languages; one of my few annoyances with Haskell is that it can t (at least with production quality) be compiled into something like Java bytecode or something else that isn t architecture-dependent. I have long had a soft spot for functional languages. I haven t had such a soft spot for static type checking, but Haskell s type inference changed that for me. Also I have an interest in writing Android apps, which means some sort of Java tie-in would be needed. Here are my current candidates: Of some particular interest to me is that Haskell has interpreters for Scheme, Lua, and JavaScript as well as code generators for some of these languages (though not generic Haskell-to-foo compilers). Languages not in the running because I already know them include: OCaml, POSIX shell, Python, Perl, Java, C, C++, Pascal, BASIC, Common Lisp, Prolog, SQL. Languages I have no interest in learning right now include Ruby (not different enough from what I already know plus bad experiences with it), any assembly, anything steeped in the Microsoft monoculture (C#, VB, etc.), or anything that is hard to work with outside of an Emacs or vim environment. (If your language requires or strongly encourages me to use your IDE or proprietary compiler, I m not interested that means you, flash.) Brief Reivews of Languages I Have Used To give you a bit of an idea of where I m coming from:

22 April 2010

Matt Brubeck: Headless Web Workers: Does the web need background apps?

At my last job, I created several web applications designed to replace built-in apps on mobile phones. While modern browsers and HTML5 made this incredibly easy in many ways, we still ended up writing native (i.e. non-web) code for most of our applications. There were a few different areas where the browser alone didn't meet our needs, but one that I found suprisingly common was background processing. Consider the following mobile applications: Ideally, each of these apps will perform some actions even when the user does not have it open. (Background processing is not strictly necessary for the e-reader, but it would be useful to ensure the library is up-to-date even when opened in a place with no network connection.) You can't do this with a web app. Web Workers don't solve the problem, because they run only while the web page is open. What we need are headless web workers.1 The API Headless workers could use almost the same API as Web Workers. Instead of responding to messages from a web page, they would listen to events from the host system (browser or OS). These events might include time intervals, power-on/resume, changes in network connection, geographic locations, or "push" notifications from a remote server. The event-driven architecture of JavaScript in the browser allows the host system a high level of discretion over resource consumption. There's no special code needed to suspend processes and later restore their state, because JavaScript workers are naturally inactive between events. The host can provide limits on CPU or memory usage per event, with a separate message to notify processes whose handlers were aborted. And it can limit the number of concurrent processes by choosing when to dispatch events to listeners. Some listeners could even be disabled completely at times (like if the device is busy or the battery is low), and notified later of the events they missed. This is almost a return to the old days of cooperative multitasking. Mobile computing is definitely driving everyone towards higher-level process control in the OS, and different assumptions for applications. It's not surprising that my whole proposal resembles Android and iPhone 4.0 multitasking in several ways, since I've been doing development on Android for the last 18 months and encountering many of the same issues. The UI Headless workers do need some way to interact with the user. They could display standard system notifications (via Growl on the Mac, libnotify on Ubuntu, the status bar in Android, etc.) using W3C Web Notifications, which already have an experimental implementation in Chrome. Users also needs to know which sites have background tasks installed. Headless workers could be represented by icons in a standard location (perhaps a toolbar in desktop browsers, or the home screen on a mobile device). The icons could display ambient status; clicking one would reveal a menu with options to configure or remove it. Questions This proposal might be hard to standardize, especially where it's tied to specific OS capabilities. For now I'm just curious: would it be useful? You can write a native app or a browser extension to solve this problem today. But would it be worthwhile to have a standard, cross-platform way to do it? Has anyone else run into problems that this approach could solve?
  1. Because all web standards should have names that sound like Harry Potter creatures.

3 March 2010

Christian Perrier: I'm going to DC10

Or should I say *we* are going to Debconf 10? Last week-end, I booked the whole set of plane tickets for the Perrier family to fly to USA next summer: Nice.

21 February 2010

Michael Banck: 21 Feb 2010

Application Indicators: A Case of Canonical Upstream Involvement and its Problems I always thought Canonical could do a bit more to contribute to the GNOME project, so I was happy to see the work on application indicators proposed to GNOME. Application indicators are based on the (originally KDE-driven, I believe) proposed cross-desktop status notifier spec. The idea (as I understand it) is to have a consistent way of interacting with status notifiers and stop the confusing mix of panel applets and systray indicators. This is a very laudable goal as mentioned by Colin Walters:
"First, +5000 that this change is being driven by designers, and +1000 that new useful code is being written. There are definite problems being solved here."
The discussion that followed was very useful, including the comments by Canonical's usability expert Matthew Paul Thomas. Most of the discussion was about the question how this proposal and spec could be best integrated into GTK, the place where most people seemed to agree this belongs (rather than changing all apps to provide this, this should be a service provided by the platform) However, on the same day, Canonical employee Ted Gould proposed libappindicator as an external dependency. The following thread showed a couple of problems, both technical and otherwise: What I personally disliked is the way the Cody Russel and Ted Gould are papering over the above issues in the thread that followed. For examples, about point one, Ted Gould writes in the proposal:
Q: Shouldn't this be in GTK+?
A: Apparently not.
while he himself said on the same day, on the same mailing list: "Yes, I think GTK/glib is a good place" and nobody was against it (and in fact most people seemed to favor including this in GTK). To the question about why libappindicator is not licensed as usual under the LGPL, version 2.1 or later, Canonical employee Cody Russell even replied:
"Because seriously, everything should be this way. None of us should be saying "LGPL 2.1 or later". Ask a lawyer, even one from the FSF, how much sense it makes to license your software that way."
Not everybody has to love the FSF, but proposing code under mandated copyright assignments which a lot of people have opposed and at the same time insinuating that the FSF was not to be trusted on their next revision of the LGPL license seems rather bold to me. Finally, on the topic of copyright assignments, Ted said:
"Like Clutter for example ;) Seriously though, GNOME already is dependent on projects that require contributor agreements."
It is true that there are (or at least were) GNOME applications which require copyright assignments for contributions (evolution used to be an example, but the requirement was lifted), however, none of the platform modules require this to my knowledge (clutter is an external dependency as well). It seems most people in the GNOME community have the opinion that application indicators should be in GTK at least eventually, so having libappindicator as an external dependency with copyright assignments might work for now but will not be future proof. In summary, Most of the issues could be dealt with by reimplementing it for GTK when the time comes for this spec to be included, but this would mean (i) duplication of effort, (ii) possibly porting all applications twice and (iii) probably no upstream contribution by Canonical. Furthermore, I am amazed at how the Canonical people approach the community for something this delicate (their first major code drop, as far as I am aware). To be fair, neither Ted nor Cody posted the above using their company email addresses, but nevertheless the work is sponsored by Canonical, so their posts to desktop-devel-list could be seen as writing with their Canonical hat on. Canonical does not have an outstanding track record on contributing code to GNOME, and at least to me it seems this case is not doing much to improve things, either.

Next.

Previous.