Search Results: "eddie"

30 March 2020

Axel Beckert: How do you type on a keyboard with only 46 or even 28 keys?

Some of you might have noticed that I m into keyboards since a few years ago into mechanical keyboards to be precise. Preface It basically started with the Swiss Mechanical Keyboard Meetup (whose website I started later on) was held in the hackerspace of the CCCZH. I mostly used TKL keyboards (i.e. keyboards with just the for me useless number block missing) and tried to get my hands on more keyboards with Trackpoints (but failed so far). At some point a year or two ago, I looking into smaller keyboards for having a mechanical keyboard with me when travelling. I first bought a Vortex Core at Candykeys. The size was nice and especially having all layers labelled on the keys was helpful, but nevertheless I soon noticed that the smaller the keyboards get, the more important is, that they re properly programmable. The Vortex Core is programmable, but not the keys in the bottom right corner which are exactly the keys I wanted to change to get a cursor block down there. (Later I found out that there are possibilities to get this done, either with an alternative firmware and a hack of it or desoldering all switches and mounting an alternative PCB called Atom47.) 40% Keyboards So at some point I ordered a MiniVan keyboard from The Van Keyboards (MiniVan keyboards will soon be available again at The Key Dot Company), here shown with GMK Paperwork (also bought from and designed by The Van Keyboards):
The MiniVan PCBs are fully programmable with the free and open source firmware QMK and started to use that more and more instead of bigger keyboards. Layers With the MiniVan I learned the concepts of layers. Layers are similar to what many laptop keyboards do with the Fn key and to some extent also what the German standard layout does with the AltGr key: Layers are basically alternative key maps you can switch with a special key (often called Fn , Fn1 , Fn2 , etc., or especially if there are two additional layers Raise and Lower ). There are several concepts how these layers can be reached with these keys: My MiniVan Layout For the MiniVan, two additional layers suffice easily, but since I have a few characters on multiple layers and also have mouse control and media keys crammed in there, I have three additional layers on my MiniVan keyboards:

TRNS means transparent, i.e. use the settings from lower layers.
I also use a feature that allows me to mind different actions to a key depending if I just tap the key or if I hold it. Some also call this tap dance . This is especially very popular on the usually rather huge spacebar. There, the term SpaceFn has been coined, probably after this discussion on Geekhack. I use this for all my layer switching keys: With this layout I can type English texts as fast as I can type them on a standard or TKL layout. German umlauts are a bit more difficult because it requires 4 to 6 key presses per umlaut as I use the Compose key functionality (mapped to the Menu key between the spacebars and the cursor block. So to type an on my MiniVan, I have to:
  1. press and release Menu (i.e. Compose); then
  2. press and hold either Shift-Spacebar (i.e. Shift-Fn1) or Slash (i.e. Fn2), then
  3. press N for a double quote (i.e. Shift-Fn1-N or Fn2-N) and then release all keys, and finally
  4. press and release the base character for the umlaut, in this case Shift-A.
And now just use these concepts and reduce the amount of keys to 28: 30% and Sub-30% Keyboards In late 2019 I stumbled upon a nice little keyboard kit shop on Etsy which I (and probably most other people in the mechanical keyboard scene) didn t take into account for looking for keyboards called WorldspawnsKeebs. They offer mostly kits for keyboards of 40% size and below, most of them rather simple and not expensive. For about 30 you get a complete sub-30% keyboard kit (without switches and keycaps though, but that very common for keyboard kits as it leaves the choice of switches and key caps to you) named Alpha28 consisting of a minimal Acrylic case and a PCB and electronics set. This Alpha28 keyboard is btw. fully open source as the source code, (i.e. design files) for the hardware are published under a free license (MIT license) on GitHub. And here s how my Alpha28 looks like with GMK Mitolet (part of the GMK Pulse group-buy) key caps:
So we only have character keys, Enter (labelled Data as there was no 1u Enter key with that row profile in that key cap set; I ll also call it Data for the rest of this posting) and a small spacebar, not even modifier keys. The Default Alpha28 Layout The original key layout by the developer of the Alpha28 used the spacbar as Shift on hold and as space if just tapped, and the Data key switches always to the next layer, i.e. it switches the layer permanently on tap and not just on hold. This way that key rotates through all layers. In all other layers, V switches back to the default layer. I assume that the modifiers on the second layer are also on tap and apply to the next other normal key. This has the advantage that you don t have to bend your fingers for some key combos, but you have to remember on which layer you are at the moment. (IIRC QMK allows you to show that via LEDs or similar.) Kinda just like vi. My Alpha28 Layout But maybe because I m more an Emacs person, I dislike remembering states myself and don t bind bending my fingers. So I decided to develop my own layout using tap-or-hold and only doing layer switches by holding down keys:

A triangle means that the settings from lower layers are used, N/A means the key does nothing.
It might not be very obvious, but on the default layer, all keys in the bottom row and most keys on the row ends have tap-or-hold configurations. Basic ideasBottom row if holdOther rows if holdHow the keys are divided into layersUsing the Alpha28 This layout works surprisingly well for me. Only for Minus, Equal, Single Quote and Semicolon I still often have to think or try if they re on Layer 1 or 2 as on my 40%s (MiniVan, Zlant, etc.) I have them all on layer 1 (and in general one layer less over all). And for really seldom used keys like Insert, PrintScreen, ScrollLock or Pause, I might have to consult my own documentation. They re somewhere in the middle of the keyboard, either on layer 1, 2, or 3. ;-) And of course, typing umlauts takes even two keys more per umlaut as on the MiniVan since on the one hand Menu is not on the default layer and on the other hand, I don t have this nice shifted number row and actually have to also press Shift to get a double quote. So to type an on my Alpha, I have to:
  1. press and release Space-F (i.e. Fn1-F) for Menu (i.e. Compose); then
  2. press and hold A-Spacebar-L (i.e. Shift-Fn1-L) for getting a double quote, then
  3. press and release the base character for the umlaut, in this case L-A for Shift-A (because we can t use A for Shift as I can t hold a key and then press it again :-).
Conclusion If the characters on upper layers are not labelled like on the Vortex Core, i.e. especially on all self-made layouts, typing is a bit like playing that old children s game Memory: as soon as you remember (or your muscle memory knows) where some special characters are, typing gets faster. Otherwise, you start with trial and error or look the documentation. Or give up. ;-) Nevertheless, typing on a sub-30% keyboard like the Alpha28 is much more difficult and slower than on a 40% keyboard like the MiniVan. So the Alpha28 very likely won t become my daily driver while the MiniVan defacto is my already my daily driver. But I like these kind of challenges as others like the game Memory . So I ordered three more 30% and sub-30% keyboard kits and WorldspawnsKeebs for soldering on the upcoming weekend during the COVID19 lockdown: And if I at some point want to try to type with even fewer keys, I ll try a Butterstick keyboard with just 20 keys. It s a chorded keyboard where you have to press multiple keys at the same time to get one charcter: So to get an A from the missing middle row, you have to press Q and Z simultaneously, to get Escape, press Q and W simultaneously, to get Control, press Q, W, Z and X simultaneously, etc. And if that s not even enough, I already bought a keyboard kit named Ginny (or Ginni, the developer can t seem to decide) with just 10 keys from an acquaintance. Couldn t resist when offered his surplus kits. :-) It uses the ASETNIOP layout which was initially developed for on-screen keyboards on tablets.

5 December 2016

Shirish Agarwal: The Anti-Pollito squad arrest and confession

Disclaimer This is an attempt at humor and hence entirely fictional in nature. While some incidents depicted are true, the context and the story woven around them are by yours truly. None of the Mascots of Debian were hurt during the blog post . I also disavow any responsibility for any hurt (real or imagined) to any past, current and future mascots. The attempt should not be looked upon as demeaning people who are accused of false crimes, tortured and confessions eked out of them as this happens quite a lot (In India for sure, but guess it s the same world over in various degrees). The idea is loosely inspired by Chocolate:Deep Dark Secrets. (2005) On a more positive note, let s start Being a Sunday morning woke up late to find incessant knocking on the door, incidentally mum was not at home. Opening the door, found two official looking gentleman. They asked my name, asked my credentials, tortured and arrested me for Group conspiracy of Malicious Mischief in second and third degrees . The torture was done by means of making me forcefully watch endless reruns of Norbit . While I do love Eddie Murphy, this was one of his movies he could have done without . I guess for many people watching it once was torture enough. I *think* they were nominated for razzie awards dunno if they won it or not, but this is beside the point. Unlike the 20 years it takes for a typical case to reach to its conclusion even in the smallest court in India, due to the torture, I was made to confess (due to endless torture) and was given summary judgement. The judgement was/is as follows a. Do 100 hours of Community service in Debian in 2017. This could be done via blog posts, raising tickets in the Debian BTS or in whichever way I could be helpful to Debian. b. Write a confessional with some photographic evidence sharing/detailing some of the other members who were part of the conspiracy in view of the reduced sentence. So now, have been forced to write this confession As you all know, I won a bursary this year for debconf16. What is not known by most people is that I also got an innocuous looking e-mail titled Pollito for DPL . While I can t name all the names as investigation is still ongoing about how far-reaching the conspiracy is . The email was purportedly written by members of cabal within cabal which are in Debian. I looked at the email header to see if this was genuine and I could trace the origin but was left none the wiser, as obviously these people are far more technically advanced than to fall in simple tricks like this Anyways, secretly happy that I have been invited to be part of these elites, I did the visa thing, packed my bags and came to Debconf16. At this point in juncture, I had no idea whether it was real or I had imagined the whole thing. Then to my surprise saw this evidence of conspiracy to have Pollito as DPL, Wifi Password Just like the Illuminati the conspiracy was for all to see those who knew about it. Most people were thinking of it as a joke, but those like me who had got e-mails knew better. I knew that the thing is real, now I only needed to bide my time and knew that the opportunity would present itself. And few days later, sure enough, there was a trip planned for Table Mountain, Cape Town . Few people planned to hike to the mountain, while few chose to take the cable car till up the mountain. First glance of the cable car with table mountain as background Quite a few people came along with us and bought tickets for the to and fro to the mountain and back. Ticket for CPT Table mountain car cable Incidentally, I was thinking if the South African Govt. were getting the tax or not. If you look at the ticket, there is just a bar-code. In India as well as the U.S. there is TIN Tax Identification Number TIN displayed on an invoice from channeltimes.com Few links to share what it is all about . While these should be on all invoices, need to specially check when taking high-value items. In India as shared in the article the awareness, knowledge leaves a bit to be desired. While I m drifting from the incident, it would be nice if somebody from SA could share how things work there. Moving on, we boarded the cable car. It was quite spacious cable car with I guess around 30-40 people or some more who were able to see everything along with the controller. from inside the table mountain cable car 360 degrees It was a pleasant cacophony of almost two dozen or more nationalities on this 360 degrees moving chamber. I was a little worried though as it essentially is a bucket and there is always a possibility that a severe wind could damage it. Later somebody did share that some frightful incidents had occurred not too long ago on the cable car. It took about 20-25 odd minutes to get to the top of table mountain and we were presented with views such as below View from Table Mountain cable car looking down The picture I am sharing is actually when we were going down as all the pictures of going up via the cable car were over-exposed. Also, it was pretty crowded on the way up then on the way down so handling the mobile camera was not so comfortable. Once we reached up, the wind was blowing at incredible speeds. Even with my jacket and everything I was feeling cold. Most of the group around 10-12 people looked around if we could find a place to have some refreshments and get some of the energy in the body. So we all ventured to a place and placed our orders the bleh... Irish coffee at top of Table Mountain I was introduced to Irish Coffee few years back and have had some incredible Irish Coffees in Pune and elsewhere. I do hope to be able to make Irish Coffee at home if and when I have my own house. This is hotter than brandy and is perfect if you are suffering from cold etc if done right, really needs some skills. This is the only drink which I wanted in SA which I never got right . As South Africa was freezing for me, this would have been the perfect antidote but the one there as well as elsewhere were all bleh. What was interesting though, was the coffee caller besides it. It looked like a simple circuit mounted on a PCB board with lights, vibrations and RFID and it worked exactly like that. I am guessing as and when the order is ready, there is an interrupt signal sent via radio waves which causes the buzzer to light and vibrate. Here s the back panel if somebody wants to take inspiration and try it as a fun project backpanel of the buzz caller Once we were somewhat strengthened by the snacks, chai, coffee etc. we made our move to seeing the mountain. The only way to describe it is that it s similar to Raigad Fort but the plateau seemed to be bigger. The wikipedia page of Table Mountain attempts to share but I guess it s more clearly envisioned by one of the pictures shared therein. table mountain panaromic image I have to say while Table Mountain is beautiful and haunting as it has scenes like these Some of the oldest rocks known to wo/man. There is something there which pulls you, which reminds you of a long lost past. I could have simply sat there for hours together but as was part of the group had to keep with them. Not that I minded. The moment I was watching this, I was transported to some memories of the Himalayas about 20 odd years or so. In that previous life, I had the opportunity to be with some of the most beautiful women and also been in the most happening places, the Himalayas. I had shared years before some of my experiences I had in the Himalayas. I discontinued it as I didn t have a decent camera at that point in time. While I don t wanna digress, I would challenge anybody to experience the Himalayas and then compare. It is just something inexplicable. The beauty and the rawness that Himalayas shows makes you feel insignificant and yet part of the whole cosmos. What Paulo Cohello expressed in The Valkyries is something that could be felt in the Himalayas. Leh, Ladakh, Himachal , Garwhal, Kumaon. The list will go on forever as there are so many places, each more beautiful than the other. Most places are also extremely backpacker-friendly so if you ask around you can get some awesome deals if you want to spend more than a few days in one place. Moving on, while making small talk @olasd or Nicolas Dandrimont , the headmaster of our trip made small talk to each of us and eked out from all of us that we wanted to have Pollito as our DPL (Debian Project Leader) for 2017. Few pictures being shared below as supporting evidence as well The Pollito as DPL cabal in action members of the Pollito as DPL where am I or more precisely how far am I from India. While I do not know who further up than Nicolas was on the coup which would take place. The idea was this If the current DPL steps down, we would take all and any necessary actions to make Pollito our DPL. Pollito going to SA - photo taken by Jonathan Carter This has been taken from Pollito s adventure Being a responsible journalist, I also enquired about Pollito s true history as it would not have been complete without one. This is the e-mail I got from Gunnar Wolf, a friend and DD from Mexico
Turns out, Valessio has just spent a week staying at my house And
in any case, if somebody in Debian knows about Pollito s
childhood That is me. Pollito came to our lives when we went to Congreso Internacional de
Software Libre (CISOL) in Zacatecas city. I was strolling around the
very beautiful city with my wife Regina and our friend Alejandro
Miranda, and at a shop at either Ram n L pez Velarde or Vicente
Guerrero, we found a flock of pollitos. http://www.openstreetmap.org/#map=17/22.77111/-102.57145 Even if this was comparable to a slave market, we bought one from
them, and adopted it as our own. Back then, we were a young couple Well, we were not that young
anymore. I mean, we didn t have children. Anyway, we took Pollito with
us on several road trips, such as the only time I have crossed an
international border driving: We went to Encuentro Centroamericano de
Software Libre at Guatemala city in 2012 (again with Alejandro), and
you can see several Pollito pics at: http://gwolf.org/album/road-trip-ecsl-2012-guatemala-0 Pollito likes travelling. Of course, when we were to Nicaragua for
DebConf, Pollito tagged along. It was his first flight as a passenger
(we never asked about his previous life in slavery; remember, Pollito
trust no one). Pollito felt much welcome with the DebConf crowd. Of course, as
Pollito is a free spirit, we never even thought about forcing him to
come back with us. Pollito went to Switzerland, and we agreed to meet
again every year or two. It s always nice to have a chat with him. Hugs!
So with that backdrop I would urge fellow Debianities to take up the slogans LONG LIVE THE DPL ! LONG LIVE POLLITO ! LONG LIVE POLLITO THE DPL ! The first step to make Pollito the DPL is to ensure he has a @debian.org (pollito@debian.org) We also need him to be made a DD because only then can he become a DPL. In solidarity and in peace
Filed under: Miscellenous Tagged: #caller, #confession, #Debconf16, #debian, #Fiction, #history, #Pollito, #Pollito as DPL, #Table Mountain, Cabal, memories, south africa

2 January 2016

Daniel Pocock: The great life of Ian Murdock and police brutality in context

Tributes: (You can Follow or Tweet about this blog on Twitter) Over the last week, people have been saying a lot about the wonderful life of Ian Murdock and his contributions to Debian and the world of free software. According to one news site, a San Francisco police officer, Grace Gatpandan, has been doing the opposite, starting a PR spin operation, leaking snippets of information about what may have happened during Ian's final 24 hours. Sadly, these things are now starting to be regurgitated without proper scrutiny by the mainstream press (note the erroneous reference to SFGate with link to SFBay.ca, this is British tabloid media at its best). The report talks about somebody (no suggestion that it was even Ian) "trying to break into a residence". Let's translate that from the spin-doctor-speak back to English: it is the silly season, when many people have a couple of extra drinks and do silly things like losing their keys. "a residence", or just their own home perhaps? Maybe some AirBNB guest arriving late to the irritation of annoyed neighbours? Doesn't the choice of words make the motive sound so much more sinister? Nobody knows the full story and nobody knows if this was Ian, so snippets of information like this are inappropriate, especially when somebody is deceased. Did they really mean to leave people with the impression that one of the greatest visionaries of the Linux world was also a cat burglar? That somebody who spent his life giving selflessly and generously for the benefit of the whole world (his legacy is far greater than Steve Jobs, as Debian comes with no strings attached) spends the Christmas weekend taking things from other people's houses in the dark of the night? The report doesn't mention any evidence of a break-in or any charges for breaking-in. If having a few drinks and losing your keys in December is such a sorry state to be in, many of us could potentially be framed in the same terms at some point in our lives. That is one of the reasons I feel so compelled to write this: somebody else could be going through exactly the same experience at the moment you are reading this. Any of us could end up facing an assault as unpleasant as the tweets imply at some point in the future. At least I can console myself that as a privileged white male, the risk to myself is much lower than for those with mental illness, the homeless, transgender, Muslim or black people but as the tweets suggest, it could be any of us. The story reports that officers didn't actually come across Ian breaking in to anything, they encountered him at a nearby street corner. If he had weapons or drugs or he was known to police that would have almost certainly been emphasized. Is it right to rush in and deprive somebody of their liberties without first giving them an opportunity to identify themselves and possibly confirm if they had a reason to be there? The report goes on, "he was belligerent", "he became violent", "banging his head" all by himself. How often do you see intelligent and successful people like Ian Murdock spontaneously harming themselves in that way? Can you find anything like that in any of the 4,390 Ian Murdock videos on YouTube? How much more frequently do you see reports that somebody "banged their head", all by themselves of course, during some encounter with law enforcement? Do police never make mistakes like other human beings? If any person was genuinely trying to spontaneously inflict a head injury on himself, as the police have suggested, why wouldn't the police leave them in the hospital or other suitable care? Do they really think that when people are displaying signs of self-harm, rounding them up and taking them to jail will be in their best interests? Now, I'm not suggesting this started out with some sort of conspiracy. Police may have been at the end of a long shift (and it is a disgrace that many US police are not paid for their overtime) or just had a rough experience with somebody far more sinister. On the other hand, there may have been a mistake, gaps in police training or an inappropriate use of a procedure that is not always justified, like a strip search, that causes profound suffering for many victims. A select number of US police forces have been shamed around the world for a series of incidents of extreme violence in recent times, including the death of Michael Brown in Ferguson, shooting Walter Scott in the back, death of Freddie Gray in Baltimore and the attempts of Chicago's police to run an on-shore version of Guantanamo Bay. Beyond those highly violent incidents, the world has also seen the abuse of Ahmed Mohamed, the Muslim schoolboy arrested for his interest in electronics and in 2013, the suicide of Aaron Swartz which appears to be a direct consequence of the "Justice" department's obsession with him. What have the police learned from all this bad publicity? Are they changing their methods, or just hiring more spin doctors? If that is their response, then doesn't it leave them with a cruel advantage over those people who were deceased? Isn't it standard practice for some police to simply round up anybody who is a bit lost and write up a charge sheet for resisting arrest or assaulting an officer as insurance against questions about their own excessive use of force? When British police executed Jean Charles de Menezes on a crowded tube train and realized they had just done something incredibly outrageous, their PR office went to great lengths to try and protect their image, even photoshopping images of Menezes to make him look more like some other suspect in a wanted poster. To this day, they continue to refer to Menezes as a victim of the terrorists, could they be any more arrogant? While nobody believes the police woke up that morning thinking "let's kill some random guy on the tube", it is clear they made a mistake and like many people (not just police), they immediately prioritized protecting their reputation over protecting the truth. Nobody else knows exactly what Ian was doing and exactly what the police did to him. We may never know. However, any disparaging or irrelevant comments from the police should be viewed with some caution. The horrors of incarceration It would be hard for any of us to understand everything that an innocent person goes through when detained by the police. The recently released movie about The Stanford Prison Experiment may be an interesting place to start, a German version produced in 2001, Das Experiment, is also very highly respected. The United States has the largest prison population in the world and the second-highest per-capita incarceration rate. Many, including some on death row, are actually innocent, in the wrong place at the wrong time, without the funds to hire an attorney. The system, and the police and prison officers who operate it, treat these people as packages on a conveyor belt, without even the most basic human dignity. Whether their encounter lasts for just a few hours or decades, is it any surprise that something dies inside them when they discover this cruel side of American society? Worldwide, there is an increasing trend to make incarceration as degrading as possible. People may be innocent until proven guilty, but this hasn't stopped police in the UK from locking up and strip-searching over 4,500 children in a five year period, would these children go away feeling any different than if they had an encounter with Jimmy Saville or Rolf Harris? One can only wonder what they do to adults. What all this boils down to is that people shouldn't really be incarcerated unless it is clear the danger they pose to society is greater than the danger they may face in a prison. What can people do for Ian and for justice? Now that these unfortunate smears have appeared, it would be great to try and fill the Internet with stories of the great things Ian has done for the world. Write whatever you feel about Ian's work and your own experience of Debian. While the circumstances of the final tweets from his Twitter account are confusing, the tweets appear to be consistent with many other complaints about US law enforcement. Are there positive things that people can do in their community to help reduce the harm? Sending books to prisoners (the UK tried to ban this) can make a difference. Treat them like humans, even if the system doesn't. Recording incidents of police activities can also make a huge difference, such as the video of the shooting of Walter Scott or the UK police making a brutal unprovoked attack on a newspaper vendor. Don't just walk past a situation and assume everything is under control. People making recordings may find themselves in danger, it is recommended to use software that automatically duplicates each recording, preferably to the cloud, so that if the police ask you to delete such evidence, you can let them watch you delete it and still have a copy. Can anybody think of awards that Ian Murdock should be nominated for, either in free software, computing or engineering in general? Some, like the prestigious Queen Elizabeth Prize for Engineering can't be awarded posthumously but others may be within reach. Come and share your ideas on the debian-project mailing list, there are already some here. Best of all, Ian didn't just build software, he built an organization, Debian. Debian's principles have helped to unite many people from otherwise different backgrounds and carry on those principles even when Ian is no longer among us. Find out more, install it on your computer or even look for ways to participate in the project.

21 April 2011

Alastair McKinstry: How much wind energy can we use?

So how much wind energy can we realistically use? A paper (Earth System Dynamics, DOI: 10.5194/esd-2-1-2011) by Miller,Gans and Kleidon (MGK10) has generated a lot of controversy courtesy of a New Scientist review (See Climate Progress for a good rebuttal). They argue that the effective limit we can safely extract without (bad) climatic consequences is around 18--68 TW (we currently use 0.2 TW of wind power). This paper is a follow-on to Kleidons work on the thermodynamics of the climate, where he points out that the free energy is the important measure of what energy can be generated by renewables. The free energy of the atmosphere is on the scale of hundreds of TW, he argues, much less than the heat energy of ~10^5 TW that comes from the sun. He argues that not enough attention is paid to the free energy budget, and argues that from MEP theory that free energy flows through different geophysical processes are more important than people currently think: such energy and momentum flows then lead to faster depletion of gradients than people currently use in their models. Frequently we use "diffusive-like" approximations to stuff we can't explicitly handle,such as convection and eddies below the resolution of our computer models. That the free energy budget, and small-scale processes need more detailed study (such as the transfer of momentum and energy from wind to water), I agree. But paradoxically, MGK10's estimates based on diffusive processes are what gets it into trouble. The critiques (e.g. Jacobsen and Archer) seem to concentrate on the estimates for diffusive and other energy losses (eg. wake fields) and climatic consequences are very loose, and out by factors of 50-100. They also ignore the fact that using wind as a substitute for more destructive energy sources produces a net gain, climate-wise. Based on these numbers, the real limits on wind power are more limited by where we can realistically place turbines, and the use and availability of rare earth metals in turbine magnets (MW sized turbines use over a ton of neodynium each in magnets!) One of the focuses of research at ICHEC and Met Eireann is wind for wind power: this becomes increasingly important for Ireland as we have more energy production by wind. I've been recently involved in a European project Weather Intelligence for Renewable Energies on investigating weather and climate forecasting needs of wind energy. In our weather models, we are getting to finer and finer resolutions (down to 2-5 km operationally, but working on 1 km in research), but as yet do not take account of wind energy changes. Modelling the wake turbulence, etc. and other effects of turbines is a 'fluid dynamics' problem done by the wind turbine manufacturers, mostly, in order to lay out turbines in a farm efficiently; this is done on the timescale of metres and seconds, while on the other end of the spectrum work is done in climate research on the changes in potential winds to see if wind energy will change in a few decades. But little is done on 'weather' scales of kilometres, minutes and hours: it turns out there are wind patterns that we don't predict in our models (vortices, etc.; especially for offshore turbines) that affect wind energy production, but would be very useful for wind farm and electrical grid operators to know about. So we should think about adapting our weather models for these. Either way, it looks like more detailed work on the interactions between atmospheric dynamics and wind farms would be a good thing. Incidentally, whatever about the alarmist nature of the article, kudos to New Scientist for their linking to papers: they quoted the names of the journals (Earth System Dynamics,Philosophical Transactions of the Royal Society), but where the article was not yet published (or behind a paywall), linked to the Arxiv.org version. More could do this. Tags , , , , ,

11 July 2010

Enrico Zini: Computing time offsets between EXIF and GPS

Computing time offsets between EXIF and GPS I like the idea of matching photos to GPS traces. In Debian there is gpscorrelate but it's almost unusable to me because of bug #473362 and it has an awkward way of specifying time offsets. Here at SoTM10 someone told me that exiftool gained -geosync and -geotag options. So it's just a matter of creating a little tool that shows a photo and asks you to type the GPS time you see in it. Apparently there are no bindings or GIR files for gtkimageview in Debian, so I'll have to use C. Here is a C prototype:
/*
 * gpsoffset - Compute EXIF time offset from a photo of a gps display
 *
 * Use with exiftool -geosync=... -geotag trace.gpx DIR
 *
 * Copyright (C) 2009--2010  Enrico Zini <enrico@enricozini.org>
 *
 * This program is free software; you can redistribute it and/or modify
 * it under the terms of the GNU General Public License as published by
 * the Free Software Foundation; either version 2 of the License, or
 * (at your option) any later version.
 *
 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 * GNU General Public License for more details.
 *
 * You should have received a copy of the GNU General Public License
 * along with this program; if not, write to the Free Software
 * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
 */
#define _XOPEN_SOURCE /* glibc2 needs this */
#include <time.h>
#include <gtkimageview/gtkimageview.h>
#include <libexif/exif-data.h>
#include <stdio.h>
#include <stdlib.h>
static int load_time(const char* fname, struct tm* tm)
 
    ExifData* exif_data = exif_data_new_from_file(fname);
    ExifEntry* exif_time = exif_data_get_entry(exif_data, EXIF_TAG_DATE_TIME);
    if (exif_time == NULL)
     
        fprintf(stderr, "Cannot find EXIF timetamp\n");
        return -1;
     
    char buf[1024];
    exif_entry_get_value(exif_time, buf, 1024);
    //printf("val2: %s\n", exif_entry_get_value(t2, buf, 1024));
    if (strptime(buf, "%Y:%m:%d %H:%M:%S", tm) == NULL)
     
        fprintf(stderr, "Cannot match EXIF timetamp\n");
        return -1;
     
    return 0;
 
static time_t exif_ts;
static GtkWidget* res_lbl;
void date_entry_changed(GtkEditable *editable, gpointer user_data)
 
    const gchar* text = gtk_entry_get_text(GTK_ENTRY(editable));
    struct tm parsed;
    if (strptime(text, "%Y-%m-%d %H:%M:%S", &parsed) == NULL)
     
        gtk_label_set_text(GTK_LABEL(res_lbl), "Please enter a date as YYYY-MM-DD HH:MM:SS");
      else  
        time_t img_ts = mktime(&parsed);
        int c;
        int res;
        if (exif_ts < img_ts)
         
            c = '+';
            res = img_ts - exif_ts;
         
        else
         
            c = '-';
            res = exif_ts - img_ts;
         
        char buf[1024];
        if (res > 3600)
            snprintf(buf, 1024, "Result: %c%ds -geosync=%c%d:%02d:%02d",
                    c, res, c, res / 3600, (res / 60) % 60, res % 60);
        else if (res > 60)
            snprintf(buf, 1024, "Result: %c%ds -geosync=%c%02d:%02d",
                    c, res, c, (res / 60) % 60, res % 60);
        else
            snprintf(buf, 1024, "Result: %c%ds -geosync=%c%d",
                    c, res, c, res);
        gtk_label_set_text(GTK_LABEL(res_lbl), buf);
     
 
int main (int argc, char *argv[])
 
    // Work in UTC to avoid mktime applying DST or timezones
    setenv("TZ", "UTC");
    const char* filename = "/home/enrico/web-eddie/galleries/2010/04-05-Uppermill/P1080932.jpg";
    gtk_init (&argc, &argv);
    struct tm exif_time;
    if (load_time(filename, &exif_time) != 0)
        return 1;
    printf("EXIF time: %s\n", asctime(&exif_time));
    exif_ts = mktime(&exif_time);
    GtkWidget* window = gtk_window_new(GTK_WINDOW_TOPLEVEL);
    GtkWidget* vb = gtk_vbox_new(FALSE, 0);
    GtkWidget* hb = gtk_hbox_new(FALSE, 0);
    GtkWidget* lbl = gtk_label_new("Timestamp:");
    GtkWidget* exif_lbl;
     
        char buf[1024];
        strftime(buf, 1024, "EXIF time: %Y-%m-%d %H:%M:%S", &exif_time);
        exif_lbl = gtk_label_new(buf);
     
    GtkWidget* date_ent = gtk_entry_new();
    res_lbl = gtk_label_new("Result:");
    GtkWidget* view = gtk_image_view_new();
    GdkPixbuf* pixbuf = gdk_pixbuf_new_from_file(filename, NULL);
    gtk_box_pack_start(GTK_BOX(hb), lbl, FALSE, TRUE, 0);
    gtk_box_pack_start(GTK_BOX(hb), date_ent, TRUE, TRUE, 0);
    gtk_signal_connect(GTK_OBJECT(date_ent), "changed", (GCallback)date_entry_changed, NULL);
     
        char buf[1024];
        strftime(buf, 1024, "%Y-%m-%d %H:%M:%S", &exif_time);
        gtk_entry_set_text(GTK_ENTRY(date_ent), buf);
     
    gtk_widget_set_size_request(view, 500, 400);
    gtk_image_view_set_pixbuf(GTK_IMAGE_VIEW(view), pixbuf, TRUE);
    gtk_container_add(GTK_CONTAINER(window), vb);
    gtk_box_pack_start(GTK_BOX(vb), view, TRUE, TRUE, 0);
    gtk_box_pack_start(GTK_BOX(vb), hb, FALSE, TRUE, 0);
    gtk_box_pack_start(GTK_BOX(vb), exif_lbl, FALSE, TRUE, 0);
    gtk_box_pack_start(GTK_BOX(vb), res_lbl, FALSE, TRUE, 0);
    gtk_widget_show_all(window);
    gtk_main ();
    return 0;
 
And here is its simple makefile:
CFLAGS=$(shell pkg-config --cflags gtkimageview libexif)
LDFLAGS=$(shell pkg-config --libs gtkimageview libexif)
gpsoffset: gpsoffset.c
It's a simple prototype but it's a working prototype and seems to do the job for me. I currently cannot find out why after I click on the text box, there seems to be no way to give the focus back to the image viewer so I can control it with keys. There is another nice algorithm to compute time offsets to be implemented: you choose a photo taken from a known place and drag it on that place on a map: you can then look for the nearest point on your GPX trace and compute the time offset from that. I have seen that there are programs for geotagging photos that implement all such algorithms, and have a nice UI, but I haven't seen any in Debian. Are there any such softwares that can be packaged? If not, the interpolation and annotation tasks can now already be performed by exiftool, so it's just a matter of building a good UI, and I would love to see someone picking up the task.

25 January 2009

David Pashley: Subversion and "(502 Bad Gateway) in response to COPY request" errors

Was attempting to merge a branch in one of my projects and upon committing the merge, I kept getting this error:
mojo-jojo david% svn commit -m "merge in the maven branch"
Sending        trunk
Sending        trunk/.classpath
Sending        trunk/.project
Adding         trunk/.settings
svn: Commit failed (details follow):
svn: Server sent unexpected return value (502 Bad Gateway) in response
to COPY request for '/svn/eddie/!svn/bc/314/branches/maven/.settings'
A quick search found several other people having the same problem. Seems it only happens for https repositories using mod_dav_svn. The solution is to make sure that your virtual host in apache has explicit SSL config options, even if you are using an SSL config from a default virtual host. For example, I added the following to my subversion vhost, which was just copied from my default vhost:
SSLEngine on
SSLCertificateFile /etc/apache2/ssl/catnip.org.uk.crt
SSLCertificateKeyFile /etc/apache2/ssl/catnip.org.uk.key
Read Comments (0)

23 July 2008

Gunnar Wolf: Virtually having fun

Several weeks ago, the people in charge of maintaining the Windows machines in my institute were desperate because of a series of virus outbreaks - Specially, as expected, in the public lab - but the whole network smell virulent. After seeing their desperation, I asked Rolman to help me come up with a solution. He suggested me to try replacing the Windows workstations by substituting local installations by a server having several virtual machines, all regenerated from a clean image every day, and exporting rdesktop sessions. He suggested using Xen for this, as it is the virtualization/paravirtualization solution until now best offered and supported by most Linux distributions (including, of course, RedHat, towards which he is biased, and Debian, towards I am... more than biased, even bent). So far, no hassle, right?
Of course, I could just stay clear of this mess, as everything related to Windows is off my hands... But in October, we will be renewing ~150 antivirus licences. I want to save that money by giving a better solution, even if part of that money gets translated to a big server.
Get the hardware
But problems soon arose. The first issue was hardware. Xen can act in its paravirtualization mode on basically any x86 machine - but it requires a patched guest kernel. That means, I can paravitualize many several different free OSs on just any computer I lay my hands on here, but Windows requires full- or hardware-assisted- virtualization. And, of course, only one of the over 300 computers we have (around 100 of which are recent enough for me to expect to be usable as a proof-of-concept for this) has a CPU with VT extensions - And I'm not going to de-comission my firewall to become a test server! ;-)
When software gets confused for hardware
So, I requested a Intel Core 2 Quad Q9300 CPU, which I could just drop in any box with a fitting motherboard. But, of course, I'm not the only person requiring computer-related stuff. So, after pestering the people in charge for buying stuff on a daily basis for three weeks, the head of acquisitions came smiling to my office with a little box in his hands.
But no, it was not my Core 2 Quad CPU.
It was a box containing... Microsoft Visio. Yes, they spent their effort looking for the wrong computer-related thingy :-/ And meanwhile, Debconf 8 is getting nearer and nearer. Why does that matter? Because I have a deadline: By October, I want the institute to decide not to buy 150 antivirus licenses! Debconf will take some time off that target from me.
Anyway... The university vacations started on July 5. The first week of vacations I went to sweat my ass off at Monterrey, by Monday 14 I came back to my office, and that same day I finally got the box, together with two 2GB DIMMs.
Experiences with a nice looking potential disaster
Anyway, by Tuesday I got the CPU running, and a regular Debian install in place. A very nice workhorse: 5GB RAM, quad core CPU at 2.5GHz, 6MB cache (which seems to be split in two 3MB banks, each for two cores - but that's pure speculation from me). I installed Lenny (Debian testing), which is very soon going to freeze and by the time this becomes a production server will be very close to being a stable release, and I wanted to take advantage of the newest Xen administration tools. Of course, the installation was for AMD64 - Because 64 bitness is a terrible thing to waste.
But I started playing with Xen - And all kind of disasters stroke. First, although there is a Xen-enabled 2.6.25 Linux kernel, it is -686 only (i.e. no 64 bit support). Ok, install a second system on a second partition. Oh, but this kernel is only domU-able (this is, it will correctly run in a Xen paravirtualized host), but not dom0-able (it cannot act as a root domain). Grmbl.
So, get Etch's 2.6.18 AMD64 Xen-enabled kernel, and hope for the best. After all, up to this point, I was basically aware of many of the facts I mentioned (i.e. up to this point I did reinstall once, but not three times)... And I hoped the kernel team would have good news regarding a forward-port of the Xen dom0 patches to 2.6.25 - because losing dom0 support was IMO a big regression.
But quite on time, this revealing thread came up on the debian-devel mailing list. In short: Xen is a disaster. The Xen developers have done their work quite far away from the kernel developers, and the last decent synchronization that was made was in 2.6.18, over two years ago. Not surprisingly, enterprise-editions of other Linux distributions also ship that kernel version. There are some forward-patches, but current support in Xen is... Lacking, to say the least. From my POV, Xen's future in the Linux kernel looks bleakish.
Now, on the lightweight side...
Xen is also a bit too complicated - Of course, its role is also complicated as well, and it has a great deal of tunability. But I decided to keep a clean Lenny AMD64 install, and give KVM, the Kernel Virtual Machine a go. My first gripe? What a bad choice of name. Not only Google searches for KVM gives completely unrelated answers (to a name that's already well known, even in the same context, even in the same community).
KVM takes a much, much simpler approach to virtualization (both para- and full-): We don't need no stinkin' hypervisors. The kernel can just do that task. And then, kvm becomes just another almost-regular process. How nice!
In fact, KVM borrows so very much from qemu that it even refers to qemu's manpage for everything but two command-line switches.
Qemu is a completely different project, which gets to a very similar place but from the other extreme - Qemu started off as Bochs, a very slow but very useful multi-architecture emulator. Qemu started adding all kinds of optimizations, and it is nearly useful (i.e. I use it in my desktop whenever I need a W2K machine).
Instead of a heavyweight framework... KVM is just a modprobe away - Just ask Linux to modprobe kvm, and kvm -hda /path/to/your/hd/image gets you a working machine.
Anyway - I was immediatly happy with KVM. It took me a week to get a whole "lab" of 15 virtual computers (256MB RAM works surprisingly well for a regular XP install!) configured to start at boot time off a single master image over qcow images.
KVM's shortcomings
Xen has already been a long time in the enterprise, and has a nice suite of administrative tools. While Xen depends on having a configuration file for each host, KVM expects them to be passed at the command line. To get a bird-eye view of the system, xen has a load of utilities - KVM does not. And although RedHat's virt-manager is said to support KVM and qemu virtualization (besides its native Xen, of course), it falls short of what I need (i.e. it relies on a configuration file... which lacks expresivity to specify a snapshot-based HD image).
To my surprise, KVM has attained much of Xen's most amazing capabilities, such as the live migration. And although it's easier to just use fully virtualized devices (i.e. to use an emulation of the RTL8139 network card), as they require no drivers extraneous to the operating system, performance can be greatly enhanced by using the VirtIO devices. KVM is quickly evolving, and I predict it will largely overtake Xen's (and of course, vmware and others) places.
Where I am now
So... Well, those of us that adopt KVM and want to get it into production now will have some work of building the tools to gracefully manage and report it, it seems. I won't be touching much my setup until after Debconf, but so far I've done some work over Freddie Cash's kvmctl script. I'm submitting him some patches to make his script (IMHO) more reliable and automatizable (if you are interested, you can get my current version of the script as well). And... Starting September, I expect to start working on a control interface able to cover my other needs (such as distributing configuration to the terminals-to-be, or centrally managing the configurations).

13 April 2008

Clint Adams: They will hear me say as the pavement whirls

It was just like Christmas. I finished urinating on the Mother Church, then we headed over to Fenway for to do some trespassing. Snowflakes fell and swirled in the breeze, their eddies making that ugly fluorescent light seem pretty, so pretty. The crash of duckpins resounded through the night, and hipsters flashed jazz hands at one another. It was just like Christmas. Not one insalata 3ndy could be found.

1 February 2008

Romain Francoise: Recent movies (5)

(Note: this post had been sitting in my drafts folder for some time so some of these are not so recent. I thought I'd mention them anyway.)

20 January 2008

Clint Adams: They will hear me say as the pavement whirls

It was just like Christmas. I finished urinating on the Mother Church, then we headed over to Fenway for to do some trespassing. Snowflakes fell and swirled in the breeze, their eddies making that ugly fluorescent light seem pretty, so pretty. The crash of duckpins resounded through the night, and hipsters flashed jazz hands at one another. It was just like Christmas. Not one insalata 3ndy could be found.

Clint Adams: They will hear me say as the pavement whirls

It was just like Christmas. I finished urinating on the Mother Church, then we headed over to Fenway for to do some trespassing. Snowflakes fell and swirled in the breeze, their eddies making that ugly fluorescent light seem pretty, so pretty. The crash of duckpins resounded through the night, and hipsters flashed jazz hands at one another. It was just like Christmas. Not one insalata 3ndy could be found.

31 December 2007

MJ Ray: Strange Traditions: Dinner for One

The last day of each year sees one of the strangest traditions I've ever noticed. A 40-year-old English-language film of an 80-year-old music hall comedy is played repeatedly on various stations in Germany - and in Austria, Switzerland, Denmark and Australia too. There's not only the film, but edited versions, localised versions, spoofs and other shows based around it too. The film still isn't shown on any English stations, as far as I can tell. I'd never heard of it before getting satellite TV. So, to see what the fuss is about, grab your Dinner for One times (may contain annoying ads if you switch Javascript on) and sit down with the satellite TV (best), or enjoy this description and script from analoguesat, a German-language description or this BBC report which includes this snippet:-
"To the end of his life, Freddie Frinton heartily disliked Germany and the Germans, thanks to his own wartime experience. He refused to allow a German-language version to be made. That is what has led to the extraordinary fact that today the Germans as a nation have embraced a trifling one-act play in English as their all-time favourite entertainment."
Anyway, the last joke is on we English. It's much better than the aging boy bands on our TV at New Year, isn't it? (Updated 2008-01-03 with 1 comment)

10 November 2007

Clint Adams: Will the light flicker, will the hills lift?

I spent 10 hours unraveling a horrible place: from Q to A to B to the valley. I watched countless mystic forces acting one way, then another, but the ones I could see were not the ones tugging on my heartstrings. I doubt Freddie knows, but will Emily call? Will Amy? At some point blood will rain down from those allied stars. At midnight, I turned into a pumpkin, but no one thought to inform me, least of all before I encountered the pink ribbon barriers. After that, all was lost, and Torres was forgotten, though his jowls were not.
Unwanted boots and unwanted bridges
Thinking of shells and penne with ridges
A picture was not there.

25 June 2007

MJ Ray: Blogging from debconf7: 4. Sleepin ur BoF, hijakin ur sesh, climbin ur volkano

This has been delayed for a few days, for reasons I explain in another post, so my remaining debconf posts will be published quite quickly, every few hours. Monday started with me nearly asleep in the back of the Debian-Med session. Interesting stuff and I'm starting to get involved in supplying web apps to the health sector, but I must watch the session back. Then bureado's excellent debian in venezuela talk and a big shock for me when I got called out. The afternoon and evening were mentioned in the last post. Tuesday opened with me confused in the Xen session and then I went to a good-looking session on combining the debian-installer, ubuntu OEM bits, backuppc and openvpn but lkcl didn't show. (His diary suggests he was around. Wonder what happened.) I kicked the session along a bit and learned stuff from others about preseeding, FAI, backuppc, bacula, amanda and so on, but it didn't seem like anyone had plugged these together in quite the same way. At the end of the session, I bumped into Eddie Bleasdale who it seems is working with Phil Hands and lkcl on a very interesting project. He spoke more about it in a meeting the afternoon, hoping that Phil would arrive to handle the technical side. Phil timed his entry to perfection, about 30 minutes after the end of the talk. In between the hijacked talks, I went for a walk up to the top of Arthur's Seat and back with jumpula and someone else whose name I've forgotten - sorry! Oh, and what is it with all the loldebian posts this week?

16 April 2007

Alastair McKinstry: Measuring Power Consumption

Following a previous thread on Planet Debian (allowing for kids delaying projects, as usual), I borrowed a power meter to see what the consumption of various computers and electronics around the house was. An interesting and worthwhile exercise. Computers My main gateway system (hosting this blog, mail, etc. on a static IP) is a Pegasos PowerPC; it is on 24 x 7, and pulls approx. 60W. Other computers around the house include a Dell 2.5 GHz P4, which is my main development machine (with multiple disks, etc. onboard) pulling 72W. Downstairs, for family use we have an Intel MacMini pulling approx. 30W. Peripherals add up: on the Pegasos there is an ADSL router and a Linksys Wifi router, pulling approx. 5 W each continuously, and a UPS which consumes around 10W (I'm not sure if this is continuous, or high because its recharging a battery which was drained by pulling power plugs to measure the power usage ...). The LCD monitor upstairs (used by Pegasos and Dell) draws 20W. If I was purchasing again, I'd get a MacMini for the gateway. The current setup costs, I estimate, 7.50 Euro /month, which is probably worth it to have a 24x7 server (I'm not sure if moving to a hosting facility would be worth it, as it has advantages for running bittorrent, etc. that a remote server wouldn't, as well as being able to host content without the legal threat that a hosted server would have: with the current setup, if anyone challenges the content of stuff on my server, my ISP is not legally involved (as I host it), and hence won't (or shouldn't) pull the plug. This is worth something to me, even if I don't spend my days in court defending libel actions). One surprise was the CRT monitor used with the MacMini draws 10W when powered off, not on standby. Turn it off at the mains switch rather than the monitor button. Wall warts (chargers, etc.) draw around 1-2W due to eddies when nothing is connected. Conclusions: Swapping the Pegasos and MacMini are not an option as the family prefer the convenience of the fast start-up OS X. (This has made me think fast booting and suspend-to-disk are good priority goals for Debian.) I might however replace the Pegasos with an old pentium-class laptop, which would, at 25W, drop 40-50W as I would no longer need the UPS for 24x7 use. Put all peripherals, wall warts, etc. on a separate power strip that can be easily turned off together when not in use. TV We have a TV with the growing set of set-top-boxes: a DVD Player, Satellite unit and VCR, along with a SCART switch to connect them all together. Between them, they pull 86W. With the TV turned off, the boxes draw 20W. (TV pulls 10W on standby). I've been looking at getting an "Intelligent" mains lead, which cuts the power to the peripherals when the main item (e.g. the TV) is powered down, either totally or on standby. At nearly 40 Euro, however, it will take over a year to pay for itself. The alternative is just turning them off by hand, but of course as in most houses, the main switch is buried behind the TV cabinet out of reach. I'll swap the power strip with one with a single main switch that can be reached easily at the end of the evening instead. Other Big Users The house was designed with energy-efficiency in mind, and theres not much scope for savings elsewhere. The main users are underfloor electric heating, water heating, washing machine and dryer (condensor type). The water heating we hope to replace / augment with Solar over the summer; ditto the drier, with an outdoor shed /gazebo for drying clothes in the rainy Galway days. While the drier consumes a lot of electricity, at least the hot air is recycled via a heat-exchanger to heat the house in winter when its most used. Its worth keeping an eye on the drier and washing machine, though. We mostly use these at night, using cheaper Night-rate electricity, so while our usage patterns may not change, keep an eye on the energy consumption of these devices: as they grow old the bearings, motor, etc. begin to wear and energy consumption goes up. Look out for when to replace them. The other big niggle I have is a light fitting in the main living room. This is expensive, with 3 x 60 W spotlight bulbs in it. The kitchen, which is approx. half the size of the living room, gets lit by a single 20 W CFL. Swapping the spotlights with CFLs is not an option, and replacing it means replacing a ceiling mounted fitting: as the wires are embedded in the ceiling, moving it is not trivial. Any recommendations ?

26 January 2007

David Pashley: Eddie 0.2 RSS and Atom Parser

I noticed today that Mark Pilgrim linked to Eddie, my liberal RSS and Atom parsing library for Java, so I figured I should make a new release. It's been a few months since I did any serious work on the parser, but in the last few days I've reduced the number of test case failures to less than 100 out of 3502 test cases which come as part of Mark's Feedparser parser for python. The majority of the failures are in the date parsing routines and due to bugs in the Jython library which cause literal dictionaries not to match with classes inherited fro PyDictionary. Improvements in this version include:
  • Massively improved support for different character encodings. With Java 6, it also has support for UTF32 feeds.
  • CDF Support.
  • Optional support of TagSoup for sanitizing of HTML in entries.
  • Improved support for different input sources including String, InputStream and byte[].
  • Numerous bug fixes, with 97% of test cases passing, up from 90%
If you use Eddie, drop me an email. I'd like to thank Mark Pilgrim again for providing the community with a fantastic and comprehensive suite of test cases, extensive documentation and a first class Python library.

25 January 2007

David Pashley: Speedy Java 6

I was quietly minding my own business, fixing some encoding bugs in Eddie, my liberal RSS and Atom parser, when I noticed that Java 6 included support for UTF-32, which is one of the encoding tests that was failing. I downloaded and installed the Ubuntu packages and installed it, and decided to run a quick benchmark using my unit tests. First up was the Sun Java 5 JVM. I'd been running the unit tests all night, but timed it this time,and got these results:
Ran 3502 tests
Passed 3322 tests
Failed 180 tests
real    1m10.293s
user    0m40.375s
sys     0m3.632s
Next I tried the Sun Java 6 JVM, using the same jar files and got;
Ran 3502 tests
Passed 3326 tests
Failed 176 tests
real    0m56.059s
user    0m39.198s
sys     0m4.212s
One thing to note was that it spend a couple of seconds noticing new jars to read, so I decided to run it again and got:
Ran 3502 tests
Passed 3326 tests
Failed 176 tests
real    0m45.317s
user    0m34.770s
sys     0m3.516s
Wow, I'd gone from 70 seconds to 45 seconds using the new runtime, and interestingly enough, past 4 more tests in the process. I'm assuming they are the UTF-32 tests, although I have't checked yet. The other thing for me to try is recompiling the code to see if that has any additional benefits. Update: Got around to checking what Java 6 fixed and it turned out it was the additional support for koi-u and cspc862latinhebrew encodings. After I fixed the UTF32 support in Eddie, it passed an additional 16 tests. Down to just 160 out of 3502. I just wish they would add support for some of the stranger encodings. Maybe this will happen when it's open source.

19 December 2006

Paul Wise: 18 Dec 2006

[RV Heraclitus SE Asia voyage: blog photos] The past few weeks, we changed our route to visit an uninhabited island (aka Ko Pulau Island) said to be "National Geographic, man" by some Americans we met in Kupang. On the way to Ko Pulau Island we saw a large school of pilot whales and dolphins, a humpback or other whale close to shore, a blue starfish and hot water vents nearby on the same mostly dead reef, a flock of birds feasting on a dense school of fish, a manta ray, a bonfire on the beach shared with the kids of Rote (who we swapped roast banannas and coconuts with), a clean hull and renewed sea-sickness. At Ko Pulau Island, we saw a long white beach made of small bead things instead of sand, with surf at either end and reef in between, a green lagoon with islands being eaten away at the base, a monkey-head rock, pink coral, reef fish, sea urchins, various pieces of flotsam washed up on the beach (flip-flops, a light-bulb, bottles, wood, burnie-beans, nautilus shells, a seabird egg, a dead seabird and other crap), sunset over the ocean with golden cirrus in the sky, turtle nests, tracks and hatchlings scurrying off into the water, Indonesian fishermen in need of water and turtle eggs, tidal pools with the occasional crab, ghost crabs darting towards the water, a pandanus stand, a small cave surrounded by discarded turtle eggshells, spinifex, hermit crabs, scrambling lizards, sunburn and other things. Later in our voyage, we saw a big lone flying fish, land looming mountainous on starboard, TNI, gratis reef fish, water buffalo and threatening rain clouds. The next major stop was a bay on the south side of Sumba, black cliffs to port and an eroded hillside to starboard. There, we enjoyed the excellent snorkeling against the cliffs and off the beach, birds calling from the forest, wasps - shiny blue and otherwise, meeting roaming cows in the forest, forest fungi and other sights. We met some fishermen and drove through the forested slopes toward a nearby city. On the way, we visited an Indonesian village and saw their traditional animist temple, ample baby pigs & dogs, tons of kids trying to get in photos, satellite dishes and graves in front of houses. Unfortunately, I crashed once we reached the hotel, missing eating and night life, but I did enjoy the sights from the windows of the cramped 4WD we were in. We headed for the 9.8 knot passage of Selat Sape, complete with eddies, currents, a barracuda and the steep slopes of a silent volcano. Since there, we saw an increasing number of interesting and curious Indonesian vessels, fish traps, the grey shapes of dolphins swimming in the aqua water under the bow, a floating sandal, a school of mahi-mahi jumping out of the water, a misty night, numerous schools of feeding fish, entangled luminescent trails left by dolphins swimming in the phosphorescent water beneath the bow, flashes of lightning in the distance, our first rain since Cairns, the associated storm, Jack the fisherman (a mast hallucination) and other things. Our next stop was Kalimunjava (north of Java, Indonesia), more than half way to Phuket. We spent a week there, watched lightning, collected rain, visited the local school, dived and snorkelled on the magnificant reef with some really nice university students (hi Lely, Dudu, Jaos and others) from Java who were doing a study on the corals and hiked up the steep slopes of the island. From there we ambled past Borneo, towards Bintan, near Singapore, experiencing the first non-calm seas in ages, dolphins in the storm, floating lines of debris, big barges, container ships and megatankers, a fancy, shiny yacht, fishing vessels with 50,000 lights, Rain Drop and it's egg (child of Rain the gecko), amazing cloudscapes at sunrise, throughout the day and at sunset on the way. Amazingly, we met the 8 (Infinity, the new PCRF vessel) one find day in the South China Sea on their way to Bali. Eddie saw them from 5 miles away and knew almost straight away it was them. Michelle came on board and a lone daytime cumi (squid) swam between us as we parted. Before we arrived at Pulau Bintan (near Singapore), we saw seasnakes and a palm tree floating and lots of wind and rain. We will probably arrive in Thailand by January and I'm thinking of passing thru Sydney on the way home, so let so please mail me if you want to meet up.

24 July 2006

David Pashley: Strict feed parsers are useless

Erich, I'm not entirely sure what you did to break Planet, but using a strict feed parser will just result in you missing a significant number of entries. People sadly don't produce valid feeds and will blame your software rather than their feeds. It doesn't help that a number of validators aren't entirely strict and that RSS doesn't have a very comprehensive spec. RSS is a lot worse than Atom, in part thanks to the Atom validator and very well thought out spec. It's for this reason that I ended up writing Eddie rather than using ROME as it was a DOM parser and just failed to get any information out of a non-wellformed feed. Eddie on the other hand is a SAX-based parser. In a recent comparison, an Eddie based aggregator managed to correctly parse several more entries than a ROME based aggregator one particular day. You also have major aggregators being liberal. Sam Ruby discussed this recently with Bloglines becoming the defacto validator; if bloglines parses it, then it's valid. We had the same problem with HTML with people making sure their pages worked in a browser rather than met the spec. I suspect the problem you had with Planet is that you failed to close a tag, causing the rest of the page to be in bold or be a link etc. This is fairly easily solvable and in fact has been with FeedParser, which is the feed parsing library Planet uses. It has support for using HTMLTidy and similar libraries for fixing unbalanced elements. Eddie uses TagSoup to do a similar thing. As a result I've not noticed any particular entry leaking markup and breaking the page. Parhaps Planet Debian just needs to install one of the markup cleaning libraries. I agree that people should use XML tools where possible. Unfortunately, most blogging tools use text based templating systems, which makes producing non-wellformed XML too easy. To deal with this I pass all my output through an XSLT filter, which means that everything is either well formed or doesn't output at all. Unfortunately I don't think everyone would be capable or willing to use XSLT.

1 June 2006

David Pashley: Tidying Up

A few hours ago I got stressed about the lack of leg room under my desk and ended up spending the next few tidying and moving all of my computers to under the next desk. I also made the mistake of starting to remove keys from my keyboard to clean something sticky and found myself surrounded by keys and a keyless keyboard. It's now nice and shiny, which is more than can be said for the rest of the flat, which is now overrun with all the crap that was around my desk. Another thing that could do with a tidy up is Eddie, my Java liberal feed parsing library. After the initial coding sprint, I've had time to sit back and look at the design of the library and clean up any thing that sticks out. As mentioned in a previous entry, one of the things that has bothered me is that when ever you need to call an object method, you need to be certain that the object is not null. The means you end up with code like:
if (string != null && strong.equals("string"))  
This quickly becomes tiresome and the test for null distracts from the meaning of the code. Fortunately I was reminded of an improvement for string objects. Ideally, we should all be writing comparison conditionals like rvalue == lvalue. (an rvalue mostly is an expresion you can't assign to). The most common rvalue is a literal value like a string constant. The advantage of getting into the habit of writing code like this is that you'll discover at compile time when you accidentally write = rather than ==. Because you can't assign to an rvalue, the compiler will complain. What makes this interesting from a java string point of view is that you can call methods on string literals. Comparing a variable to a string literal, rather than calling .equals() on a variable is that the string literal is not going to be null, so you can remove the test for null and simplify the code:
if("string".equals(string))  
I know it's not everyone's cup of tea, but I prefer it to testing for null every time I look at a string. The other thing is that I've been reading Hardcore Java by Robert Simmons at work. Considering I've only got a few pages in so far. I've received a surprisingly large number of ideas to improve my code. The one that sticks in my head is using assert for doing post and pre conditions on your functions. Using asserts have number of advantages over throwing exceptions, including the fact they get optimised away when you do a production release. In Eddie, during a <feed> element I determine the version of Atom that we are parsing. This had a number of nested if/else if/else blocks. At the end of the function, I wanted to make sure I had set the version string to something, so had the following code:
if (!this.feed.has("format"))  
   throw new SAXParseException("Failed to detect Atom format", this.locator);
 
However, using assertions I can write this as
assert(this.feed.has("format")) : "Failed to detect Atom format";
I highly recommend the Hardcore java book if you want to improve your java programming. It includes sections on the new features of Java 1.5 and using collections. I've made a couple of other cleanups including going through member variable access specifiers to make sure they are right and making several public methods and variables and making them priavte. I also have a couple of ideas about refactoring some of the code to clean it up. Redesigning and refactoring code is almost more fun than writing it in the first place. You get to be in competition with yourself, challenging yourself to write better code and end up with cleaner code in the process. A couple of things I want to do in the near future is use a profiler and code coverage tools. If anyone has recommendations for either of these tools that integrates nicely with eclipse, I'd love to know.

Next.