Search Results: "zed"

22 April 2014

Erich Schubert: Kernel-density based outlier detection and the need for customization

Outlier detection (also: anomaly detection, change detection) is an unsupervised data mining task that tries to identify the unexpected.
Most outlier detection methods are based on some notion of density: in an appropriate data representation, "normal" data is expected to cluster, and outliers are expected to be further away from the normal data.
This intuition can be quantified in different ways. Common heuristics include kNN outlier detection and the Local Outlier Factor (which uses a density quotient). One of the directions in my dissertation was to understand (also from a statistical point of view) how the output and the formal structure of these methods can be best understood.
I will present two smaller results of this analysis at the SIAM Data Mining 2014 conference: instead of the very heuristic density estimation found in above methods, we design a method (using the same generalized pattern) that uses a best-practise from statistics: Kernel Density Estimation. We aren't the first to attempt this (c.f. LDF), but we actuall retain the properties of the kernel, whereas the authors of LDF tried to mimic the LOF method too closely, and this way damaged the kernel.
The other result presented in this work is the need to customize. When working with real data, using "library algorithms" will more often than not fail. The reason is that real data isn't as nicely behaved - it's dirty, it seldom is normal distributed. And the problem that we're trying to solve is often much narrower. For best results, we need to integrate our preexisting knowledge of the data into the algorithm. Sometimes we can do so by preprocessing and feature transformation. But sometimes, we can also customize the algorithm easily.
Outlier detection algorithms aren't black magic, or carefully adjusted. They follow a rather simple logic, and this means that we can easily take only parts of these methods, and adjust them as necessary for our problem at hand!
The article persented at SDM will demonstrate such a use case: analyzing 1.2 million traffic accidents in the UK (from data.gov.uk) we are not interested in "classic" density based outliers - this would be a rare traffic accident on a small road somewhere in Scotland. Instead, we're interested in unusual concentrations of traffic accidents, i.e. blackspots.
The generalized pattern can be easily customized for this task. While this data does not allow automatic evaluation, many outliers could be easily verified using Google Earth and search: often, historic imagery on Google Earth showed that the road layout was changed, or that there are many news reports about the dangerous road. The data can also be nicely visualized, and I'd like to share these examples with you. First, here is a screenshot from Google Earth for one of the hotspots (Cherry Lane Roundabout, North of Heathrow airport, which used to be a double cut-through roundabout - one of the cut-throughs was removed since):
Screenshot of Cherry Lane Roundabout hotspot
Google Earth is best for exploring this result, because you can hide and show the density overlay to see the crossroad below; and you can go back in time to access historic imagery. Unfortunately, KML does not allow easy interactions (at least it didn't last time I checked).
I have also put the KML file on Google Drive. It will automatically display it on Google Maps (nice feature of Drive, kudos to Google!), but it should also allow you to download it. I've also explored the data on an Android tablet (but I don't think you can hide elements there, or access historic imagery as in the desktop application).
With a classic outlier detection method, this analysis would not have been possible. However, it was easy to customize the method; and the results are actually more meaningful: instead of relying on some heuristic to choose kernel bandwidth, I opted for choosing the bandwidth by physical arguments: 50 meters is a reasonable bandwidth for a crossroad / roundabout, and for comparison a radius of 2 kilometers is used to model the typical accident density in this region (there should other crossroads within 2 km in Europe).
Since I advocate reproducible science, the source code of the basic method will be in the next ELKI release. For the customization case studies, I plan to share them as a how-to or tutorial type of document in the ELKI wiki; probably also detailing data preprocessing and visualization aspects. The code for the customizations is not really suited for direct inclusion in the ELKI framework, but can serve as an example for advanced usage.
Reference:
E. Schubert, A. Zimek, H.-P. Kriegel
Generalized Outlier Detection with Flexible Kernel Density Estimates
In Proceedings of the 14th SIAM International Conference on Data Mining (SDM), Philadelphia, PA, 2014.
So TLDR of the story: A) try to use more established statistics (such as KDE), and B) don't expect an off-the-shelf solution to do magic, but customize the method for your problem.
P.S. if you happen to know nice post-doc positions in academia:
I'm actively looking for a position to continue my research. I'm working on scaling these methods to larger data and to make them work with various real data that I can find. Open-source, modular and efficient implementations are very important to me, and one of the directions I'd like to investigate is porting these methods to a distributed setting, for example using Spark. In order to get closer to "real" data, I've started to make these approaches work e.g. on textual data, mixed type data, multimedia etc. And of course, I like teaching; which is why I would prefer a position in academia.

16 April 2014

Wouter Verhelst: Call for help for DVswitch maintenance

I've taken over "maintaining" DVswitch from Ben Hutchings a few years ago, since Ben realized he didn't have the time anymore to work on it well. After a number of years, I have to admit that I haven't done a very good job. Not becase I didn't want to work on it, but mainly because I don't have enough time to fix DVswitch against the numerous moving targets that it uses; the APIs of libav and of liblivemedia are fluent enough that just making sure everything remains compilable and in working order is quite a job. DVswitch is used by many people; DebConf, FOSDEM, and the CCC are just a few examples, but I know of at least three more. Most of these (apart from DebConf and FOSDEM) maintain local patches which I've been wanting to merge into the upstream version of dvswitch. However, my time is limited, and over the past few years I've not been able to get dvswitch into a state where I confidently felt I could upload it into Debian unstable for a release. One step we took in order to get that closer was to remove the liblivemedia dependency (which implied removing the support for RTSP sources). Unfortunately, the resulting situation wasn't good enough yet, since libav had changed API enough that current versions of DVswitch compiled against current versions of libav will segfault if you try to do anything useful. I must admit to myself that I don't have the time and/or skill set to maintain DVswitch on an acceptable level all by myself. So, this is a call for help: If you're using DVswitch for your conference and want to continue doing so, please talk to us. The first things we'll need to do: See you there?

13 April 2014

Andreas Metzler: balance sheet snowboarding season 2013/14

Little snow, but above-average season. The macro weather situation was very stable this year, very high snowfall in Austria's south (eastern tyrol and carinthia), and long periods of warm and sunny weather with little precipitation on the northern side of the alps (i.e. us). This had me going snowboarding a lot, but almost exclusively in Dam ls since it is characterized by a) grassy terrain (no stones) and b) huge numbers of snow cannons. I started early (December 7) with another 6 days on piste in December. If there had been more snow the season would have been a long one, too. - Season's end depends on the timimg of easter (because of the holidays) which would have been late. However I again stopped rather early, last day was March 30. In addition to the days listed below I had an early season's opening at the glacier in Pitztal. I attended the pureboarding in November (21st to 23rd). Looking back at the season I am not quite satisfied with my progress, I just have not managed to implement and practise the technique I should have learned there. It is next to impossible when the slopes are full, and when they aren't one likes to give it a run. ;-) Here is the balance sheet:
2005/06 2006/07 2007/08 2008/09 2009/10 2010/11 2011/12 2012/13 2013/14
number of (partial) days251729373030252330
Dam ls1010510162310429
Diedamskopf154242313414191
Warth/Schr cken030413100
total meters of altitude12463474096219936226774202089203918228588203562274706
highscore10247m8321m12108m11272m11888m10976m13076m13885m12848m
# of runs309189503551462449516468597

11 April 2014

Joey Hess: propellor introspection for DNS

In just released Propellor 0.3.0, I've improved improved Propellor's config file DSL significantly. Now properties can set attributes of a host, that can be looked up by its other properties, using a Reader monad. This saves needing to repeat yourself:
hosts = [ host "orca.kitenet.net"
        & stdSourcesList Unstable
        & Hostname.sane -- uses hostname from above
And it simplifies docker setup, with no longer a need to differentiate between properties that configure docker vs properties of the container:
 -- A generic webserver in a Docker container.
    , Docker.container "webserver" "joeyh/debian-unstable"
        & Docker.publish "80:80"
        & Docker.volume "/var/www:/var/www"
        & Apt.serviceInstalledRunning "apache2"
But the really useful thing is, it allows automating DNS zone file creation, using attributes of hosts that are set and used alongside their other properties:
hosts =
    [ host "clam.kitenet.net"
        & ipv4 "10.1.1.1"
        & cname "openid.kitenet.net"
        & Docker.docked hosts "openid-provider"
        & cname "ancient.kitenet.net"
        & Docker.docked hosts "ancient-kitenet"
    , host "diatom.kitenet.net"
        & Dns.primary "kitenet.net" hosts
    ]
Notice that hosts is passed into Dns.primary, inside the definition of hosts! Tying the knot like this is a fun haskell laziness trick. :) Now I just need to write a little function to look over the hosts and generate a zone file from their hostname, cname, and address attributes:
extractZoneFile :: Domain -> [Host] -> ZoneFile
extractZoneFile = gen . map hostAttr
  where gen = -- TODO
The eventual plan is that the cname property won't be defined as a property of the host, but of the container running inside it. Then I'll be able to cut-n-paste move docker containers between hosts, or duplicate the same container onto several hosts to deal with load, and propellor will provision them, and update the zone file appropriately.
Also, Chris Webber had suggested that Propellor be able to separate values from properties, so that eg, a web wizard could configure the values easily. I think this gets it much of the way there. All that's left to do is two easy functions:
overrideAttrsFromJSON :: Host -> JSON -> Host
exportJSONAttrs :: Host -> JSON
With these, propellor's configuration could be adjusted at run time using JSON from a file or other source. For example, here's a containerized webserver that publishes a directory from the external host, as configured by JSON that it exports:
demo :: Host
demo = Docker.container "webserver" "joeyh/debian-unstable"
    & Docker.publish "80:80"
    & dir_to_publish "/home/mywebsite" -- dummy default
    & Docker.volume (getAttr dir_to_publish ++":/var/www")
    & Apt.serviceInstalledRunning "apache2"
main = do
    json <- readJSON "my.json"
    let demo' = overrideAttrsFromJSON demo
    writeJSON "my.json" (exportJSONAttrs demo')
    defaultMain [demo']

10 April 2014

Joey Hess: Kite: a server's tale

My server, Kite, is finishing its 20th year online. It started as kite.resnet.cornell.edu, a 486 under the desk in my dorm room. Early on, it bounced around the DNS -- kite.ithaca.ny.us, kite.ml.org, kite.preferred.com -- before landing on kite.kitenet.net. The hardware has changed too, from a succession of desktop machines, it eventually turned into a 2u rack-mount server in the CCCP co-op. And then it went virtual, and international, spending a brief time in Amsterdam, before relocating to England and the kvm-hosting co-op. Through all this change, and no few reinstalls from scratch, it's had a single distinct personality. This is a multi-user unix system, of the old school, carefully (and not-so-carefully) configured and administered to perform a grab-bag of functions. Whatever the users need. I read the olduse.net hacknews newsgroup, and I see, in their descriptions of their server in 1984, the prototype of Kite and all its ilk. It's consistently had a small group of users, a small subset of my family and friends. Not quite big enough to really turn into a community, and we wall and talk less than we once did.
Exhibit: Kite as it appeared in the 90's [Intentionally partially broken, being able to read the cgi source code is half the fun.] Kite was an early server on the WWW, and garnered mention in books and print articles. Not because it did anything important, but because there were few enough interesting web sites that it slightly stood out.
Many times over these 20 years I've wondered what will be the end of Kite's story. It seemed like I would either keep running it indefinitely, or perhaps lose interest. (Or funding -- it's eaten a lot of cash over the years, especially before the current days of $5/month VPS hosting.) But I failed to anticipate what seems to really be happening to it. Just as I didn't fathom, when kite was perched under my desk, that it would one day be some virtual abstract machine in a unknown computer in anther country. Now it seems that what will happen to Kite is that most of the important parts of it will split off into a constellation of specialized servers. The website, including the user sites, has mostly moved to branchable.com. The DNS server, git server and other crucial stuff is moving to various VPS instances and containers. (The exhibit above is just one more automatically deployed, soulless container..) A large part of Kite has always been about me playing with bleeding-edge stuff and installing random new toys; that has moved to a throwaway personal server at cloudatcost.com which might be gone tomorrow (or might keep running for free for years). What it seems will be left is a shell box, with IMAP access to a mail server, and a web server for legacy /~user/ sites, and a few tools that my users need (including that pine program some of them are still stuck on.) Will it be worth calling that Kite?
[ Kite users: This transition needs to be done by December when the current host is scheduled to be retired. ]

Christine Spang: a tuturial about search

Today I gave a tutorial at PyCon 2014 entitled Search 101: An Introduction to Information Retrieval. It was an experiment of sorts: the first workshop I've run primarily by myself, my first tutorial at PyCon, my first paid teaching gig. It was an opportunity to take some of the lessons I learned from teaching the Boston Python Workshop and apply them to a new situation. The material itself is a distillation of many hours of frustration with the documentation for various open source search engine libraries, frustration that they didn't tell me where to start or about the big picture, they just jumped straight into the details. Here's what worked: Here's what caused problems: Intermediate students are a different crowd than beginners. There was less of an air of discovery in the room, though I organized the class around open-ended tasks. Since the material allowed for folks to take it in the direction of their interest, I found it a bit difficult to gauge whether people were following or not. Overall though, everyone was attentive and studious. I had fun. Ruben and Stuart, the PyCon tutorial organizers, had logistics running super smoothly, AV, lunch, everything. Thanks for that you guys, you rock. :) And thanks as well to my helpers: Leo, the tutorial host, Eben, my TA, and Roberto, on AV. It's impossible to pay adequate attention to 20+ people as a single person, couldn't have done a decent job without y'all. ;)

8 April 2014

Russ Allbery: Review: With Charity for All

Review: With Charity for All, by Ken Stern
Publisher: Anchor
Copyright: November 2013
ISBN: 0-307-74381-0
Format: Trade paperback
Pages: 244
I've been reading a lot lately about how to measure charitable organizations and where best to give your money, which has already prompted a rethinking of which organizations I want to support. Most of that has been shorter, on-line blog posts, but I've been wanting a deeper analysis. This is the first of a couple of books I've found on the topic. (The second, an analysis of microfinance, is waiting for me to have more time to read.) For this kind of book, the background of the author matters a lot. Stern was the former CEO of National Public Radio for eight and a half years, so he has practical experience in the business side of non-profits. He's credited with improving NPR's financial position and management and oversaw an era of rapid growth. In With Charity for All, he speaks frankly about his own personal experiences as well as his research. My impression is that he's typical of competent business managers of non-profits: not particularly radical, focused mostly on business issues, and the sort of person that a non-profit needs to turn a mission or cause into a sustainable organization. It's worth noting in advance that With Charity for All is specifically about the United States charitable sector and US laws and approaches. Some of this may generalize to other countries, but non-profit and charity law is very different between different countries, and the US attitude towards government, non-profits, and the appropriate division of responsibilities between them often doesn't match the prevailing view elsewhere. This book may well be interesting to people outside the US, but only if they're curious about the US charitable sector. The short summary of the message of this book is that we're not getting our money's worth out of charity, either as donors or as taxpayers who are indirectly subsidizing these organizations. This is not (usually) due to any outright fraud, although there certainly is some of that due to the almost total lack of oversight of charitable organizations. Rather, it's the result of a complex combination of history and institutional momentum, focus on the aspects of a charity that play well to potential donors, a concerted effort at feel-good marketing, well-intentioned but misguided donor dictates, adoption of approaches and business practices from the for-profit sector, and, above everything else, the almost complete absence of evidence-based evaluation of charities. Evidence-based evaluation is currently a hot trend among charities. I heard about this book via one of the organizations that is focusing on this (GiveWell). This is not what sites like Charity Navigator do, and indeed Stern talks at some length about the serious limitations of the Charity Navigator approach. All that sites of that type can do is look at spending breakdowns. Stern makes a strong argument that a focus on reducing overhead, as measured by those financial forms, is misguided and leads to very ineffective and unorganized charities. Worse, all Charity Navigator can tell you is if the charity is spending money on something they declare to be in line with their goal, not whether that money did any good whatsoever. The short and depressing news is that most of that money does little or no good, and worse, that most charities have no systematic way of measuring whether it does any good. I'm particularly interested in how to ground charities in more concrete evidence, and Stern's discussion does not disappoint. But one of the things I liked about the book is that he doesn't stop there. Other topics include the dizzying variety of charities and the resulting confusion and dispersion of resources, the non-profits that are practically indistinguishable from for-profit companies except that they get huge tax breaks (sports organizations, as you might expect, are probably the worst offenders, but Stern has harsh commentary on the US hospital system), the mess of compensation in the non-profit sector, and the serious problems created by major donors who want to get directly involved and oversee how their money is spent. All of these topics are fascinating, if a bit depressing in the aggregate. As is typical for a book of this sort, it's a lot easier to see the problems than to describe the solutions. Stern does try to offer his own prescriptions, most of which are variations on more oversight: more regulation, more effort put into measuring results, more concrete standards for success or failure, and a stronger culture of treating tax-exempt status as a privilege, not a right, that has to be defended by showing clear social benefit. I'm convinced those steps would be helpful, although less convinced that we're able to get there from here. One of his ideas I am already using on a small scale: pooling charitable donations with professional evaluators who can take the time to look deeper than financial breakdowns and do some independent measurement of results, or at least survey relevant studies done by others and possibly demand more be done. This model of a mutual fund for charitable giving has some promise, I think (more than a managed mutual fund, in fact), and is very similar to what GiveWell is doing. This is an advocacy book, and one that's primarily focused on concrete, measurable deliverables and on the finances. There are a few places where I have reason to be dubious of the black-and-white conclusions Stern draws. For example, he makes much of the bill collecting practices of charitable hospitals, which I agree is a travesty, but doesn't mention that non-profit hospitals produce better health outcomes. I'm also not sure his analysis is as directly applicable to advocacy organizations, where measuring success is a much more challenging proposition. I think the Southern Poverty Law Center is one of the best charities in the United States (admittedly based largely on gut feeling and presentation), but the nature of their work is not amenable to the style of analysis that Stern is doing here. It's therefore worth taking this book with a grain of salt. But, that said, Stern's conclusions are in line with things I've read elsewhere. I think he's pointing in the right direction, and I would recommend this book to anyone who gives regularly to charity. Hopefully it won't just cause you to stop. There are charities out there that are trying hard to measure their results and ensure they deliver social value for the money that we give them. Our system just doesn't support them very well, yet. Rating: 8 out of 10

7 April 2014

Andrew Pollock: [life] Day 69: Walk to King Island, a picnic at Wellington Point, the long slow acquisition of some linseed and a split lip

Today was a really good day, right up until the end, when it wasn't so good, but could have been a whole lot worse, so I'm grateful for that. I've been wanting to walk out to King Island at low tide with Zoe for a while, but it's taken about a month to get the right combination of availability, weather and low tide timing to make it possible. Today, there was a low tide at about 10:27am, which I thought would work out pretty well. I wasn't sure if the tide needed to be dead low to get to King Island, so I thought we could get there a bit early and possibly follow the tide out. I invited Megan and Jason to join us for the day and make a picnic of it. It turned out that we didn't need a really low tide, the sand bar connecting King Island to Wellington Point was well and truly accessible well before low tide was reached, so we headed out as soon as we arrived. I'd brought Zoe's water shoes, but from looking at it, thought it would be walkable in bare feet. We got about 10 metres out on the sand and Zoe started freaking out about crabs. I think that incident with the mud crab on Coochiemudlo Island has left her slightly phobic of crabs. So I went back to Jason's car and got her water shoes. I tried to allay her fears a bit by sticking my finger in some of the small holes in the sand, and even got her to do it too. I'm actually glad that I did get her water shoes, because the shell grit got a bit sharp and spiky towards King Island, so I probably would have needed to carry her more than I did otherwise. Along the way to the island we spotted a tiny baby mud crab, and Zoe was brave enough to hold it briefly, so that was good. We walked all the way out and partially around the island and then across it before heading back. The walk back was much slower because where was a massive headwind. Zoe ran out of steam about half way back. She didn't like the sand getting whipped up and stinging her legs, and the wind was forcing the brim of her hat down, so I gave her a ride on my shoulders for the rest of the way back. We had some lunch after we got back to Wellington Point, and Zoe found her second wind chasing seagulls around the picnic area. After an ice cream, we went over to the playground and the girls had a great time playing. It was a pretty good park. There was this huge tree with a really big, thick, horizontal branch only about a metre or two off the ground. All the kids were climbing on it and then shimmying along the branch to the trunk. Zoe's had a few climbs in trees and seems not afraid of it, so she got up and had a go. She did really well and did a combination of scooting along, straddling the branch and doing a Brazilian Jiu-Jitsu-style "bear crawl" along the branch. It was funny seeing different kids' limits. Zoe was totally unfazed by climbing the tree. Megan was totally freaking out. But when it came to walking in bare feet in an inch of sea water, Zoe wanted to climb up my leg like a rat up a rope, in case there were crabs. Each to their own. Zoe wanted to have a swim in the ocean, so I put her into her swimsuit, but had left the water shoes back in the car. Once again, she freaked out about crabs as soon as we got ankle deep in the water, and was freaking out Megan as well, so the girls elected to go back to playing in the park. After a good play in the park, we headed back home. We'd carpooled in Jason's truck, with both girls in the back. I'd half expected Zoe to fall asleep on the way back, but the girls were very hyped up and had a great time playing games and generally being silly in the back. When we got back to our place, Jason was in need of a coffee, so we walked to the Hawthorne Garage and had coffee and babyccinos, before Megan and Jason went home. It was about 3:30pm at this point, and I wanted to make a start on dinner. I was making a wholemeal pumpkin quiche, which I've made a few times before, and I discovered we were low on linseed. I thought I'd push things and see if Zoe was up for a scooter ride to the health food shop to get some more and kill some time. She was up for it, but ran out of steam part way across Hawthorne Park. Fortunately she was okay with walking and didn't want me to carry her and the scooter. It took us about an hour to get to the health food shop. Zoe immediately remembered the place from the previous week where we'd had to stop for a post-meltdown pit stop and declared she needed to go to the toilet again. We finally made it out of the shop. I wasn't looking forward to the long walk back home, but there were a few people waiting for a bus at the bus stop near the health food shop, and on checking the timetable, the bus was due in a couple of minutes, so we just waited for the bus. That drastically shortened the trip back. Zoe managed to drop the container of linseed on the way home from the bus stop, but miraculously the way it landed didn't result in the loss of too much of the contents, it just split the container. So I carefully carried the container home the rest of the way. By this stage it was quite a bit later than I had really wanted to be starting dinner, but we got it made, and Zoe really liked the pumpkin quiche, and ate a pretty good dinner. It was after dinner when things took a turn for the worse. Zoe was eating an ice block for dessert, and for whatever reason, she'd decided to sit in the corner of the kitchen next to the dishwasher, while I was loading it. I was carrying over one of the plates, and the knife managed to fall off the plate, bounce off the open dishwasher door and hit her in the mouth, splitting her lip. Zoe was understandably upset, and I was appalled that the whole thing had happened. She never sits on the kitchen floor, let alone in the corner where the dishwasher is. And this knife came so close to her eye. Fortunately the lip didn't look too bad. It stopped bleeding quickly, and we kept some ice on it and the swelling went down. I hate it when accidents happen on my watch. I feel like I'm fighting the stigma of the incompetent single Dad, or the abusive single Dad, so when Zoe sustains an injury to the face like a fat lip, which could be misinterpreted, I, well, really hate it. This was such a freak accident, and it could have gone so much worse. I'm just so glad she's okay. Zoe recovered pretty well from it, and I was able to brush her teeth without aggravating her lip. She went to bed well, and I suspect she's going to sleep really well. It's a bit cooler tonight, so I'm half-expecting a sleep in in the morning with any luck.

Russ Allbery: Review: Fantasy & Science Fiction, September/October 2011

Review: Fantasy & Science Fiction, September/October 2011
Editor: Gordon van Gelder
Issue: Volume 121, No. 3 & 4
ISSN: 1095-8258
Pages: 258
Another review of a magazine that I finished quite some time ago. Apologies for any inaccuracies or lack of depth in the reviews. There wasn't much in Charles de Lint's reviews in this issue that interested me, but Michelle West covers a great selection of books. Two of them (The Wise Man's Fear and The Quantum Thief) are already on my to-read list; the third, The Postmortal, sounded interesting and would go on my list to purchase if I didn't already have so many good books I've not read. Otherwise, this issue is short on non-fiction. The only other essay entry is a film review from Kathi Maio, which is the typical whining about all things film that F&SF publishes. "Rutger and Baby Do Jotenheim" by Esther M. Friesner: Baby is a former pole dancer with a toy poodle named Mister Snickers, which warns you right away that this story is going to involve a few over-the-top caricatures and more use of the word "piddle" than one might ideally want. Rutger is a mythology professor who tolerates her for the standard reasons in this sort of pairing. They're travelling across country to Baby's sister's wedding when their car breaks down in Minnesota, prompting an encounter with frost giants. As you might expect, this is a sort of fractured fairy tale, except based on Norse mythology instead of the more typical Grimm fare. The fun is in watching these two apparent incompetents (but with enough knowledge of mythology to clue in the reader) reproduce the confrontation between Thor and Utgard-Loki. The fight with old age is particularly entertaining. If you've read any of Friesner's other stories, you know what to expect: not much in the way of deeper meaning, but lots of fun playing with stereotypes and an optimistic, funny outcome. Good stuff. (7) "The Man Inside Black Betty" by Sarah Langan: This story comes with a mouthful of a subtitle: "Is Nicholas Wellington the World's Best Hope?" It's also a story that purports to be written by a fictional character, in this case one Saurub Ramesh (with Langan credited as having done "research"). It's told in the style of first-person journalism, relating the thoughts and impressions of Ramesh as he interviews Nicholas Wellington. The topic is Black Betty: a black hole above Long Island Sound. Wellington is a scientific genius and iconoclast with radical theories of black holes that contradict how the government has been attempting to deal with Black Betty, unsuccessfully. The structure here was well-handled, reminding me a lot of a Michael Lewis article during the financial collapse. Langan has a good feel for how journalism of this type mixes personalities, politics, and facts. But it's all setup and no story. We get some world building, and then it's over, with no resolution except pessimism. Meh. (4) "A Borrowed Heart" by Deborah J. Ross: Ross starts with the trappings of urban fantasy transplanted into a Victorian world: supernatural creatures about, a protagonist who is a high-class prostitute, and sex and a sucubus by the second page. It evolves from there into a family drama and an investigation, always giving the reader the impression that a vampire will jump out at any moment. But the ending caught me entirely by surprise and was far more effective due to its departure from the expected path. Well done. (7) "Bright Moment" by Daniel Marcus: The conflict between terraforming and appreciation for the universe as we find it is an old story pattern in science fiction, and Marcus doesn't add much here. I think the story would have been stronger if he'd found a way to write the same plot with a pure appeal to environmental beauty without the typical stakes-raising. But he does sprinkle the story with a few interesting bits, including a pod marriage and a futuristic version of extreme sports as a way of communing with nature. (6) "The Corpse Painter's Masterpiece" by M. Rickert: This is typical of my reaction to a Rickert story: shading a bit too much towards horror for me, a bit too cryptic, well-written but not really my thing. It's about a corpse painter who does the work of an informal mortician, improving the appearance of bodies for their funerals, and the sheriff who brings him all the dead bodies. It takes an odd macabre twist, and I have no idea what to make of the ending. (4) "Aisle 1047" by Jon Armstrong: Armstrong is best known for a couple of novels, Grey and Yarn, which entangle their stories in the future of marketing and commerce. One may be unsurprised, then, that this short story is on similar themes, with the intensity turned up to the parody point. Tiffan3 is a department-store saleswoman, spouting corporate slogans and advertising copy while trying to push customers towards particular products. The story follows the escalation into an all-out brand war, fought with the bubbly short-cut propaganda of a thirty-second commercial. For me, it fell awkwardly between two stools: it's a little too over-the-top and in love with its own bizarre alternate world to be effective satire, but the world is more depressing than funny and the advertising copy is grating. More of a curiosity than a successful story, I think. (5) "Anise" by Chris DeVito: Stories that undermine body integrity and focus on the fascinated horror of violation of physical boundaries aren't generally my thing, so take that into account in this review. Anise's husband died, but that's not as much of a problem as it used to be. Medical science can resurrect people via a sort of permanent, full-body life support system, making them more cyborg than human. "Anise" is about the social consequences of this technology in a world where a growing number of people have a much different relationship with their body than the typical living person today. It's a disturbing story that is deeply concerned with the physical: sex, blood, physical intimacy in various different forms, and a twisted type of psychological abuse. I think fans of horror will like this more than I did, although it's not precisely horror. It looks at the way one's perception of self and others can change by passing through a profound physical transformation. (5) "Spider Hill" by Donald Mead: I liked this story a lot better. It's about witchcraft and farm magic, about family secrets, and a sort of coming-of-age story (for a girl rather than a boy, for once). The main character is resourceful, determined, but also empathetic and aware of the impact of her actions, which made her more fun to read about. I doubt I'll remember this for too long, but when skimming through it again for a review, I had fond memories of it. (6) "Where Have All the Young Men Gone?" by Albert E. Cowdrey: Cowdrey in his paranormal investigation mode, which I like better than his horror mode. For once, the protagonist isn't even a lower-class or backwoods character. Instead, he's a military historian travelling in Austria who runs across a local ghost story. This is a fairly straightforward ghost investigation that follows a familiar path (albeit to an unusual final destination), but Cowdrey is a good story-teller and I liked the protagonist. (7) "Overtaken" by Karl Bunker: This is the sort of story that delivers its moral with the force of a hammer. It's not subtle. But if you're in the right mood for that, it's one of the better stories of its type. It's about a long-journey starship, crew in hibernation, that's overtaken by a far newer and faster mechanized ship from Earth that's attempting to re-establish contact with the old ships. The story is a conversation between the ship AIs. Save this one until you're in the mood for an old-fashioned defense of humanity. (8) "Time and Tide" by Alan Peter Ryan: Another pseudo-horror story, although I think it's better classified as a haunting. A wardrobe recalls a traumatic drowning in the childhood of the protagonist. As these things tend to do in stories like this, reality and memory start blurring and the wardrobe takes on a malevolent role. Not my sort of thing. (3) "What We Found" by Geoff Ryman: Any new Geoff Ryman story is something to celebrate. This is a haunting story on the boundaries between the scientific method and tribal superstition, deeply entangled with the question of how one recovers from national and familial trauma. How can we avoid passing the evils and madness of one generation down to the next? Much of the story is about family trauma, told with Ryman's exceptional grasp of character, but the science is entangled in an ingenious way that I won't spoil. As with Air, this is in no way science fiction. The science here would have fascinating and rather scary implications for our world, but clearly is not how science actually works. But as an insight into politics, and into healing, I found it a startlingly effective metaphor. I loved every bit of this. By far the best story of the issue. (9) Rating: 7 out of 10

3 April 2014

Daniel Pocock: LogAnalyzer and rsyslog MongoDB support now in wheezy-backports and Ubuntu

LogAnalyzer is a powerful but simple log file analysis tool. The upstream web site gives an online demo. It is developed in PHP, runs in Apache and has no other dependencies such as databases - it can read directly from the log files. For efficiency, however, it is now trivial to make it work with MongoDB on Debian. Using a database (including MongoDB and SQL backends) also means that severity codes (debug/info/notice/warn/error/...) are retained. These are not available from many log files. The UI can only colour-code and filter the messages by severity if it has a database backend. Package status The packages just entered Debian recently. It has now been migrated to wheezy-backports and Ubuntu so anybody on wheezy or Ubuntu can use it. Quick start with MongoDB The version of rsyslog in Debian wheezy does not support MongoDB output. It is necessary to grab 7.4.8 from backports. Some versions, up to 7.4.4 in backports, had bugs with MongoDB support - if you tried those, please try again now. The backported rsyslog is a drop-in replacement for the standard rsyslog package and for users with a default configuration it is unlikely you will notice any difference. For users who customized the configuration, as always, make a backup before trying the new version.
  • Install all the necessary packages: apt-get install rsyslog-mongodb php5-mongo mongodb-server
  • Add the following to /etc/rsyslog.conf:
    module (load="ommongodb")
    *.* action(type="ommongodb" server="127.0.0.1" db="logs" collection="syslog")
  • Look for the MongoDB settings in /etc/loganalyzer/config.php and uncomment them. Comment out the stuff for disk log access.
  • Restart rsyslog and then browse your logs at http://localhost/loganalyzer

Johannes Schauer: mapbender - maps for long-distance travels

Back in 2007 I stumbled over the "Plus Fours Routefinder", an invention of the 1920's. It's worn on the wrist and allows the user to scroll through a map of the route they planned to take, rolled up on little wooden rollers. At that point I thought: that's awesome for long trips where you either dont want to take electronics with you or where you are without any electricity for a long time. And creating such rollable custom maps of your route automatically using openstreetmap data should be a breeze! Nevertheless it seems nobody picked up the idea. Years passed and in a few weeks I'll go on a biking trip along the Weser, a river in nothern Germany. For my last multi-day trip (which was through the Odenwald, an area in southern Germany) I printed a big map from openstreetmap data which contained the whole route. Openstreetmap data is fantastic for this because in contrast to commercial maps it doesnt only allow you to just print the area you need but also allows you to highlight your planned route and objects you would probably not find in most commercial maps like for example supermarkets to stock up on supplies or bicycle repair shops. Unfortunately such big maps have the disadvantage that to show everything in the amount of detail that you want along your route, they have to be pretty huge and thus easily become an inconvenience because the local plotter can't handle paper as large as DIN A0 or because it's a pain to repeatedly fold and unfold the whole thing every time you want to look at it. Strong winds are also no fun with a huge sheet of paper in your hands. One solution would be to print DIN A4 sized map regions in the desired scale. But that has the disadvantage that either you find yourself going back and forth between subsequent pages because you happen to be right at the border between two pages or you have to print sufficiently large overlaps, resulting in many duplicate map pieces and more pages of paper than you would like to carry with you. It was then that I remembered the "Plus Fours Routefinder" concept. Given a predefined route it only shows what's important to you: all things close to the route you plan to travel along. Since it's a long continuous roll of paper there is no problem with folding because as you travel along the route you unroll one end and roll up the other. And because it's a long continuous map there is also no need for flipping pages or large overlap regions. There is not even the problem of not finding a big enough sheet of paper because multiple DIN A4 sheets can easily be glued together at their ends to form a long roll. On the left you see the route we want to take: the bicycle route along the Weser river. If I wanted to print that map on a scale that allows me to see objects in sufficient detail along our route, then I would also see objects in Hamburg (upper right corner) in the same amount of detail. Clearly a waste of ink and paper as the route is never even close to Hamburg.
As the first step, a smooth approximation of the route has to be found. It seems that the best way to do that is to calculate a B-Spline curve approximating the input data with a given smoothness. On the right you can see the approximated curve with a smoothing value of 6. The curve is sampled into 20 linear segments. I calculated the B-Spline using the FITPACK library to which scipy offers a Python binding.
The next step is to expand each of the line segments into quadrilaterals. The distance between the vertices of the quadrilaterals and the ends of the line segment they belong to is the same along the whole path and obviously has to be big enough such that every point along the route falls into one quadrilateral. In this example, I draw only 20 quadrilaterals for visual clarity. In practice one wants many more for a smoother approximation.
Using a simple transform, each point of the original map and the original path in each quadrilateral is then mapped to a point inside the corresponding "straight" rectangle. Each target rectangle has the height of the line segment it corresponds to. It can be seen that while the large scale curvature of the path is lost in the result, fine details remain perfectly visible. The assumption here is, that while travelling a path several hundred kilometers long, it does not matter that large scale curvature that one is not able to perceive anyways is not preserved. The transformation is done on a Mercator projection of the map itself as well as the data of the path. Therefore, this method probably doesnt work if you plan to travel to one of the poles. Currently I transform openstreetmap bitmap data. This is not quite optimal as it leads to text on the map being distorted. It would be just as easy to apply the necessary transformations to raw openstreetmap XML data but unfortunately I didnt find a way to render the resulting transformed map data as a raster image without setting up a database. I would've thought that it would be possible to have a standalone program reading openstreetmap XML and dumping out raster or svg images without a round trip through a database. Furthermore, tilemill, one of the programs that seem to be one of the least hasslesome to set up and produce raster images is stuck in an ITP and the existing packaging attempt fails to produce a non-empty binary package. Since I have no clue about nodejs packaging, I wrote about this to the pkg-javascript-devel list. Maybe I can find a kind soul to help me with it.
Here a side by side overview that doesnt include the underlying map data but only the path. It shows how small details are preserved in the transformation. The code that produced the images in this post is very crude, unoptimized and kinda messy. If you dont care, then it can be accessed here

Steve Kemp: Tagging images, and maintaining collections?

I'm an amateur photographer, although these days I tend to drop the amateur prefix, given that I shoot people for cash at least once a month. (It isn't my main job, and I'd never actually want it to be, because I'm certain I'd become unhappy hustling for jobs and doing the promotion thing.) Anyway over the years I've built up a large library of images, mostly organized in a hierarchy of directories beneath ~/Images. Unlike most photographers I don't use aperture, lighttable, or any similar library management. I shoot my images in RAW, convert to JPG via rawtherapee, and keep both versions of the images. In short I don't want to mix the "library management" functions with the "RAW conversion" because I do regard them as two separate steps. That said I'm reaching a point where I do want to start tagging images, and finding them more quickly. In the past I wrote a couple of simple tools to inject tags into the EXIF data of images, and then indexed them. But that didn't work so well in practise. I'm starting to think instead I should index images into sqlite: The downside is that this breaks utterly as soon as you move images around on-disk. Which is something my previous exif-manipulation was designed to avoid. Anyway I'm thinking at the moment, but I know that the existing tools such as F-Spot, shotwell, DigiKam, and similar aren't suitable. So I either need to go standalone and use EXIF tags, accepting the fact that the tags I enter won't be visible to other tools, or I cope with the file-rename issues by attempting to update an existing sqlite database via hash/size/etc.

Andrew Pollock: [life] Day 65: Playgroup, and the foil confetti play date

Zoe slept pretty well last night. She only woke up briefly at 4am because Cowie had fallen out of bed and she couldn't find her. Today was the last Playgroup of the term. Megan, her little sister and her Dad came as well to check it out, which was nice, because Zoe then had someone to actively play with in addition to me. After Playgroup, we went to the adjacent Bulimba Memorial Park with Megan, and then had some lunch at Grill'd. Megan's Dad wanted to do some work on their house while Megan's little sister napped, so I offered to give Megan a play date at our place. The plan was to watch a movie and chill out. The girls picked Ratatouille and I made a batch of popcorn for them. Unfortunately Megan seemed to be less of a square eyes than Zoe, and she lost interest after a bit, so we stopped watching the movie and moved out to the balcony to do some craft. Zoe had been wanting to make a crown for Mummy's boss for a while, so we made a couple of crowns with the hot glue gun. I had bought this bag of mixed craft "jewels" and it's probably the best single craft thing I've bought. Zoe loves gluing them onto everything. After that, Zoe pulled out the bag of coloured foil confetti. If the gems were the best thing I've bought, this would have to be the worst. So far, all it's done is leak in the drawer it's been stored in, and I've been avoiding using it because it was going to be messy. Today, Zoe wanted to glue it onto the outside of her cardboard box, so I decided to give in and embrace the mess, and boy, did we make a mess. It probably ended up being the longest bit of cooperative play the girls did. They'd alternate between handing each other a fistful of confetti while I applied globs of glue where directed. Probably about 10 percent of each handful ended up stuck to the rocket, so the balcony looked like quite a mess by the end of it all, but at least it was a dry mess, so I could just vacuum it all up. I suspect I'll be encountering dregs for quite a while, because I doubt it's stuck to the cardboard particularly well. After that, the girls played indoors for a bit, and watched a bit more of the movie, but Megan seemed to be scared of Anton Ego, so I think that was why it wasn't holding her attention. The other activity that the girls seemed to thoroughly enjoy was tearing around the living room squealing while they took turns at throwing a grapefruit-sized beach ball at me, and I threw it back at them. Jason came back to pick up Megan, and I started dinner. Not that long after Megan left, Sarah arrived to watch Zoe for me so I could go visit my cousin in hospital. I had dinner on the table pretty much as soon as she walked in the door, and headed out.

1 April 2014

Marco d'Itri: Real out of band connectivity with network namespaces

This post explains how to configure on a Linux server a second and totally independent network interface with its own connectivity. It can be useful to access the server when the regular connectivity is broken. This can happen thanks to network namespaces, a virtualization feature available in recent kernels. We need to create a simple script to be run at boot time which will create and configure the namespace. First, move in the new namespace the network interface which will be dedicated to it:
ip netns add oob
ip link set eth2 netns oob
And then configure it as usual with iproute, by executing it in the new namespace with ip netns exec:
ip netns exec oob ip link set lo up
ip netns exec oob ip link set eth2 up
ip netns exec oob ip addr add 192.168.1.2/24 dev eth2
ip netns exec oob ip route add default via 192.168.1.1
The interface must be configured manually because ifupdown does not support namespaces yet, and it would use the same /run/network/ifstate file which tracks the interfaces of the main namespace (this is also a good argument in favour of something persistent like Network Manager...). Now we can start any daemon in the namespace, just make sure that they will not interfere with the on-disk state of other instances:
ip netns exec oob /usr/sbin/sshd -o PidFile=/run/sshd-oob.pid
Netfilter is virtualized as well, so we can load a firewall configuration which will be applied only to the new namespace:
ip netns exec oob iptables-restore < /etc/network/firewall-oob-v4
As documented in ip-netns(8), iproute netns add will also create a mount namespace and bind mount in it the files in /etc/netns/$NAMESPACE/: this is very useful since some details of the configuration, like the name server IP, will be different in the new namespace:
mkdir -p /etc/netns/oob/
echo 'nameserver 8.8.8.8' > /etc/netns/oob/resolv.conf
If we connect to the second SSH daemon, it will create a shell in the second namespace. To enter the main one, i.e. the one used by PID 1, we can use a simple script like:
#!/bin/sh -e
exec nsenter --net --mount --target 1 "$@"
To reach the out of band namespace from the main one we can use instead:
#!/bin/sh -e
exec nsenter --net --mount --target $(cat /var/run/sshd-oob.pid) "$@"
Scripts like these can also be used in fun ssh configurations like:
Host 10.2.1.*
 ProxyCommand ssh -q -a -x -N -T server-oob.example.net 'nsenter-main nc %h %p'

Joey Hess: adding docker support to propellor

Propellor development is churning away! (And leaving no few puns in its wake..) Now it supports secure handling of private data like passwords (only the host that owns it can see it), and fully end-to-end secured deployment via gpg signed and verified commits. And, I've just gotten support for Docker to build. Probably not quite work, but it should only be a few bugs away at this point. Here's how to deploy a dockerized webserver with propellor:
host hostname@"clam.kitenet.net" = Just
    [ Docker.configured
    , File.dirExists "/var/www"
    , Docker.hasContainer hostname "webserver" container
    ]
container _ "webserver" = Just $ Docker.containerFromImage "joeyh/debian-unstable"
        [ Docker.publish "80:80"
        , Docker.volume "/var/www:/var/www"
        , Docker.inside
            [ serviceRunning "apache2"
                 requires  Apt.installed ["apache2"]
            ]
        ]
Docker containers are set up using Properties too, just like regular hosts, but their Properties are run inside the container. That means that, if I change the web server port above, Propellor will notice the container config is out of date, and stop the container, commit an image based on it, and quickly use that to bring up a new container with the new configuration. If I change the web server to say, lighttpd, Propellor will run inside the container, and notice that it needs to install lighttpd to satisfy the new property, and so will update the container without needing to take it down. Adding all this behavior took only 253 lines of code, and none of it impacts the core of Propellor at all; it's all in Propellor.Property.Docker. (Well, I did need another hundred lines to write a daemon that runs inside the container and reads commands to run over a named pipe... Docker makes running ad-hoc commands inside a container a PITA.) So, I think that this vindicates the approach of making the configuration of Propellor be a list of Properties, which can be constructed by abitrarily interesting Haskell code. I didn't design Propellor to support containers, but it was easy to find a way to express them as shown above. Compare that with how Puppet supports Docker: http://docs.docker.io/en/latest/use/puppet/
docker::run   'helloworld':
  image        => 'ubuntu',
  command      => '/bin/sh -c "while true; do echo hello world; sleep 1; done"',
  ports        => ['4444', '4555'],
...
All puppet manages is running the image and a simple static command inside it. All the complexities that puppet provides for configuring servers cannot easily be brought to bear inside the container, and a large reason for that is, I think, that its configuration file is just not expressive enough.

Andrew Pollock: [life] Day 63: Productive procrastination, pizza dough, podiatrist, positive

parenting and a haircut with a spot of painting How's that for some alliteration? Today was a really good day. And that's before I started drinking red wine. I got up this morning and successfully banged out a 10km run. It wasn't pretty, but I did it in under an hour, so I was happy. I got home, and after breakfast I pretty much flopped on the couch with my laptop and procrastinated instead of doing my taxes. But it was productive procrastination. I: I also (finally) got my taxes to a point where I'm ready to send them off to my US accountant and deal with the rest of it incrementally. So it was a productive day! I had a follow up appointment with my podiatrist in the afternoon to see how my orthotics were going. I biked to Kindergarten early, ditched the bike trailer, and then biked over to the podiatrist, and made it back to Kindergarten about 10 minutes before pick up time. Zoe was, unsurprisingly, fast asleep. I decided to try applying sunscreen to her while she was asleep as a way of killing two birds with one stone. I got as far as getting her legs done before she woke up and had a massive meltdown. Poor kid really doesn't deal well with being woken up. One of the teachers took pity on us and distracted Zoe by letting her cuddle one of the baby chicks, which snapped her out of it for the duration, but she had another meltdown once it was over. Another teacher gave her a cuddle for a bit, and she eventually calmed down enough for me to get sunscreen on her arms. I'd foolishly left the bike trailer separated from the bike, so I had to drag the trailer back to the bike, whilst carrying Zoe. Fortunately another teacher took sympathy on us and helped me with the trailer. Turns out trying to drag a single-wheeled trailer single-handed whilst carrying a toddler and having excess sunscreen on my hands is extremely difficult. We finally got the trailer on the bike, and Zoe in the trailer, and headed towards home. Zoe's ballerina pumps aren't great on the bike because the straps on the pedals cross the tops of her exposed feet and irritate her, so there were multiple meltdowns on the way home, culminating in needing to go to the toilet "right now" before we got home. I stopped at the health food shop on the way home, to see if they had a toilet we could use. Luckily I'm a customer and the naturopath let us use the toilet in the clinic. Zoe had another meltdown in there, announcing she "didn't like being woken up". Poor kid. It wasn't a good afternoon for her. I'm just glad I was in a sufficiently good mood to be able to deal with it all in a satisfactorily positive parenting way. We finally made it home, and I'd promised her we could have a big cuddle on the couch once we got home, so we did that and read a library book, and then it was time to head to the hairdressers for our haircuts. We started out on foot, and had made it one block from home, and she saw another kid on a scooter and announced she wanted to ride her scooter too. Initially tried saying we couldn't do it this time, because we'd be late, but she was on the verge of having another meltdown, so I capitulated, and we went back and grabbed it. I'm actually really glad I did, because she was as happy as Larry from that point on, and we were only a couple of minutes late at the hairdresser. We did her fringe trim first, and then she had a great play in the kid's corner while I got my haircut. She even cleaned up the corner better than she found it without argument. We then had plenty of time to scooter back home, so I decided to check out the 'OO'niverse Family Cafe, which is next door to the Hawthorne Cinemas. It's this thing I've never gotten around to checking out, and it was the second thing I was glad I did this afternoon. It's not a big place, and it was super quiet. There was just us and two twin Kindergarten-aged girls being babysat. Zoe had a banana milkshake and got a dolphin painted on her arm and a balloon dog made, and I had a coffee and we just chilled out for a bit while I chatted with the owner (who was babysitting the twins). By this stage Anshu was already at my place, and Sarah wasn't far off leaving work to pick up Zoe, so we made our way back home. Zoe had a great time playing with Anshu until Sarah arrived. So I was basically really happy that I managed to turn around a massively molten afternoon and give Zoe a really good afternoon instead. Anshu and I then proceeded to made a couple of really fantastic pizzas. I really love my Thermomix. I made the pizza dough earlier today, and tonight I made some pizza sauce, some pesto sauce and caramelized some onions in it, and we were still done with dinner by 8pm.

31 March 2014

John Goerzen: Springtime in the Mountains

The scene: early one morning as the sun has just started to rise. Jacob and Oliver, ages 7 and 4, are the first people to wake up in the house their grandparents in California, where the four of us are visiting for the first time as a family. They have a conversation and decide that would be a good to go find a mystery. They decide to take their flashlights pink and blue, matching each boy s favorite color and slowly, but not very quietly, open their bedroom door and creep out. Brother, you forgot your flashlight! says Oliver. Oh, thanks brother! I ll get it! says Jacob. Meanwhile, Laura s mom wakes up, and notices two boys with flashlights creeping through the living room. Pretty soon they reach the kitchen, open the dishwasher, spy a suspicious-looking bowl, and decide that they have found the mystery a clean bowl! Or, at least that s the story that I pieced together based on what a 4-year-old and the grandma he awakened told me. We were on our first family trip to the Fresno, California area, to visit Laura s parents Jacob and Oliver s new grandparents. They ve played together before, but as this was our first visit to their place, there was quite the excitement. The boys had flown before, but it was several years ago and neither of them remember it well, so they were excited about that, too. The night before, Jacob woke up to tell me Dad, I am too excited to sleep. I think I will go downstairs and watch some TV. He didn t get too far with that plan. But he was excited. We went through security at the laid-back Wichita airport (where the TSA agents smile and there are often no security lines at all). We found our gate with enough time to grab lunch, which we did. The boys and I then did what we often do to kill time: explore. We explored the terminal, watching carpet-layers cut out carpet for the jetway, watching the construction of the new Terminal 3 out the window. And, of course, watching airplanes take off and land from the terminal s large windows. Finally it was our turn to board, and we all got on the plane: Jacob and Oliver with their backpacks of on-board activites, Laura and me with the rest of our carryon luggage, for the short trip to Denver. Jacob and Oliver s noses were pressed against the windows. Or, well, Jacob s was. Oliver s window was a little too high for him, but he was thrilled anyhow. They delighted in the airplane snacks, and the fact that they were allowed to drink pop on the plane. We packed books and some new art supplies for them (colored post-its, pages from a train-themed page-a-day calendar, a notebook, and a set of colored pens really seemed to do the trick.) We had a choice of 35 minutes or 4 hours between flights in Denver, and I had chosen 4 hours, thinking that would be a lot less stressful with boys. And it was. We found a nice corner of the mezzanine to sit for awhile they did art projects and played a game with Laura. Then I took them exploring Denver. We rode the moving sidewalks up and down the terminal, took a train ride to another terminal and back, ate supper all together, and flew to Fresno. We had stopped in the Wichita airport to buy them each a souvenir airplane, and these came out often during the rest of the trip. They enjoyed the mockups of the sequoias in Fresno Yosemite International Airport, enjoyed their beds and their room at the house, and did actually manage to fall asleep eventually. We had a few days there, where they played in a park, with bubbles on the patio, or croquet in the yard (I even discovered Jacob happily using the cast his broken arm is in as a hammer to pound the hoops into the ground!) There are a lot of miniatures in the house, and the boys enjoyed exploring the dollhouses and especially the N-gauge model train. Jacob enjoyed it so much he asked me to record a video of him playing with the trains. Evenings often brought book-reading, from the many children s books in the house. At home, Laura and I and both boys often scrunch onto an oversized chair and read a book and sing a song (one I make up on whatever topic they choose). Over there, we often had Laura, Jacob, Oliver, Laura s mom, and me scrunched up somewhere while the boys heard a story read to them by their grandma. That happened plenty of times other than bedtime, too. (Or Jacob would take his favorite books and read them to himself.) Laura s parents organized a reception Sunday for us, for the people from that area that couldn t make it to our wedding. Jacob and Oliver, predictably, had fun playing and even talked to some of the adults. The adults that didn t ask Jacob about his cast, anyhow (he dislikes talking about it). The boys discovered a live mic at the church where the reception was, and do I detect two future pastors in our midst? We had a great time at Laura s uncle and aunt s place. The boys were happy to discover an orange tree in their backyard, a tetherball post not far away, and an uncle ready to give them a demonstration of a swimming pool vacuum cleaner or sit at the piano with them. Jacob s favorite part, though, was when the hamburger buns his great uncle were toasting were left on the grill during the prayer before the meal, got a bit scorched, and the uncle remarked with a chuckle that I guess the Lord was tired of listening to me drone on! Jacob loved his meal, and cackled at the thought of a prayer causing buns to get scorched. But their highlight was the visit to the sequoias at Kings Canyon National Park the next day. The excitement had been building for that day all weekend. On the way out, we stopped at a fruit stand and bought some delicious strawberries the fresh, juicy, sweet and tasty kind that are red all the way through. We continued up through the foothills, stopping periodically to get out and stretch, look at the sights, take some photos, or borrow grandpa s binoculars. I knew we d be traveling in two cars, so I had the thought to pack some 2-way radios before we left. I gave one to the boys and one to the grandparents. All weekend long, whenever the six of us went somewhere, the boys (and especially Jacob) would give directions to the car that was following. Turn right! The light is green! Catch up, you re going too slow! So all the way into the mountains, Jacob would send back instructions on what to do. We saw Grant Grove, home to the worlds third-largest tree (267ft/81m tall and 3000 years old). It s quite the impressive tree the trunk s diameter near the ground is 29ft or almost 9m. As we walked the trails, their speed kept increasing as they were hunting for the tree tunnel I had told them about a tree that fell centuries ago and had been hollowed out to make a home. That trunk was easily 8ft or more in diameter, and I could stand up completely in places. We found it, to much delight from the boys So this is what it s like to be inside a tree! Our trip home brought a delay in Denver and a missed flight, which excited the boys when I told them now we get to eat supper in the airport! I wonder how long that tactic will work But Jacob was also excited because the plane we were put on in the end was bigger than the one we were scheduled on, so that was another piece of excitement. We got home, and I carried two sleeping boys in from the car, upstairs, tucked them in, pulled off their shoes, and put their favorite stuffed animals in their arms. They were happy to be home, and with memories to treasure for a long time.

28 March 2014

Matthias Klumpp: Appstream: The next step

appstream-logoWith the release of GNOME-Software 3.12, it s also time for another update about what is going on behind the scenes of Appstream, the project providing all the data needed to make software-centers work. So here is the long overdue update :) If you have attended my FOSDEM talk or seen the slides, you know about the concept of component metadata to describe (almost) all software components which make up a Linux system, as well as their public interfaces they provide for users and other software to access. This metadata specification was originally designed as part of the Listaller project for use with the 3rd-party software installer.
But we already have the Appstream specification, which describes applications available on a distribution. And technically, applications are nothing but specialized components.
When I was asked about extending the Appstream spec with fonts and codecs, I decided to give up the Listaller components concept and merge it with the Appstream project, effectively creating a single source of metadata for all software in distributions. Appstream and components were separate, because Appstream was designed just for application metadata shown in software-centers, while component-metadata was more technical and not user-visible. This separation, however, did not reflect reality. Software centers want to show things components were designed for (codecs, fonts, ), and application meta-information also can contain technical items not shown in a software-center, but useful for other functionality (e.g. for mimehandler finding). So, if you like clear naming of things (like I do), you can now think of the upcoming Appstream release 0.6 as Appstream not being a stream of application metadata, but rather metadata provided for use in applications. ;-) Extending Appstream with components has a number of benefits:
First of all, users will soon get systems which are able to automatically install missing firmware, codecs, fonts and even libraries or Python-modules. This happens in a distro-agnostic way, so any application can request installations of missing components through Appstream and PackageKit without having to care about distribution details. For distributors, you will get more information about the software upstream provides: Which public interfaces does it make available? What is the upstream s homepage? Summary? Short release-descriptions? Exposed library symbols? etc. This data was available before, of course, but we made it machine-readable and standardized now. This will also allow us to match software across distributions quickly, allowing efficient exchange of patches between distributions, and answering questions upstream has like in which distribuions is my software available, and in which version? . Tracking upstream software is also easier now. Last but not least, the metadata format gives upstreams a small say in how their software gets packaged (= how it gets split / if there is a metapackage depending on the split parts). For upstream authors, it is much easier to maintain just a few Appstream XML metafiles, instead of adding a Listaller component file as well (which was not XML). Most data for creating the metainfo files is already available, some can even be filled in automatically by the build-system. With all the advantages this new system has comes a small disadvantage: These changes broke the backwards-compatibility of the Appstream spec, so to use Appstream data in version 0.6 or higher, you need to adjust your parsers. But of course you are using a library to access the data anyway, which already does that for you ;-) GNOME-Software 3.12 has almost complete support for the new (but still not released) Appstream 0.6 spec through it s libappstream-glib library. The libappstream library in Appstream s Git branch will get 0.6 support as well very soon. The only thing you need to do is adjust your code for the new library version, which broke it s API as well due to a rewrite in C (I hope that we can provide language bindings again very soon).
Soon, there will also be a Qt library for accessing Appstream, since some people expressed a demand for it (and working with GObject in Qt C++ code always feels a litle bit odd).
The 0.6 specification is still being fine-tuned to iron out some issues which might come up and mainly to improve the documentation and adjust all tools. It will be released in 2-3 weeks, and bring not only components support, but also lots of other small improvements and additions for better description of applications/components. In some cases, tag names have changed to have less ambiguous names. Thanks to Richard Hughes for helping with the spec our discussions about the Appstream spec are highly productive and in the end lead to a better specification. Also, the new 0.6 specification includes many extensions used in GNOME-Software by default now, which makes it easier for other distributions to provide them for GS. The current draft for the Appstream 0.6 specifications can be found here. Some details might still change before the final release. As soon as the spec is final, there will be a ML announcement for distributors as well (watch the XDG distributions list), so there is enough time for distributors to future-proof their Appstream distro-data generators (I will also describe the changes more detailed then). If you are confused now about the whole components stuff: Fear not! ;-) I will do some more detailed posts in a few weeks explaining upstream projects how to provide metadata properly (after the 0.6 release). I will also finish the wiki page for Appstream AppData in KDE projects, where we can then use 0.6-style data in KDE right away.

Dirk Eddelbuettel: Upcoming Rcpp talks and workshops

A number of talks about Rcpp are scheduled over the next couple of months: So the month of June is still pretty open. If someone wants me to talk somewhere else, preferably on a Saturday, feel free to get in touch.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

26 March 2014

Steve Kemp: A diversion on off-site storage

Yesterday I took a diversion from thinking about my upcoming cache project, largely because I took some pictures inside my house, and realized my offsite backup was getting full. I have three levels of backups: Unfortunately my rsync-based personal server is getting a little too full, and will certainly be full by next year. S3 is pricy, and I don't trust the "unlimited" storage people (backblaze,etc) to be sustainable and reliable long-term. The pricing on Google-drive seems appealing, but I guess I'm loathe to share more data with Google. Perhaps I could dedicated a single "backup.account@gmail.com" login to that, separate from all-else. So the diversion came along when I looked for Amazon S3-comptible, self-hosted, servers. There are a few, most of them are PHP-based, or similarly icky. So far cloudfoundry's vlob looks the most interesting, but the main project seems stalled/dead. Sadly using s3cmd to upload files failed, but certainly the curl based API works as expected. I looked at Gluster, CEPH, and similar, but didn't yet come up with a decent plan for handling offsite storage, but I know I have only six months or so before the need becomes pressing. I imagine the plan has to be using N-small servers with local storage, rather than 1-Large server, purely because pricing is going to be better that way. Decisions decisions.

Next.

Previous.