Continuing in photo blog posts, here is our last vacation - in the beginning of September we've traveled around Slovenia, mostly for hiking, but we've enjoyed other things as well. Our first short hike was around lake Bled, which is nice lake placed in between mountains: Afternoon we had some time left, so we've walked around Peri nik waterfall, which was close to Mojstrana where we were accommodated (if you will be looking for accommodation there, check River Cottage): Then it came time for some serious hiking and we went up to the mountains, in the end we did not make it to Triglav because of weather, but we've gone on Cmir: After that we had to relax a bit and visited Ljubljana and ended up near kocjan Caves which we've visited on next day: After relaxing evening in Saksida winery, it was time to go back to mountains. We've started visiting Tolmin gorges, which is nice natural area worth short detour: Last days we've spent near Bovec and hiked around a bit, one of the things we've enjoyed most is trip to Krnsko lake: This was definitely not our last trip to Slovenia as there are still some places we want to visit :-).

Filed under: English Photography Travelling 2 comments Flattr this!

Here in the US, the Affordable Care Act is finally going into effect, with accompanying drama. I managed to get signed up today at healthcare.gov. After not having health insurance since 2000, I will finally be covered starting January 1 2014. Since my income is mosty publically known anyway, I thought it might be helpful to blog about some details. I was uninsured for 14 years due to a combination of three factors:

1. Initially, youthful stupidity and/or a perfectly resonable cost/benefit analysis. (Take your pick.)
2. Due to the US health insurance system being obviously broken, and my preference to avoid things that are broken. Especially when the breakage involved insurers refusing to cover me at any sane level due to a minor and easily controlled pre-existing condition.
3. Since I'm not much motivated by income levels, and am very motivated to have time to work on things that are important to me, my income has been on average pretty low, and perhaps more importantly, I have intentionally avoided being a full-time employee of anyone at any point in the past 14 years (have rejected job offers), and so was not eligible for any employee plans which were the only reasonable way to be covered in the US. (See point #2.)

So, if you're stuck waiting in line on healthcare.gov (is this an entirely new online experience brought to us by the US government?), or have seen any of the dozen or so failure modes that I saw just trying to register for a login to the site, yeah, it's massively overloaded right now, and it's also quite broken on a number of technical levels. But you can eventually get though it. Based on some of the bugs I saw, it may help to have an large number of email addresses and use a different one for each application attempt. It also wouldn't hurt to write some programs to automate the attempts, because otherwise you may have to fill out the same form dozens of times. And no, you can't use "you+foo@bar.com" for your email; despite funding the development of RFC-822 in the 80's, the US government is clueless about what consititutes a valid email address. But my "favorite" misfeature of the site is that it refuses to let you enter any accented characters, or characters not in the latin alphabet when signing up. Even if they're, you know, part of your name. (Welcome back to Ellis Island..) I want to check the git repository to see if I can find the backstory for these and other interesting technical decisions, but they have forgotten to push anything to it for over 3 months. The good news is that once you get past the initial signup process, and assuming you get the confirmation mail before the really short expiration period of apparently less than 1 hour (another interesting technical choice, given things like greylisting), the actual exchange is not badly overloaded, and nor is it very buggy (comparatively). I was able to complete an application in about an hour. The irony is that after all that, I was only able to choose from one health insurer covering my area on the so-called "exchange". (Blue Cross/Blue Shield) I also signed up for dental insurance (it was a welcome surprise that the site offers this at all) and had a choice of two insurers for that. The application process was made more uncertian for me since I have no idea what I'll end up doing for money once my current crowdsourced year of work is done. The site wants you to know how much income you'll have in 2014, and my guess is anywhere between $6000 (from a rental property) and about what I made this year (approx $25000 before taxes). Or up, if I say, answered the Google pings. The best choice seemed to be to answer what I made this year, which is also close to what I made last year. So, I'll be paying around $200/month for a combination of not very good health insurance, and not very good dental insurance. There is around $750/year of financial aid to people at my guesstimated 2014 income level, which would drop that to $140/month, but I will let them refund me whatever that turns out to be in a lump sum later instead. For comparison, I am lucky to spend rather less renting a three bedroom house situated in 25 acres of woods.. It's strange to think that all of this is an improvement to the system here in the US, especially given all the better options that could have been passed instead, but it seems that it probably is. Especially when I consider the many people around me who are less fortunate than myself. If you'd like a broader perspective on this, see Tobias Buckell's "American healthcare was already socialized by Reagan, we re just fighting about how to pay for it".

Download links often point to CGI scripts which actually generate (or just fetch, i.e. proxy) the actual file to be downloaded, e.g. URLs like http://www.example.com/download.cgi?file=foobar.txt. Most of such CGI scripts send the real file name in the Content-Disposition header as specified in the MIME Specification. All browsers I know (well, at least those I use regularily :-) handle that perfectly and propose the file name sent in the Content-Disposition header as file name for saving the downloaded name which is usually exactly what I want. All browsers do that, , just not my favourite commandline download tool GNU Wget Downloading the above URL with wget would look like this with default settings:

$ wget 'http://www.example.com/download.cgi?file=foobar.txt'
--2013-10-02 16:04:16--  http://www.example.com/download.cgi?file=foobar
Resolving www.example.com (www.example.com)... 93.184.216.119, 2606:2800:220:6d:26bf:1447:1097:aa7
Connecting to www.switch.ch (www.example.com) 2606:2800:220:6d:26bf:1447:1097:aa7 :80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2020 (2.0K) [text/plain]
Saving to:  download.cgi?file=foobar.txt'
100%[============================================>] 2,020       --.-K/s   in 0s
2013-10-02 16:04:24 (12.5 MB/s) -  download.cgi?file=foobar.txt' saved [2020/2020]

Meh! But luckily Wget can do that, it s just not enabled by default because it s an experimental and possibly buggy feature, at least according to the man page. Well, works for me! :-) You can easily enabled it by default for either your user or the whole system by placing the following line in your ~/.wgetrc or /etc/wgetrc:

content-disposition = on

Given the CGI script sends an appropriate Content-Disposition header, the above output now looks like this:

$ wget 'http://www.example.com/download.cgi?file=foobar.txt'
--2013-10-02 16:04:16--  http://www.example.com/download.cgi?file=foobar
Resolving www.example.com (www.example.com)... 93.184.216.119, 2606:2800:220:6d:26bf:1447:1097:aa7
Connecting to www.switch.ch (www.example.com) 2606:2800:220:6d:26bf:1447:1097:aa7 :80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2020 (2.0K) [text/plain]
Saving to:  foobar.txt'
100%[============================================>] 2,020       --.-K/s   in 0s
2013-10-02 16:04:24 (12.5 MB/s) -  foobar.txt' saved [2020/2020]

Now Wget does what I mean! You can also set this as flag on the commandline, but typing wget --content-disposition everytime is surely not what I want. ;-)

This week's lesson on the Arte y cultura en circulaci n: crear y compartir en tiempos digitales course talks about piracy and the circulation of culture, a topic that over time has been debated over and over. And a topic, yes, that can always lead to interesting discussions. This time, we are requested to choose one among ten ideas among the media groups' discourse on what piracy is and means for the "cultural industry". There are tons of material written already on several of those ten lines (i.e. piracy disincentivates creativity, or two that can be seen as two faces of the same argument, If a consumer can have free access to cultural products, he will stop spending his money on them and Every time a consumer has access to an illegal copy, the industry loses a sale), and some are quite obvious (i.e. Piracy makes job positions be lost... Just look at the amount of people the unauthorized distribution industry feeds! Or possibly, Piracy is a prosper industry that gives money to people distributing illegal products Of course, that is true. The problem is, what causes said products to be illegal to begin with? Should they be so?). Some other ideas talk about harsher penalties and ways to punish illegal copying in order to drive actors out of that sector (and into the... void?) So, I chose item #4: Cultural products have a high cost because their production is complex (and a tag could be made, linking complex with expensive). I think this item can lead to a long discussion as to what does this complexity and cost mean. Some cultural products do require quite a bit of investment, yes. Others don't. How do content producers make the jump to produce expensive works? If I am a new programmer/artist/writer/screenplayer/whatever, most likely, my products will be not very complex or expensive. I will start small. And if I excel at my work, somebody will look at me and, in some way, become my patron, my sponsor. Being a sponsor might mean that, based on the results of my good work, I could get hired as a software developer at a large company, or an editorial company would buy the patrimonial rights of my book/music (be it for a fixed fee or for a percentage of sales), or whatever. But the leap is not made quantically A newcomer to the cultural scene will at first, most likely, have a hard time selling his products. At first, it takes convincing just getting people to take a shot at looking at your work ( Hey, please take a look at my program and tell me what you think about it! , Would you be interested in listening to my latest song? And those two are by far ahead of the first attempts where the interactions would more likely be Turn off that $#^#!^ computer, it's well past bedtime or stop murdering that guitar, I'm having a headache ). Maybe the toughest part is to get people to agree to read/hear your work. And there, you start into a continuum Selling your CDs while performing on the street, then getting to play to a bar, then getting somebody to want to produce (maybe even "discover"!) you. Publish some short stories in your school magazine, then in a "From our audience" section in a larger magazine, then a collective book, your self-published book, yet-unwritten books by contract... The same story over and over again, in each different field. Ok, yes, but... This logic succession still leaves space for the Important Producers with the Mighty Big Pockets for the most wanted/largest productions, right? And were unauthorized distribution (piracy) to be the norm (as it currently is, dare I say), wouldn't they stop producing an important portion of cultural works? I'd be tempted to say so. However, a different actor comes into play. When Mighty Big Pockets comes into play, they no longer worry only about getting money from each cultural creation, but from all derived uses of it. And the cultural creation industry (when seen as an industry) goes very much hand in hand with the advertising, marketing industries They end up blending with each other. So, the biggest best sellers will most likely have a hit from illegal copiers. Books are still a great business, but hey An even better business is (usually) movie making. And when you make a movie out of a great story, you will surely link some advertising into it (or at the very least, push advertising/product pushing campaigns to go after it). And there, illegal distribution actually helps the money circle to grow stronger. In the early 1990s, the link between dinosaurs and carbonated drinks was a top seller (because Pepsi was a Jurassic Park sponsor). Although I have always loathed the madness around the World Cups (and basically anything that involves football of any kind), I can perfectly remember several of the theme songs for most of the world cups played during my lifetime. So, in short... No. Illegal distribution does marginally little harm to the money income to the cultural business, at any level. And where it does get some direct harm, it increases the money flux given the auxiliary channels.

After 1.5 month of work, I am proud to officialy release OUnit 2.0.0. This is a major rewrite of OUnit to include various feature that I think was missing from OUnit1. The very good news is that the port of the OASIS test suite has proven that this new version of OUnit can drastically improve the running time of a test suite. OUnit is a unit test framework for OCaml. It allows one to easily create unit-tests for OCaml code. It is based on HUnit, a unit testing framework for Haskell. It is similar to JUnit, and other XUnit testing frameworks. Download OUnit v2.0.0 Documentation of v2.0.0 Website The basic features:

• better configuration setup
  • environment variable
  • command line options
  • configuration files
• improved output of the tests:
  • allow vim quickfix to jump in the log file where the error has happened
  • output HTML report
  • output JUnit report
  • systematic logging (verbose always on), but output log in a file
• choose how to run your test:
  • run tests in parallel using processes (auto-detect number of CPU and run as many worker processes)
  • run tests concurrently using threads
  • use the old sequential runner
• choose which test to run with a chooser that can do smart selection of tests:
  • simple: just run test in sequence
  • failfirst: run the tests that failed in the last run first and skip the success if they are still failing
• some refactoring:
  • bracket: use a registration in the context but easier to use
  • remove all useless functions in the OUnit2 interface
• non-fatal section: allow to fail inside non-fatal section without quitting immediately the whole test
• allow to use OUnit1 test inside OUnit2 (smooth the transition)
• timer that makes tests fail if they take too long, only when using the processes runner (I was not able to do it cleanly using threads and sequential)
• allow to parametrize filenames so that you can use OUNIT_OUTPUT_FILE=ounit-$(suite_name),log and have $(suite_name) replace by the test suite name
• create locks to avoid accessing the same resources within a single process or the whole application (typically to avoid doing a chdir while another thread is doing a chdir elsewhere)
• create a in_testdata_dir function to locate test data, if any

Migration path OUnit 2.0.0 still provides the OUnit module which is exactly the same as the last OUnit 1.X version. This way, you are not forced to migrate. However, this means that you will gain no advantage of the new release and even some slowdown due to increase complexity of the code. Though, I strongly recommend to upgrade to OUnit2. Here is a checklist to do the migration:

• replace all open OUnit by open OUnit2
• the test function now takes test_ctxt argument, so replace all fun () -> ...) by fun test_ctxt -> ...
• bracket are now inlined so bracket setUp f tearDown is now let x = bracket setUp tearDown test_ctxt in
• make sure that you don't change global process state like chdir or Unix.putenv or that you don't rely on another test setting something for the next test

The OASIS test suite migration In order to check that everything was working correctly, I have migrated the OASIS test suite to OUnit2. This is a big test suite (210 test cases) and it includes quite big sequences of tests (end to end tests from calling oasis setup to compiling and installing the results). This was really time consuming and I wish to see a significant speedup for the tests with OUnit2. You can see the result in term of code of the full migration here. Here are the results on my Intel Core i7 920/SSD:

• Pristine test suite (210 tests):
  • oUnit v1: 52.36s (i.e. latest OUnit v1.x, reference time)
  • oUnit1 over oUnit2: 60.39s (OUnit v2.0.0 using the OUnit v1 layer)
• Migration to OUnit2 (166 tests):
  • processes (8 shards): 10.12s
  • processes (autodect, 4 shards): 12.99s
  • sequential: 58.77s

The migration was quite heavy because this test suite had a big design problem. It uses in-place modification of the test data. I think I pick this design because I thought this was a good idea to decrease the running time. As a matter of fact, this was a huge mistake, that keeps popping failed test cases because one of the previous tests was failing. I have refactored all this and now we start by copying the test data into a temporary directory, which ensure that everything is always starting from pristine test data. During the redesign, I have decided to reduce the number of tests by merging some of them. This should have no big impact on the running time, although this is not as pure 1:1 comparison with OUnit v1. Although it is still testing exactly the same thing. This explain the loss of ~50 tests, which in fact has been merged in other tests. The overall speedup is 4x compared to OUnit v1, when using processes. However there is a 12% increase compared to OUnit v1 for sequential and a 15% increase when using OUnit v1 compatibility layer. While this is not very good score, I hope this is small enough to compensate the huge win of being able to run tests in parallel with processes. And now the magic ! At this point, if you read carefully the numbers, you would have noticed that there is a 4.5x decrease in speed when you compare sequential and 4 processes for OUnit2. Since we are only actively testing in 4 shards, it looks strange. I don't expect a super-linear speedup due to the use of processes. I have checked that every tests was indeed running and found no solutions to this mystery. Right now, I think this is due to the fact that we are running less tests in more processes which should lighten the load of the GC (which may not trigger at all). I am not sure about this explanation and will welcome any bug report which shows a problem in the implementation of either sequential or processes runner. Although, this is great. Help still wanted If you find any bugs with OUnit v2, this is the time to submit a bug. OUnit BTS If you want to try to fix bugs by yourself, please checkout the latest version of OUnit:

 $> darcs get http://forge.ocamlcore.org/anonscm/darcs/ounit/ounit

Patches always welcome.

Matt Palmer wrote an insightful post about the use of the word professional [1]. It s one factor that makes me less inclined to be a member of professional societies. The TED blog has an interesting article about Wikihouse which is a project to create a set of free designs for houses to be cut out of plywood with a CNC milling machine [2]. The article also links to a TED talk by Alastair Parvin of the Wikihouse project which covers many interesting things other than designing houses. An XKCD comic has one of the best explanations of bullying I ve ever seen [3]. If you aren t familiar with XKCD then make sure you hover your mouse over it to read the hidden text. The Fair Phone is a project to develop a smart phone starting with conflict-free resources and with fully free software (not like a typical Android build) [4]. It s an interesting project and the price and specs seem within the normal range so you re not paying a huge premium for a conflict-free phone. Unfortunately they only have one model with a 4.3 display, if they had a competitor for the Galaxy Note then I d be interested. Patrick Stokes wrote an insightful article about why I m entitled to my opinion is a bogus argument [5]. Jim Daly wrote an interesting TED blog post interviewing Rishi Manchanda about Upstream Doctors who look for the root causes of medical problems rather than just treating the symptoms [6]. Brian Krebs wrote an insightful article about the value of a hacked email account [7]. If you are trying to convince your users to use better passwords then this should help. Ron Garrett wrote an insightful series or articles on morality hooked on the premise of whether it s wrong to torture kittens [8]. Part of his conclusion is that people who believe it s wrong to do such things tend to be more capable of working in large groups and forming a productive and efficient society. The TED blog has an interesting post by Karen Eng summarising Andreas Raptopoulos talk about using autonomous drones to deliver parcels in parts of the world that don t have usable roads [9]. Delivering parcels (which would start with medical supplies but would presumably move on to commercial transport) by drone is apparently really cheap. Being cheaper than building roads isn t going to be difficult but it seems that they are going to make it cheaper than paying people to deliver parcels even if the roads were built. The main web site about this project is www.matternet.us, they are hiring electrical engineers. Here is the link for Andreas TED talk [10]. The TOR blog has an interesting article by Emily Asher-Perrin comparing the different houses of Hogwarts [11]. It s an insightful article about personality attributes and gives more information than is available in the movies (I d read the books if I had time).

• [1] http://hezmatt.org/~mpalmer/blog/2013/05/14/a-modest-vocabulary-proposal.html
• [2] http://blog.ted.com/2013/05/23/how-to-print-out-your-own-house/
• [3] http://xkcd.com/1216/
• [4] http://www.fairphone.com/
• [5] http://theconversation.com/no-youre-not-entitled-to-your-opinion-9978
• [6] http://tinyurl.com/n2rvo94
• [7] http://krebsonsecurity.com/2013/06/the-value-of-a-hacked-email-account/
• [8] http://tinyurl.com/nczyspd
• [9] http://tinyurl.com/kxu8dxp
• [10] http://www.youtube.com/watch?v=eqcX3AM5oh8
• [11] http://tinyurl.com/o4sbt3l

By popular demand: (source: myeerah's avatar on Why, God, Why? forum. I guess (s)he's the author)

A new version of Rcpp is now on the CRAN network for GNU R; binaries for Debian have been uploaded as well. Once more, this release brings a large number of exciting changes to Rcpp. Some concern usability, some bring new features, some increase performance; see below for the detailed list. We have now released three updates on a quarterly cycle; if we keep this up the next version ought to be ready at the end of December. As in the past, we tested the release rather rigorously by checking against all packages I could (relatively easily) built on my server: this time it successfully passed \code R CMD check for all 107 packages I can build locally out of a total of 136 packages. (Two failed: one for an error in \code Makevars , and one for the need of an X11 server during tests; this may get addressed in the in test script next time). As all of these 107 packages passed, we do not expect any issues with dependent packages. Should there be issues we would appreciate a note, preferably with reproducible code, to the rcpp-devel mailing list. The complete NEWS entry for 0.10.4 is below; more details are in the ChangeLog file in the package and on the Rcpp Changelog page.

Changes in Rcpp version 0.10.5 (2013-09-28)

• Changes in R code:
  • New R function demangle that calls the DEMANGLE macro.
  • New R function sizeof to query the byte size of a type. This returns an object of S3 class bytes that has a print method showing bytes and bits.
• Changes in Rcpp API:
  • Add defined(__sun) to lists of operating systems to test for when checking for lack of backtrace() needed for stack traces.
  • as<T*>, as<const T*>, as<T&> and as<const T&> are now supported, when T is a class exposed by modules, i.e. with RCPP_EXPOSED_CLASS
  • DoubleVector as been added as an alias to NumericVector
  • New template function is<T> to identify if an R object can be seen as a T. For example is<DataFrame>(x). This is a building block for more expressive dispatch in various places (modules and attributes functions).
  • wrap can now handle more types, i.e. types that iterate over std::pair<const KEY, VALUE> where KEY can be converted to a String and VALUE is either a primitive type (int, double) or a type that wraps. Examples :
    • std::map<int, double> : we can make a String from an int, and double is primitive
    • boost::unordered_map<double, std::vector<double> >: we can make a String from a double and std::vector<double> can wrap itself
    Other examples of this are included at the end of the wrap unit test file (runit.wrap.R and wrap.cpp).
  • wrap now handles containers of classes handled by modules. e.g. if you expose a class Foo via modules, then you can wrap vector<Foo>, ... An example is included in the wrap unit test file.
  • RcppLdFlags(), often used in Makevars files of packages using Rcpp, is now exported from the package namespace.
• Changes in Attributes:
  • Objects exported by a module (i.e. by a RCPP_MODULE call in a file that is processed by sourceCpp) are now directly available in the environment. We used to make the module object available, which was less useful.
  • A plugin for openmp has been added to support use of OpenMP.
  • Rcpp::export now takes advantage of the more flexible as<>, handling constness and referenceness of the input types. For users, it means that for the parameters of function exported by modules, we can now use references, pointers and const versions of them. The file Module.cpp file has an example.
  • No longer call non-exported functions from the tools package
  • No longer search the inline package as a fallback when loading plugins for the the Rcpp::plugins attribute.
• Changes in Modules:
  • We can now expose functions and methods that take T& or const T& as arguments. In these situations objects are no longer copied as they used to be.
• Changes in sugar:
  • is_na supports classes DatetimeVector and DateVector
• Changes in Rcpp documentation:
  • The vignettes have been moved from inst/doc/ to the vignettes directory which is now preferred.
  • The appearance of the vignettes has been refreshed by switching to the Bistream Charter font, and microtype package.
• Deprecation of RCPP_FUNCTION_*:
  • The macros from the preprocessor_generated.h file have been deprecated. They are still available, but they print a message in addition to their expected behavior.
  • The macros will be permanently removed in the first Rcpp release after July 2014.
  • Users of these macros should start replacing them with more up-to-date code, such as using 'Rcpp attributes' or 'Rcpp modules'.

Thanks to CRANberries, you can also look at a diff to the previous release 0.10.4. As always, even fuller details are on the Rcpp Changelog page and the Rcpp page which also leads to the downloads, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page

The BBC coverage of the UCI Women Road Race World Championship wasn t starting until 3pm, BBC Radio 5 Live had football and Sports Extra was playing an advert loop (really BBC?), Eurosport wasn t covering the race at all, RAI Sport 2 had coverage which was fine while I was watching the TV, but my Italian isn t good enough to follow the commentary and I wanted to get some other stuff done. So the obvious thing is to have the computer watch for changes to the great http://live.cyclingnews.com/ ticker and read them out, right? Well, it was to me. Here s the script I used:

#!/usr/bin/rc
# Send updates to live cycling news reports to the speech synth
# initialise
url=http://live.cyclingnews.com/
if (~ $1 *:*)  
  url=$1
 
last=""
while (true)  
    # download the update and strip html - if you don't have html2text try
    # sed -e '1,/<ol/d;s/<br[^>]*>/n/g;s/<[^>]*>//g;s/ / /g;s/[[:space:]]*$//'
    next= () curl -s $url   html2text -width 9999   sed -e 's/^ *[0-9]*. //'  
    # look for some added lines in the update section - timezone will need changing for non-European races
    say= () diff -u < echo $last  < echo $next  
        sed -ne '/^+/!d;s///;s/**//g;/ CES*T/,/^ *$/p'   head -9  
    # feed them to the speech synth in reverse order
    echo $say '
      Race Update'   tac   spd-say -e
    # update variables, pause and loop
    last=$next
    # set this to a prime number over 60, like 61, 67, 71, 73, 79, 83, 89
    sleep 67
 

Any questions, comments or improvements? I could have done more, like not saying Race Update even when there was no update, but I wanted to start listening as soon as possible!

---

• Mahout k-Means took 51 minutes on average per iteration (The raw numbers are 38:24, 62:29, 46:21, 56:20, 52:15, 45:11, 57:12, 44:58, 52:01, 50:26, so you can see there is a surpisingly high amount of variance there).
• ELKI k-Means on a single CPU core took 4 minutes 25 seconds per iteration, and 45 minutes total, including parsing the input data from an ascii file. Maybe I will try a parallel implementation next.

---

• ELKI is quite well optimized. The Java Hotspot VM does a good job at optimizing this code, and I have seen it to be on par with R's k-means, which is written in C. I'm not sure if Mahout has received a similar amount of optimization yet. (In fact, 15% of the Mahout runtime was garbage collection runtime - indicating that it creates too many objects.)
• ELKI can use the data in a uniform way, similar to a column store database. It's literally crunching the raw double[] arrays. Mahout on the other hand - as far as I can tell - is getting the data from a sequence file, which then is deserialized into a complex object. In addition to the actual data, it might be expecting sparse and dense vectors mixed etc.
• Size: this data set fits well into memory. Once this no longer holds, ELKI will no longer be an option. Then MapReduce/Hadoop/Mahout wins. In particular, such an implementation will by design not keep the whole data set in memory, but need to de-serialize it from disk again on each iteration. This is overhead, but saves memory.
• Design: MapReduce is designed for huge clusters, where you must expect nodes to crash during your computation. Well, chances are that my computer will survive 45 minutes, so I do not need this for this data size. However, when you really have large data, and need multiple hours on 1000 nodes to process it, then this becomes important to survive losing a node. The cost is that all interim results are written to the distributed filesystem. This extra I/O comes, again, at a cost.

---

A friend is teaching a class on data analysis. She is building a simple and rough data set for the class to examine, and to spur discussion. You can contribute in half a minute! Here's how:

1. get a stopwatch or other sort of timer (whatever device you're reading this on probably has such a thing).
2. start the timer, but don't look at it.
3. wait for what you think is 30 seconds, and then look at the timer
4. how many actual seconds elapsed?

The data doesn't need to be particularly high-precision (whole second values are fine). The other data points my friend is looking for are age (in years, again, whole numbers are fine) and gender. You can send me your results by e-mail, (i suspect you can find my address if you're reading this blog). Please put "half a minute for science" in the subject line, and make sure you include:

• actual seconds elapsed
• age in years
• gender

Science thanks you!Tags: science

As a maintainer of several software packages I often find myself copying text snippets from the README file into different places (savannah, github, freecode, emails, etc). Recently I had a need to generate a list of software packages that included every project s name, brief summary, license and URL. I could have generated that list manually by copying the text from every project s README and COPYING files. However then I would have to maintain my list manually to keep it in sync with all the projects. This easily leads to stale information, so I thought a better approach would be to put the information I needed into each projects source code version control system. The advantage is that the manual work to extract the information may be automated by a script, since the data is in a usable format. I ll explain here how my solution works. To be able to find the information using only the URL to the repository, I needed a filename convention. The filename I chose was BLURB; for the etymology, see Wikipedia s page about Blurb. The data format in the file is similar to normal email headers. An example illustrate the principles well. Below is the BLURB file for one of my projects.

Author: Yubico
Basename: libyubikey
Homepage: http://opensource.yubico.com/yubico-c/
License: BSD-2-Clause
Name: Yubico C low-level Library
Project: yubico-c
Summary: C library for manipulating Yubico YubiKey One-Time Passwords (OTPs)

The format is simple: UTF-8 text with each line starting with a header followed by a colon ( : ), some whitespace, and some content. If a line starts with whitespace, it is a continuation of the previous line s content (trim leading whitespace). The following table describes the fields that I use. I may update this blog post in the future with new fields or improved explanations (for reference, current date is 2013-09-24).

Header	Meaning
Project	Short identifier for the project (e.g., gcc , emacs )
Name	Official name of the project in English
Summary	Brief one-line summary of the project s purpose in English
Author	Origin of the project
Homepage	URL to the project s website
License	License keyword, preferrably using one of the SPDX license identifiers
Basename	The tarball basename, if different from the project name

Finally some reflection of the solution. After quick design, I thought that I couldn t be the first one with this problem, and I tried to find other similar efforts. I haven t been able to find any standardization effort that have the following properties:

• Stores the information inside each upstream project s own source code repository
• Provides a filename convention so that it is possible to find the data with only the source code repository link
• Encode data in a format that is easy to extract using simple command line tools
• Not encode information about releases (i.e., what happened in a particular version)

The related efforts that I found were SPDX which at first look seemed to offer what I wanted. However on closer examination it failed to deliver on all the requirements above, and appeared to have somewhat different goals. However I found the SPDX license list useful and refer to it. Another effort is Eric S. Raymond s freecode-submit and shipper but its primary focus is to encode information about each release. The design of the BLURB file is clearly influenced by these tools. Another influence has been Debian s specification for machine-readable copyright information. The Free Software Foundation s list of software projects seemed like another candidate, but it doesn t suggest any way to store the information in the upstream project itself.

Like it or not, XML has been used everywhere, even in cases where text-based formats would have been sufficient. Unfortunately, standard tools such as grep, sed or awk are not really adapted to work with XML. Let us take the following example:

<chapter
    xmlns="http://docbook.org/ns/docbook" version="5.0">
    <title>The Debian distribution</title>
    <para>Debian is a free operating system, describing itself as  the
    universal operating system . It is mostly known as a GNU/Linux
    distribution, but it also exist in other variants such as GNU/Hurd
    and GNU/kFreeBSD </para>
</chapter>

PYX and xml2 There are at least two line-oriented alternative formats for XML:

• PYX is an even-oriented format derived from an SGML subset, which can be used with the tool XMLStarlet,
• xml2 is a tool that can transform XML to a content-oriented format.

This is what our example would like in PYX:

(chapter
Aversion 5.0
-\n
(title
-The Debian distribution
)title
-\n\n
(para
-Debian is a free operating system, describing itself as
- the\n    universal operating system . It is mostly known as a GNU/Linux\n    distribution, but it also exist in other variants such as GNU/Hurd\n    and GNU/kFreeBSD 
)para
-\n
)chapter

And in the xml2 format:

/chapter/@xmlns=http://docbook.org/ns/docbook
/chapter/@version=5.0
/chapter/title=The Debian distribution
/chapter/para=Debian is a free operating system, describing itself as  the
/chapter/para=    universal operating system . It is mostly known as a GNU/Linux
/chapter/para=    distribution, but it also exist in other variants such as GNU/Hurd
/chapter/para=    and GNU/kFreeBSD 

Examples of use We want to extract the DocBook version number. This is not easy to do in a reliable way using the XML directly, but it appears directly with xml2:

$ xml2 < chapter.xml   grep '^/chapter/@version=' \
      cut -d= -f2
5.0

We want to move the title into an info tag, using PYX:

$ xmlstarlet pyx chapter.xml   sed -e '/^(title$/i\
(info
/^)title$/a\
)info'   xmlstarlet p2x
<chapter version="5.0">
    <info><title>The Debian distribution</title></info>
    [ ]

We could go further, adding a keywords entry in that info tag for instance, but you get the idea: when you want to work with XML in a reliable way, try xmlstarlet pyx or xml2.

I ve just been given a set of Hive Bluetooth speakers by MobileZap (see this link for all their Bluetooth speakers) [1]. The speakers charge by a micro-USB cable so I started charging them in my car immediately after collecting them. To connect them to a phone or other Bluetooth device you just press the Bluetooth button on top and get the phone to be visible and scanning for devices, they identify themselves as Hive , after that they just work. My first test of using them was playing Ingress and the quality of the sound was impressive, I had thought that the Ingress recommendation to use headphones was due to the risk of annoying other people or alerting other players, but the quality of the sound was impressive and the internal speakers of a phone can t do it justice. After getting home I did some tests listening to music. For watching music videos it didn t work so well as the sound was too far removed from the video, but the audio quality was very good. I listened to Vow by Garbage (a good benchmark for stereo sound) and even though the Hive speakers are only 16.5cm wide I could still notice the stereo effect when they were about 1.5m away from me. The audio quality didn t compare well with my Bose QC-15 headphones, but for affordable and portable speakers it was quite good and an obvious improvement over the speakers that are built in to any phone I ve used. According to the Bluetooth Wikipedia page the range of a class 2 device is 10m and the range of a class 3 device is 1m. When my Samsung Galaxy Note 2 is talking to it I get a reliable range of about 5 meters and a mostly working range of 6 or 7 meters (sound randomly drops out and gets choppy). It could be that other phones would support a longer range due to having a higher transmission power (either class 1 or being closer to the limits of class 2) and a more sensitive receiver. But it doesn t seem likely that a 5m range is going to be a problem. Volume and Quality The speakers are rated at 5 Watt, when running at maximum volume (both through the phone volume setting and the volume control on the speakers) the sound is reasonably distortion free, as good as can be expected from playing an MP3 that s not compressed with the highest quality. Sound Meter [2] reports the sound volume as almost 85dB on a Galaxy S3 and as almost 100dB on a Galaxy Note 2, that would be somewhere between the volume of a busy street or alarm clock and the volume of a subway train or blow dryer which seems like a reasonable description, I find it very unpleasant to be within a meter of the speakers at maximum volume. With the typical amount of background noise in my house I can play music on the Hive speakers at one end of my house and hear it clearly at the other end. These speakers are more than capable of supplying the music for any party I d want to host or attend. I m not really into wild parties, but I think that anyone who has a one room party would be more than satisfied with the Hive speakers. Obviously the sound quality of portable speakers in a box that s 16.5cm wide and 6cm high isn t going to equal that of a full size set of speakers, but I think that hardly anyone who attends a party would expect better sound quality than the Hive speakers can provide. The aim of such speakers is to be portable, not really expensive, and to provide good sound quality within those constraints. I think that they meet such aims well. Over the years there have been many occasions when I have used a Thinkpad to provide the music for a party and found it to be quite loud enough. My current Thinkpad is a T420 which can produce 75dB according to my Galaxy S3 or 85dB according to my Galaxy Note 2. So it seems that I only really need about 10dB less than the maximum volume of the Hive speakers. Appearance The designers obviously made an effort on the appearance of the device. They have gone with the Hive concept and used hexagons everywhere. It really looks nice. Unfortunately when I took the photo there was some dust on it which didn t look bad to the eye but caught the camera flash. But with a matte black device there s always the problem of light colored dust. Even with a bit of dust it still looks great as a set of speakers, the dust just detracts from the appearance in photos. Line In One of the features I looked for was an audio line input so I could connect it directly to a non-Bluetooth device. I m assuming that this feature works as it s something that s difficult to stuff up when designing such a product, but I haven t got around to testing it. Once I started using the device I just found that I didn t have a real need for that feature. One thing that it might be useful for is PC desktop speakers that are powered by a USB port on the monitor. Currently I have a bearable (but not great) set of speakers for each PC and I don t need to change anything. But having the option of another set of speakers is very handy in case I suddenly need to make hardware changes. Other People s Reviews When I review a product I generally try and get opinions from random other people if possible. My mother and my mother-in-law were both impressed by the Hive speakers and expressed interest in owning a set. My mother-in-law was particularly interested as she uses her phone to listen to radio stations from outside Australia (I m going to get her onto Aldi for cheap 3G data ASAP so she can listen to Internet radio when travelling). Generally the impression that other people have of this device seems to be very positive. It seems that Bluetooth speakers aren t just a Geek toy. Conclusion While I m very impressed by this product, at this stage I m not sure whether I would pay for this one or something cheaper if I was paying for it. MobileZap offers a range of other products that look appealing at lower price points. It really depends on how much I use it. I ve just got a Makerbot Replicator 3D printer working and I ve found the Hive speakers very useful for the purpose of drowning out it s noise. If I keep doing that sort of thing then I ll get enough use out of the speakers to justify the price.

• [1] http://www.mobilezap.com.au/10059/stereo-speakers.htm
• [2] https://play.google.com/store/apps/details?id=kr.sira.sound

The third lesson of the Arte y cultura en circulaci n: crear y compartir en tiempos digitales course was quite easy, at least for me, as it's one of the topics I've been involved with for a long time: Authors' rights and ways to exercise them. This lesson starts by talking about the paradoxes created in the reality beneath copyright mostly due to the zero-cost-for-perfect-copy reality we live in today, and then goes on to introduce the most widely spread licensing schemes for cultural works nowadays: The Creative Commons licensing schemes. What I felt very interesting is that the lesson tackles the most common issues raised by authors choosing CC licensing: The Non-Commercial and Non-Derivatives restrictions. While authors are usually not very hard to convince about the virtues (and their personal benefit) in CC licensing, it's usually hard for authors to accept and relinquish the "locks" on non-derivability and non-commercial-use. Now, as to the homework: This time, I have to admit it was a no-brainer to me. The homework was to take part of a "Licenciat n": Publish some work under a open or free license (a work of any kind; it can be text, photo, video, etc.; it can be a new or a pre-existing work) and argue as to why did you choose said license. Even the act of writing this blog post means I'm releasing some work. The bottom of my blog reads, All the material found at this site is freely available, and you can use it as you wish, except when the document mentions explicitly different conditions. Yes, this is not a widely known, preexisting licensing scheme, and more or less goes along the lines of CC0 There are some good things I have posted in this blog, some things I am proud about. This post goes more towards the average quality: Not adding anything of great value to the world, just reiterating/reposting what is already said. But I know this blog is syndicated over several different places, and its content might be copied over to other places. I don't need to know or be notified about it. I just care for people to be able to reach back if they want to do it. But, of course, if I'm asked to do a homework, I prefer not to have it so self-referential. What else to license? We are over halfway across the Wiki Loves Monuments contest for 2013. This year I am clearly not as active on it as I was last year, but yesterday I uploaded about 60 photos from our recent trips, from historical monuments in some towns/cities within ~150Km of Mexico City: Puebla, Tlaxcala, Cacaxtla, Toluca, Santiago Tianguistenco. I intend to upload more. Those photos are licensed CC-BY-SA. Why? Mainly, because it is the defaul for the contest. One of the lessons learned working on Debian is that the less the license I choose deviates from what the upstream authors of a larger project prefer, the more useful my work will be. And being Wikimedia a host of many related projects, I prefer to completely abide by their recommendations unless there is a real, strong reason not to do so. Also, I know that at least most the photos I am uploading are not of contest quality (although some are quite nice, but we don't take that long to take a shot, and our camera is probably too basic to get a quality shot, if not by mere chance). I am using the WLM contest as an excuse to categorize and upload my pictures Of course, I'm not including personal photos, but... If I found a building, place or landscape interesting enough to take a photo of, why not make said photo available for whoever needs it, in a canonical, easy to reach media repository (as Wikimedia Commons) is)? And finally, I think it's time to start getting more open and forward-going with a project that I have been giving a lot of work during the past year, a project that has been even eating quite a bit of my Debian time: My upcoming book. While this book is not yet freely licensed (it is freely distributed online, but does not yet have an explicit license, and I mention it on my teaching/project page. Why? Because it still has some figures and data which are not mine, and which I have to re-elaborate to be able to legally license it. But anyway I am writing a book on the subject of Operating Systems, and am basing my teaching on it. The first semester was basically hell for me, as I was working almost full time on it, and needed to allocate some time as well to the rest of my formal work. This second semester has been quite smoother. And there is a very important (to me) initiative to which I (or should I now start saying, we) submitted this work: The LATIn Project: An initiative to collaboratively create quality, freely redistributable text books for the university level, natively in Spanish, for their use in Latin American countries. I was not able to propose this book due to some formalities Projects have to be proposed by professors in participant universities, and UNAM is not one of them. However, I offered this work to people in the Universidad Nacional de Rosario, Argentina, and they managed to push the project forward. Together with people from Universidad Aut noma de Aguascalientes (M xico) and Universidad del Cauca (Colombia), the project was approved by LATIn. And this LATIn project can also be seen as a why do you agree to license your work this way? candidate: I have already written most of what I'd be happy to call a full book. Of course, I do expect my colleagues to chime in with more content, and I expect us to produce a true collaborative project. But as I am not part of the member universities, they will get paid for their contribution And not me. What do I have to gain from this? More than anything else, the circulation of my work. I'm sure that, were I to publish from UNAM, I would have a good projection. I was (am!) thinking of looking for opportunities to present the book to teachers on the subject, as well as to computing engineering/science students, but the most attractive part (for me) for LATIn is that part of the prize will be that the book will be used to base work on for teaching at least during 2014 in the three said universities And of course, my/our work will only start then, as we will have to further circulate the work. Anyway... I expected this post to be small. But as you can see, I am happy with important work perspectives :-)

One of the earlier examples of students using MIT computer resources to lay the groundwork for a later commercial endeavour, Zork was originally written in a LISP derivative called MDL. This was later tuned into the Zork Implementation Language, a domain specific language that was compiled to target the Z-machine rather than a specific piece of hardware. Combined with machine-specific Z-machine interpreters, this allowed rapid porting of games to a wide range of platforms - the only thing that needed to be rewritten was the interpreter, and that could be reused for any future games running on the same hardware. Infocom were eventually acquired and killed off, but fan interest in their games continued. New Z-machine interpreters were written in order to allow their games (including Zork) to be run on platforms that Infocom had never targetted. One of the best known is Frotz. This has the advantage of being (a) portable and (b) including a "dumb" UI that makes no assumptions about the availablity of any vaguely useful functionality. Like, say, a Curses library. So, Frotz seemed like the natural choice when this happened. But despite having a set of functionality that makes it look much more like an OS than a boot environment, UEFI doesn't actually expose a standard C library. The EFI Application Development Kit solves this particular design decision. Porting Frotz ended up involving far more fixing up of Frotz bugs that tripped up -Werror than anything else. One note, though - make sure you include DevShell in the list of required packages at build time, otherwise file i/o will mysteriously fail. The tying of file i/o to the shell protocol unfortunately means that Frotz can't be directly launched by the firmware. The Boot to Zork images therefore contain a UEFI shell in the standard boot location (\EFI\BOOT\BOOTX64.EFI) which is executed when the firmware attempts to boot the device. The shell then looks for a file called "startup.nsh" in the root directory of the boot device and executes it. Unfortunately this doesn't actually set the shell equivalent of the current device, and so just launching Frotz from startup.nsh fails when Frotz can't open the Zork data file. The solution for this is simple, if ugly - the script walks through the list of devices, looking for one that contains ZORK1.DAT in the root directory. It then changes to that device and launches Frotz. If Frotz exits, it then resets the system. This could be avoided by doing some more work and turning Frotz into a more UEFI-native application. Teaching Frotz to make native UEFI calls would avoid the requirement for the shell protocols, and the firmware provides a mechanism to obtain the path of the currently running executable which would avoid the need to explicitly locate the device. But I'm lazy and this was a "I'm spending the day on a plane" project initially inspired by a Sazerac-fuelled conversation during the UEFI plugfest, not a demonstration of UEFI best practices. UEFI Boot To Zork and the source code to the modified Frotz can be downloaded here.

comments

Review: Throne of the Crescent Moon, by Saladin Ahmed

Series:	Crescent Moon Kingdoms #1
Publisher:	DAW
Copyright:	January 2013
ISBN:	0-7564-0778-8
Format:	Mass market
Pages:	367

Doctor Adoulla Makhslood is the last ghul hunter in Dhamsawaat. He's an old man, and tired, and increasingly uncertain whether he wants to continue putting his life in danger to fight one more fight. But there are no replacements; his young and idealistic dervish apprentice, despite obvious skill, is too inexperienced and uninterested in the knowledge and investigation that makes Adoulla so good at what he does. So, when a young boy shows up on his doorstep, a survivor of a vicious ghul attack, he goes hunting once again. And both finds something much darker and stronger than he expected and meets an unexpected ally: a shape-shifting girl who is herself a survivor of something far worse than ghuls. Throne of the Crescent Moon is a first-novel fantasy that has made quite a splash, including a Hugo nomination (although I thought a win was unlikely, and I was right). One of the more interesting comments I've read about it was the observation by Jo Walton that if it had come out in 1980, it would have been an amazing, unusually wonderful, non-normative book. That it isn't is evidence of the changes in fantasy since then. The field has grown. As you might guess from the publisher's insistence on comparing it with The Arabian Nights, and from the names mentioned above, this book is set in an unspecified Middle Eastern culture and draws heavily from its mythology. It does not contain any western characters; the entire story is native to its region. The same is true of the mythology, which gave it a refreshing originality for me. And while I'm not the best one to judge, the presentation of that setting seemed free of the tint of the exotic. People live here, there are factions and multiple cultures, and the supernatural is a backdrop to most of the city. This would be very unusual decades ago. It's a bit less unusual now, and the story itself is a familiar ensemble fantasy, if refreshingly free of extended map exploration journeys. Ahmed brings together five characters with very different backgrounds and throws their varying talents at a problem. A few of those elements are a bit too familiar: female sexuality as a threat to male power is a world-building element that I could do without, for example. I also found the villain completely inadequate in its unidimensional evil (reinforced with a few more torture scenes than I needed to read), and even the attempted complication felt too predictable to me. Except for one touch of perspective, I'd call Throne of the Crescent Moon a competent, enjoyable, but somewhat forgettable fantasy novel, albeit one that's refreshingly concerned with a city instead of countrysides and travel. That touch is in the age of several of the protagonists. We get sections following several of the main characters in tight third person, but Adoulla is the first protagonist and the focus of much of the book. And Adoulla is neither young nor inexperienced, and is not going through the typical fantasy bildungsroman. He's old, experienced, a bit set in his ways, cranky, cantankerous, and deeply in love with his city. In tone, this shifts the story towards Star Wars as seen through the eyes of Obi-Wan Kenobi instead of Luke Skywalker. And that change I really liked, and would be happy to see become more common in fantasy. There was a bit of a downside for me in three of the five main characters being older. At times, Throne of the Crescent Moon has an elegiac tone, a feeling of life, power, and time slipping away and cultures and ways of life being lost. That's not my favorite thematic element. I tend to find it more depressing than I really want in my reading. But, here, the mature perspective more than makes up for some occasional negativity. I particularly liked how the older characters treated the likely budding young romance, which would have been the center of many books in this genre, with bemused detachment and playful encouragement without giving it more significance than it deserves. There's another romance that ends up being more significant, and which needs the weight of time and well-lived lives to become so. I don't think this book deserves a Hugo, but it is an enjoyable fantasy with some interesting characters and a refreshing perspective and setting. The villain really isn't adequate to the story, there's a bit too much torture for my liking, and I could have done without a few of the tropes, but it's a solid entry in its genre and a pleasant few hours of reading. And despite being advertised as the first of a series, it comes to a satisfying conclusion. Rating: 7 out of 10

On a regular basis, I find myself the odd-man-out when it comes to talking about how to work with Python on Debian systems. I'm going to write this and post it so that I might be able to point people at my thoughts without having to write the same email in response to each thread that pops up. Turns out I don't fit in with the Debian hardliners (which is to say, the mindset that pip sucks and shouldn't exist), nor do I fit in with the Python hardliners (which is to say apt and dpkg are out of date, and neither have a place on a Development machine). I think our discourse on this topic has become petty and stupid in general. Let's all try to step back and drop a bit of the attitude. pip doesn't suck, and neither does apt. The truth is, both sides are wrong. As with any subject, the real answer here is much more nuanced than either side presents it. I'm going to try and present my opinion on this, in the way that both my Pythonista self and my Debianite self see the issue. Hopefully I can keep this short, to the point, and caked with logic. The case for dpkg (the Debianite in me) In defense of dpkg and apt, imagine having to install python-gnome2 on all your systems when you install. It'd be hell on earth. Imagine having a user try to do this. It's insane to assume that end-users will be using pip for this purpose. pip is fun and all, but it's also installing 100% untrusted code to your system (perhaps as root, if you're using pip with sudo for some reason), and hasn't been reviewed for software freeness, which is something Debian (and Debian users) take seriously. This isn't even to mention the hell that pip wreaks on dpkg controlled files / packages. Try to remember how much of your system running (yes, right now) is running because of Python or Python modules. Try to imagine how much of a pain in the ass it'd be if you couldn't boot into GNOME to use nm-applet to connect to wifi to pip install something. I'm sure even the most extreme pip'er understands the need for Operating System level package management. Debian also has a bigger problem scope - we're not maintaining a library in Debian for kicks, we're maintaining it so that end user applications may use the library. When we update something like Django, we have to make sure that we don't break anything using it (although, to be honest, the fact that we package webapps is an entire rant for later) before we get to update it to the newest release. Hell, with a few coffees, I could automate the process of releasing a .deb with a new upstream release, 100% unattended. I won't, however, since this is an insane idea. Let's go over a brief list of things I do before uploading a new package:

1. Review the entire codebase for simple mistakes.
2. Review the entire codebase for license issues.
3. Review the entire codebase for files without source, and track down (and include source for) any sourceless files (such as pickle files, etc).
4. Get to know the upstream, get to know open bugs, write something using the lib, in case I need to debug later.
5. Install the package.
6. Test the package.
7. Work out any Debian package issues (this is easy).

Now, a brief list of things I do before I update a package:

1. Review the changes between the last uploaded version (in diff format, if it's sane, otherwise get the VCS and review), ensure all the above are still OK.
2. Review for Debian-local issues (such as how it will upgrade, using piuparts, and adequate, etc).
3. Check to make sure it won't break any reverse dependencies.
4. Review for bugfixes that I might need to bring back to the stable release.
5. Figure out if I should (or even can) backport the package, if API is stable.
6. Review for bugs (upstream or in Debian) that I need to mention in the debian/changelog.

Clearly, this isn't a quick-and-dirty task. It's not a matter of getting a package updated (technically), it's a much more detailed process than that. This is also why Debian is so highly regarded for its technical virtuosity, and why the ISS decided to deploy Debian in space, despite other commercial distros such as Red Hat, or Ubuntu, and community distros, such as Fedora or Arch. It's also not Debian's job to package the world in the archive. This is an insane task, and it's not Debian's place to do it. We introduce libraries as things need them, not because we wrote some new library that someone may find slightly useful at some point in the future. maybe. Upstream developers and language communities (not only Python here) tend to lose sight of why we're doing this in the first place, which is our users. This isn't some sort of technical pissing contest to see who can distribute the software in the best way. Debian-folk always keep end users as our highest priority. I quote the Debian Social Contract, when I say that Our priorities are our users and free software. No one's trying to get developers to use dpkg to create software. In fact, as you'll see below, I actively discourage using system modules for development. The case for pip (the Pythonista in me) In defense of pip, the idea that Debian will keep the latest versions of packages is insane. The idea that we can keep pace with upstream releases is nuts, and the idea that every upstream release on pypi is ready to ship is bananas. b-a-n-a-n-a-annas. As a developer, I don't want to support every release, and I surely don't want other people depending on some random snapshot. Often times, I'll put stuff up on pypi as a preview, or to release often, and solicit feedback without having to give out instructions on using a git checkout (it's also easier to have them try a version from pypi so I can cross-ref the git tag to reproduce issues when they file them) pypi is easy, ubiquitous and works regardless of the platform, which means less of my development time is spent packaging stuff up for platforms I don't really care about (Arch, Fedora, OSX, Windows), even though I value feedback from users on those systems. The effort it takes to release something is limited to python setup.py sdist upload, and it's in a place (and in a shape) that anyone can use it without having 10 sets of platform-local instructions. Even ignoring all the above, when I'm writing a new app or bit of code, I want to be sure I'm targeting the latest version of the code I depend on, so that future changes to API won't hit me as hard. By following along with my dependencies' development, I can be sure that my code breaks early, and breaks in development, not production. Upstreams also tend to not like bug reports against old branches, so ensuring I have the latest code from pypi means I can properly file bugs. Lastly, I prefer virtualenv based setups for development, since I'm usually working on many things at once. This often means version mismatches in libraries, which brings in API changes (another whole rant here as well). I don't want to keep installing and uninstalling packages to switch between the two projects, and using a chroot(8) means a lot of overhead and that it's disconnected from my development environment / filesystem, so I resort to virtualenv to isolate my Development environment. Final notes I don't want to keep arguing about this. Just accept that the world's a big place and that there exist use-cases that both apt and pip need to exist and work in the way they're working now. At the very least, try and understand there exist smart people on both sides, and no one is trying to screw anyone over or keep their own little private club to themselves. Hopefully, going forward, we can make sure that the integration between these two tools gets better, not worse. Help make this dream a reality. Contribute to a productive tone, not a destructive one. In short:

• Use pip without sudo always. Don't tell people to use sudo.
• Use apt or dpkg when deploying system-wide.
• Understand people are going to package, and they will be more concerned about software using your library then keeping your library up to date.
• Understand Debian Developers and package maintainers have to do a lot of work when updating or sponsoring an upload.
• Understand upstream developers can't be bothered to fix every issue with every release (release early, release often) with some snapshot you introduced into unstable.
• Use pip and virtualenv in development setups, so we can upgrade your app when we upgrade the lib.

I have just uploaded Lintian 2.5.18 to unstable. While fixing 22 bugs, it only features 5 new tags.

• debian-changelog-has-wrong-weekday
• debian-rules-missing-good-practice-target-dfsg
• empty-udeb-package
• file-name-in-PATH-is-not-ASCII
• misplaced-extra-member-in-deb

The release also include fixes to some false-positives, such as python:any dependencies triggering python-script-but-no-python-dep, a rewritten README file. We also included a patch to make Lintian accept the Original-Maintainer field by default for non-Debian vendors (even if they do not have a profile and Lintian ends up loading the debian/main profile). We also added support for running Lintian directly from a git checkout or source tree without setting LINTIAN_ROOT (or passing root). Since that was the primary use-case for root that option has now been deprecated. I also had lintian and lintian-info require the include-dir and [no-]user-dir options as the first if given at all. I would like to thank Bastien Roucari s, Gaudenz Steinlin, Gunnar Wolf, J r my Bobbio and Lucas Nussbaum for contributing to Lintian and the many who submitted reports or suggestions for improvements. I would also like to thank Brian hugmeir Fraser, who assisted me in identifying and working around a bug in Perl s glob function when run under threads (filed upstream as RT#119897).