Search Results: "nate"

5 May 2014

Thomas Goirand: OpenStack Icehouse bugs all cleaned-up

I ve done some clean-up in the Debian BTS. The result can be seen in the QA graphs: The last remaining 6 bugs are only affecting OpenStack Essex (which is what Wheezy ships, and which unfortunately I have not enough time to support properly), and the last one is waiting for FTP masters approval after I added Python 3 support to oslo-config (OpenStack is slowly moving to Python3, and I ve tried to add support to Python3 in the packages as well each time it was possible). Also, the Icehouse packages have passed the tempest tests (a set of functional tests) which we run in our continuous integration system. As I was doing some triaging, it is possible that some bugs have been closed a bit quickly (like for example the Ceilometer FTBFS which I couldn t reproduce, even with sbuild and cowbuilder), so it is well possible that some more QA work will be needed later on. I m also expecting a new set of patches for supporting Ceph in Nova. I m sure there s issues which we will discover later on, however, it is nice to have this result right after the first Icehouse release of OpenStack. Next up: testing new components that I uploaded for this release: Trove (DB as a Service), Designate (DNS as a Service), Ironic (Metal as a Service, or cloud computing on bare-metal), and TripleO (OpenStack on OpenStack). I unfortunately know already that TripleO and Tuskar wont really work yet, and that it needs some patches to be sent upstream for it to support Debian correctly (let s work it out for Juno!). So please consider it as a technology preview only. Though Trove, Designate and Ironic are supposed to be in good enough shape, I didn t have the chance to test them more than just installing the packages and checking daemons are running and connected to the various components of OpenStack (eg: database, keystone and RabbitMQ). Please do test them and report bugs in the BTS. I d like to thanks Gustavo Panizzo & Thomas Bechtold who contributed to this release, and also the folks at eNovance (Emilien, Cyril, Seb ) who provided precious help with the CI and package quality. See you in the Juno Atlanta design summit next week (as hopefully, my plane ticket issue will be soon solved).

3 May 2014

Dirk Eddelbuettel: RcppSMC 0.1.3 and 0.1.4

The very useful Valgrind tool had flagged an actual error in the package which the CRAN maintainers asked us to address. This was followed by a minor brown-bad oversight of a missing delete, also tagged by Valgrind. Both are pretty ancient bugs which we probably should have found aeons ago. Releases 0.1.3 and 0.1.4 made it to CRAN yesterday in short succession. To recap, RcppSMC combines the SMCTC template classes for Sequential Monte Carlo and Particle Filters (Johansen, 2009, JSS) with the Rcpp package for R/C++ Integration (Eddelbuettel and Francois, 2011, JSS) and thereby allows for easier and more direct access from R to the computational core of the Sequential Monte Carlo algorithm. The two NEWS entries are below:
Changes in RcppSMC version 0.1.4 (2014-05-02)
  • Added missing delete operator to destructor in sampler
Changes in RcppSMC version 0.1.3 (2014-05-01)
  • Bugfix in Sampler.iterate() for memory overrun detected by valgrind
Courtesy of CRANberries, there are also diffstat reports for 0.1.3. and 0.1.4. As always, more detailed information is on the RcppSMC page,

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

1 May 2014

C.J. Adams-Collier: I think perhaps they mean could join x86 in the server room

As much as I love PowerPC, and as great as it is that they re doing this OpenPower thing, I don t think that the first step should be to eliminate the competition. Perhaps they should put some effort in to taking back a large part of the market. Didn t these guys read The Art of War?
The OpenPower Foundation, and how IBM and friends could oust x86 from the server room
I d certainly take a look at a server if they sent me one; it looks like Debian still supports powerpc arch on wheezy. ;-)

30 April 2014

Mirco Bauer: Smuxi 0.11 "Distractions" Release

And here we go again! We're proud to announce the new version of Smuxi, release 0.11 "Distractions". During the development, 11 bug reports and 2 feature requests in 112 commits were worked on. Notable highlights in this release are:

User Interface Enhancements
  • The chat list can be shrunken. This is especially handy with XMPP/Jabber and long group chat identifiers.
  • The highlight counter is now a separate column. This enhances the vertical alignment with other highlights and guarantees to be visible even if the chat name was truncated.

Multi Identity Support Smuxi cares for user feedback. Multi identity support was the most voted feature and thus it has been implemented! Now you can please your schizo^Wdesire to use different nicks, users and real names depending on the server. Simply edit the server in preferences and change the details.

Message Patterns Everybody knows text can be boring because it is all just text. Nothing can sidetrack you except reading that bare text. Text often has recurring patterns from which something useful and interactive can be created. For example, someone writes:
Hey meebey, do you know RFC2812?
RFCs are a recurring pattern with a distinct number behind it and are real references to something in the internet (collection of protocol specifications). So I would usually fire up a browser tab, copy/paste or type RFC2812 into my favorite search engine and click the first hit. Then I'd reply to the question afterwards. But with Smuxi's message patters, it turns RFC2812 into a link on which you can simply click to launch the relevant document. Wow this is very cool, but isn't this already happening with http URLs and email addresses? Exactly! Why shouldn't more information be used to create useful things from it? Smuxi message patterns allow you to define text patterns that are transformed into clickable links. This can be used for RFCs, CVEs, bug report numbers (#XXX), git commit hashes and much more. Make good use of your creativity! By default Smuxi comes with built-in message patterns for:
  • URLs
  • heuristic URLs (not starting with http:// etc)
  • email addresses
  • RFCs
  • CVEs
  • Debian Security Advisories (DSA)
  • Many popular bug trackers (GNU, GCC, kernel, Launchpad, freedesktop, GNOME, KDE, Xfce, Debian, Redhat, Novell, Xamarin, openSUSE, Mozilla, Samba, SourceForge, CPAN, boost, Claws and Smuxi)
If you know more general patterns useful for others, please submit them. For a full list of built-in message patterns or how to add your own patterns, head over to the message pattern documentation.

Hooks Enhancements
  • A bug was fixed that prevented hooks from issuing more than one command
  • New hook points:
    • engine/session/on-group-chat-person-added
    • engine/session/on-group-chat-person-removed
    • engine/session/on-group-chat-person-updated
  • New hook variables:
    • CMD
    • CMD_PARAMETER
    • CMD_CHARACTER
    • PROTOCOL_MANAGER_PRESENCE_STATUS: Unknown, Offline, Online, Away

Twitter Enhancements As of 14 Jan 2014, Twitter disallows unencrypted HTTP requests which broke Smuxi's Twitter support. Smuxi is now making exclusively encrypted requests (HTTPS) and thus works with Twitter again.

JabbR (Beta) Enhancements
  • Messages now raise Smuxi hooks
  • The Validate certificate setting is now correctly honored.

Updated Translations Smuxi should now be in your language, including:
  • Initial complete Dutch (Jeroen Baten)

Behind the Scenes
  • New Smuxi git repository @ GNOME
  • Cleaner XMPP code (Oliver Schneider)
  • Smuxi's STFL text frontend is doing a graceful shutdown on quit (Calvin B))
  • New sexy website! We hope you like it :)

Contributors Contributors to this release are the following people:
  • Mirco Bauer (98 commits)
  • Oliver Schneider (6 commits)
  • Calvin B (6 commits)
  • Andr s G. Aragoneses (1 commit)
  • Jeroen Baten (translations)
Thank you very much for your contributions to Smuxi! Want it? Go here and grab it right now!

Posted Mon Apr 14 13:23:29 2014

29 April 2014

Guido G nther: Bits from the 7th Debian groupware meeting

The seventh Debian Groupware Meeting was held in the LinuxHotel, Essen, Germany. We had one remote hacker from NYC which brings the number of attendants up to 9. This is a short summary of what happened during the weekend: Since we had a nice mix of first time Debian contributors, Debian Maintainers and Debian Developers we had lots of room for discussion and co-working which made this an exciting weekend. Groupphoto by Carsten Sch nert

Petter Reinholdtsen: Half the Coverity issues in Gnash fixed in the next release

I've been following the Gnash project for quite a while now. It is a free software implementation of Adobe Flash, both a standalone player and a browser plugin. Gnash implement support for the AVM1 format (and not the newer AVM2 format - see Lightspark for that one), allowing several flash based sites to work. Thanks to the friendly developers at Youtube, it also work with Youtube videos, because the Javascript code at Youtube detect Gnash and serve a AVM1 player to those users. :) Would be great if someone found time to implement AVM2 support, but it has not happened yet. If you install both Lightspark and Gnash, Lightspark will invoke Gnash if it find a AVM1 flash file, so you can get both handled as free software. Unfortunately, Lightspark so far only implement a small subset of AVM2, and many sites do not work yet. A few months ago, I started looking at Coverity, the static source checker used to find heaps and heaps of bugs in free software (thanks to the donation of a scanning service to free software projects by the company developing this non-free code checker), and Gnash was one of the projects I decided to check out. Coverity is able to find lock errors, memory errors, dead code and more. A few days ago they even extended it to also be able to find the heartbleed bug in OpenSSL. There are heaps of checks being done on the instrumented code, and the amount of bogus warnings is quite low compared to the other static code checkers I have tested over the years. Since a few weeks ago, I've been working with the other Gnash developers squashing bugs discovered by Coverity. I was quite happy today when I checked the current status and saw that of the 777 issues detected so far, 374 are marked as fixed. This make me confident that the next Gnash release will be more stable and more dependable than the previous one. Most of the reported issues were and are in the test suite, but it also found a few in the rest of the code. If you want to help out, you find us on the gnash-dev mailing list and on the #gnash channel on irc.freenode.net IRC server.

Russell Coker: Autism and the Treatment of Women Again

Background I ve previously written about the claim that people use Autism as an excuse for bad behavior [1]. In summary it doesn t and such claims instead lead to people not being assessed for Autism. I ve also previously written about empathy and Autism in the context of discussions about conference sexual harassment [2]. The main point is that anyone who s going to blame empathy disorders for the widespread mistreatment of women in society and divert the subject from the actions of average men to men in minority groups isn t demonstrating empathy. Discussions of the actions of average men are so often derailed to cover Autism that the Geek Feminism Wiki has a page about the issue of blaming Autism [3]. The Latest Issue Last year Shanley Kane wrote an informative article for Medium titled What Can Men Do about the treatment of women in the IT industry [4]. It s a good article, I recommend reading it. As an aside @shanley s twitter feed is worth reading [5]. In response to Shanley s article Jeff Atwood wrote an article of the same title this year which covered lots of other things [6]. He writes about Autism but doesn t seem to realise that officially Asperger Syndrome is now Autism according to DSM-V (they decided that separate diagnosis of Autism, Asperger Syndrome, and PDD-NOS were too difficult and merged them). Asperger Syndrome is now a term that refers to historic issues (IE research that was published before DSM-V) and slang use. Gender and the Autism Spectrum Jeff claims that autism skews heavily towards males at a 4:1 ratio and cites the Epidemiology of Autism Wikipedia page as a reference. Firstly that page isn t a great reference, I fixed one major error (which was obviously wrong to anyone who knows anything about Autism and also contradicted the cited reference) in the first section while writing this post. The Wikipedia page cites a PDF about the Epidemiology of Autism that claims the 4.3:1 ratio of boys to girls [7]. However that PDF is a summary of other articles and the one which originated the 4.3:1 claim is behind a paywall. One thing that is worth noting in the PDF is that the section containing the 4.3:1 claim also references claims about correlations between race and Autism and studies contradicting such claims it notes the possibility of ascertainment bias . I think that anyone who reads that section should immediately consider the possibility of ascertainment bias in regard to the gender ratio. Most people who are diagnosed with Autism are diagnosed as children. An Autism diagnosis of a child is quite subjective, an important part is an IQ test (where the psychologist interprets the intent of the child in the many cases where answers aren t clear) to compare social skills with IQ. So whether a child is diagnosed is determined by the psychologist s impression of the child s IQ vs the impression of their social skills. Whether a child is even taken for assessment depends on whether they act in a way that s considered to be obviously different. Any child who is suspected of being on the Autism Spectrum will be compared other children who have been diagnosed (IE mostly boys) and this will probably increase the probability that a boy will be assessed. So an Aspie girl might not be assessed because she acts like other Aspie girls not like the Aspie boys her parents and teachers have seen. The way kids act is not solely determined by neuro-type. Our society expects and encourages boys to be louder than girls and take longer and more frequent turns to speak, this is so widespread that I don t think it s possible for parents to avoid it if their kids are exposed to the outside world. Because of this boys who would be diagnosed with Asperger Syndrome by DSM-IV tend to act in ways that are obviously different from other kids. While the combination of Autism and the the social expectations on girls tends to result in girls who are quiet, shy, and apologetic. The fact that girls are less obviously different and that their differences cause fewer difficulties for parents and teachers makes them less likely to be assessed. Note that the differences in behavior of boys and girls who have been diagnosed is noted by the professionals (and was discussed at a conference on AsperGirls that my wife attended) while the idea that this affects assessment rates is my theory. Jeff also cites the book The Essential Difference: Male And Female Brains And The Truth About Autism by Professor Simon Baron-Cohen (who s (in)famous for his Extreme Male Brain theory). The first thing to note about the Extreme Male Brain theory are that it depends almost entirely on the 4.3:1 ratio of males to females on the Autism Spectrum (which is dubious as I noted above). The only other evidence in support of it is subjective studies of children which suffer from the same cultural issues this is why double blind tests should be used whenever possible. The book Delusions of Gender by Cordelia Fine [8] debunks Simon Baron-Cohen s work among other things. The look inside feature of the Amazon page for Delusions of Gender allows you to read about Simon Baron-Cohen s work [9]. Now even if the Extreme Male Brain theory had any merit it would be a really bad idea to cite it (or a book based on it) if you want to make things better for women in the IT industry. Cordelia s book debunks the science and also shows how such claims about supposed essential difference are taken as exclusionary. The Problem with Jeff Atwood Jeff suggests in his post that men should listen to women. Then he and his followers have a huge flame-war with many women over twitter during which which he tweeted Trying to diversify my follows by following any female voices that engaged me in a civil, constructive way recently . If you only listen to women who agree with you then that doesn t really count as listening to women. When you have a stated policy of only listening to women who agree then it seems to be more about limiting what women may feel free to say around you. The Geek Feminism wiki page about the Tone Argument [10] says the following: One way in which the tone argument frequently manifests itself is as a call for civility. A way to gauge whether a request for civility is sincere or not is to ask whether the person asking for civility has more power along whatever axes are contextually relevant (see Intersectionality) than the person being called incivil , less power, or equal power. Often, people who have the privilege of being listened to and taken seriously level accusations of incivility as a silencing tactic, and label as incivil any speech or behavior that questions their privilege. For example, some men label any feminist thought or speech as hostile or impolite; there is no way for anybody to question male power or privilege without being called rude or aggressive. Likewise, some white people label any critical discussion of race, particularly when initiated by people of color, as incivil. Writing about one topic is also a really good idea. A blog post titled What Can Men Do should be about things that men can do. Not about Autism, speculation about supposed inherent differences between men and women which are based on bad research, gender diversity in various occupations, etc. Following up a post on What Can Men Do with discussion (in blog comments and twitter) about what women should do before they are allowed to join the conversation is ridiculous. Jeff s blog post says that men should listen to women, excluding women based on the tone argument is gross hypocrisy. Swearing Jeff makes a big deal of the fact that Shanley uses some profane language in her tweets. This combines a couple of different ways of silencing women. It s quite common for women to be held to a high standard of ladylike behavior, while men get a free pass on doing the same thing. One example of this is the Geek Feminism article about the results of Sarah Sharp s request for civility in the Linux kernel community [11]. That s not an isolated incident, to the best of my recollection in 20+ years my local Linux Users Group has had only one debate about profanity on mailing lists in that case a woman (who is no longer active in the group) was criticised for using lesser profanity than men used both before and after with no comment (as an experiment I used some gratuitous profanity a couple of weeks later and no-one commented). There is also a common difference in interpretation of expressions of emotion, when a woman seems angry then she invariably has men tell her to change her approach (even when there are obvious reasons for her anger) while when a man is angry the possibility that other people shouldn t make him angry will usually be considered. The issues related to the treatment of women have had a large affect on Shanley s life and her friend s lives. It s quite understandable that she is angry about this. Her use of profanity in tweets seems appropriate to the situation. Other Links Newsweek s Gentlemen in Technology article has a section about Jeff [12], it s interesting to note his history of deleting tweets and editing his post. I presume he will change his post in response to mine and not make any note of the differences. Jacob Kaplan-Moss wrote a good rebuttal to Jeff s post [13]. It s a good article and has some other relevant links that are worth reading.

28 April 2014

Daniel Pocock: SMS logins: an illusion of security

The IT security world is still reeling from the impact of the OpenSSL Heartbleed bug. Thanks to the bug, many experts have been reviewing other technologies to try and find similar risks. While Heartbleed was hidden away in the depths of the OpenSSL code base, another major security risk has been hiding in plain sight: SMS authentication for web site logins. Remarkably, a number of firms have started giving customers the ability to receive single-use passwords over SMS for logging into their secure web sites. Some have even insisted that customers can no longer log in without it, denying customers the right to make an important choice about their own security preferences. Unfortunately, SMS is no substitute to the one-time-passwords generated using proper authentication tokens or the use of other strong authentication schemes such as cryptographic smart cards. Even telephone companies themselves advise that SMS should not be used to secure financial transactions. Ocean's 11 in real life: exploiting the weakest link in the chain To deliver single-use SMS passwords, the SMS must travel through various networks from the firm's headquarters, to a wholesale SMS gateway, international SMS network and finally down the line of the local phone company. In comparison, properly certified token devices generate a code inside the device in the palm of your hand. The code only travels from the screen to your eyes. In a litany of frauds coming in all shapes and sizes, telephone networks have been exploited over and over again because they are almost always the weakest link in the chain. Using the mobile SMS network for authentication is not building on solid ground - some experts even feel it is downright stupidity. One of the most serious examples was the theft of $150,000,000 from a pension fund deposited with JP Morgan: it was described as a real-life case of Ocean's 11. The authentication was meant to be a phone call rather than an SMS: a phone company employee who was in on the scam duly ensured the call never reached the correct place. The insecurity of traditional telephone networks has been on display for all the world to see in the ongoing trial of News Corporation executives for phone hacking. If journalists from a tabloid newspaper can allegedly hack a dozen phones before their first cigarette of the day, is it really wise to use an insecure technology like SMS as the cornerstone of a security system for authorizing transactions? A fraud recently played out on many credit card holders in the UK exploited a low-tech feature of the phone system to trick people to believe they were safe by "calling back" to their bank. A plethora of new attack vectors The staggering reality of the situation is that attackers don't even have to directly hack their victim's phones to access SMS messages. As the Android API documentation demonstrates, SMS reception is notified to all apps in real-time. Apps can process the messages even when the phone is sleeping and the message is not read by the user. Just consider all the apps on a phone that have requested permission to read incoming messages. There was an uproar recently when a new version of the Facebook app started demanding permissions to read incoming SMS. The app can't be installed if the user doesn't agree to these new permissions. WhatsApp, another popular app that has SMS access rights, was recently exposed in a major security scandal which revealed they use a phone's IMEI number as the password. When people install an app like Tinder (which does not yet request SMS access) is the security of their bank account likely to be at the front of their mind? Even if Facebook intends no harm, they have opened the floodgates by further de-sensitizing users to the risks of giving apps un-necessary access to their data. These companies are looking for every piece of data that could give them an edge in their customer profiling and marketing programs. Having real-time access to your SMS is a powerful way for them to understand your activities and feelings at every moment in the day. To facilitate these data analysis techniques, replicating and archiving your messages into their cloud databases (whether you can see them there or not) is par for the course. The cloud, of course, has become a virtual smorgasboard for cyber-criminals, including both hackers and occasionally insiders wanting to peek at private data or harvest it en-masse. Social networking and communication sites are built on a philosophy of sharing data to create interaction and excitement. Unfortunately, this is orthogonal to the needs of security. In this context, the telephone network itself may no longer be the weakest link in the chain. The diligent attacker only needs to look for the cloud operator with an unplugged security hole and use their system as a stepping stone to read any SMS they want, when they want. Would you notice a stray SMS? Maybe you feel that you would notice a stray SMS carrying a login code for your bank account. Would you always be able to react faster than the criminal however? Thanks to social networks, or location data inadvertently leaked by other apps the attacker can easily work out whether you are on holiday, at the gym, at a party or sleeping or in some other situation where you are not likely to check messages immediately. If you receive a flood of SMS spam messages (deliberately sent by an attacker) in the middle of the night and you put your phone into silent mode and ignore it, you may well miss one message that was a login to your bank account. SMS technology was never designed for secure activities. The inconvenience of SMS While security is a headline issue these days, it is also worth reflecting on the inconvenience of SMS in some situations. Travel is at the top of the list: SMS doesn't work universally when abroad. These are usually the times when the only way to access the bank is through the web site. After dealing with the irritations of the hotel or airport wifi registration, do you really need more stress from your bank's systems too? For some networks, SMS can be delayed by hours or days, sometimes never arriving at all. Many people swap their SIM cards when travelling to avoid the excessive roaming charges and there is extra inconvenience in swapping SIM cards back again just to log in to a bank account. Worst of all, if you are tethering with a SIM card from the country you are visiting, then it is impossible for you to receive the SMS message from the bank on your regular SIM card while simultaneously maintaining the SSL connection to their web site over your new SIM card. Other problems like a flat battery, water damage or PIN permanently blocked by children playing with the phone can also leave you without access to your bank account for varying lengths of time. Is there any up-side to SMS authentication? The only potential benefit to SMS authentication is that it weeds out some of the most amateur attempts to compromise your bank account, but this is a false sense of security and it opens up new attack vectors through the cloud as we have just demonstrated. For all other purposes, it smells like a new form of security theater. A more likely reason why it has become popular amongst some firms is that many lenders want to ensure they have mobile phone numbers to contact customers when loan or credit card payments are missed. Making the mobile phone number mandatory for login ensures they almost always have the correct phone number for almost 100% of customers. It is not clear that this benefit justifies the failure to provide proper security and the inconvenience when travelling though. Opting out Next time you log in to a web site, if the firm does try to enrol you in an SMS authentication scheme, it may be a good idea to click the "No thanks" option. If you have already been registered into an SMS authentication scheme, fill out the online complaint form and inform the firm that you will only accept a proper authentication token or cryptographic smart card. These solutions are tried and tested and they are the correct tool for the job.

27 April 2014

Vincent Bernat: Local corporate APT repositories

Distributing software efficiently accross your platform can be difficult. Every distribution comes with a package manager which is usually suited for this task. APT can be relied upon on when using Debian or a derivative. Unfortunately, the official repositories may not contain everything you need. When you require unpackaged software or more recent versions, it is possible to setup your own local repository. Most of what is presented here was setup for Dailymotion and was greatly inspired by the work done by Rapha l Pinson at Orange.

Setting up your repositories There are three kinds of repositories you may want to setup:
  1. A distribution mirror. Such a mirror will save bandwidth, provide faster downloads and permanent access, even when someone searches Google on Google.
  2. A local repository for your own packages with the ability to have a staging zone to test packages on some servers before putting them in production.
  3. Mirrors for unofficial repositories, like Ubuntu PPA. To avoid unexpected changes, such a repository will also get a staging and a production zone.
Before going further, it is quite important to understand what a repository is. Let s illustrate with the following line from my /etc/apt/sources.list:
deb http://ftp.debian.org/debian/ unstable main contrib non-free
In this example, http://ftp.debian.org/debian/ is the repository and unstable is the distribution. A distribution is subdivided into components. We have three components: main, contrib and non-free. To setup repositories, we will use reprepro. This is not the only solution but it has a good balance between versatility and simplicity. reprepro can only handle one repository. So, the first choice is about how you will split your packages in repositories, distributions and components. Here is what matters:
  • A repository cannot contain two identical packages (same name, same version, same architecture).
  • Inside a component, you can only have one version of a package.
  • Usually, a distribution is a subset of the versions while a component is a subset of the packages. For example, in Debian, with the distribution unstable, you choose to get the most recent versions while with the component main, you choose to get DFSG-free software only.
If you go for several repositories, you will have to handle several reprepro instances and won t be able to easily copy packages from one place to another. At Dailymotion, we put everything in the same repository but it would also be perfectly valid to have three repositories:
  • one to mirror the distribution,
  • one for your local packages, and
  • one to mirror unofficial repositories.
Here is our target setup: Local APT repository

Initial setup First, create a system user to work with the repositories:
$ adduser --system --disabled-password --disabled-login \
>         --home /srv/packages \
>         --group reprepro
All operations should be done with this user only. If you want to setup several repositories, create a directory for each of them. Each repository has those subdirectories:
  • conf/ contains the configuration files,
  • gpg/ contains the GPG stuff to sign the repository1,
  • logs/ contains the logs,
  • www/ contains the repository that should be exported by the web server.
Here is the content of conf/options:
outdir +b/www
logdir +b/logs
gnupghome +b/gpg
Then, you need to create the GPG key to sign the repository:
$ GNUPGHOME=gpg gpg --gen-key
Please select what kind of key you want:
   (1) RSA and RSA (default)
   (2) DSA and Elgamal
   (3) DSA (sign only)
   (4) RSA (sign only)
Your selection? 1
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 4096
Requested keysize is 4096 bits
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0) 10y
Key expires at mer. 08 nov. 2023 22:30:58 CET
Is this correct? (y/N) y
Real name: Dailymotion Archive Automatic Signing Key
Email address: the-it-operations@dailymotion.com
Comment: 
[...]
By setting an empty password, you allow reprepro to run unattended. You will have to distribute the public key of your new repository to let APT check the archive signature. An easy way is to ship it in some package.

Local mirror of an official distribution Let s start by mirroring a distribution. We want a local mirror of Ubuntu Precise. For this, we need to do two things:
  1. Setup a new distribution in conf/distributions.
  2. Configure the update sources in conf/updates.
Let s add this block to conf/distributions:
# Ubuntu Precise
Origin: Ubuntu
Label: Ubuntu
Suite: precise
Version: 12.04
Codename: precise
Architectures: i386 amd64
Components: main restricted universe multiverse
UDebComponents: main restricted universe multiverse
Description: Ubuntu Precise 12.04 (with updates and security)
Contents: .gz .bz2
UDebIndices: Packages Release . .gz
Tracking: minimal
Update: - ubuntu-precise ubuntu-precise-updates ubuntu-precise-security
SignWith: yes
This defines the precise distribution in our repository. It contains four components: main, restricted, universe and multiverse (like the regular distribution in official repositories). The Update line starts with a dash. This means reprepro will mark everything as deleted before updating with the provided sources. Old packages will not be kept when they are removed from Ubuntu. In conf/updates, we define the sources:
# Ubuntu Precise
Name: ubuntu-precise
Method: http://fr.archive.ubuntu.com/ubuntu
Fallback: http://de.archive.ubuntu.com/ubuntu
Suite: precise
Components: main main multiverse restricted universe
UDebComponents: main restricted universe multiverse
Architectures: amd64 i386
VerifyRelease: 437D05B5
GetInRelease: no
# Ubuntu Precise Updates
Name: ubuntu-precise-updates
Method: http://fr.archive.ubuntu.com/ubuntu
Fallback: http://de.archive.ubuntu.com/ubuntu
Suite: precise-updates
Components: main restricted universe multiverse
UDebComponents: main restricted universe multiverse
Architectures: amd64 i386
VerifyRelease: 437D05B5
GetInRelease: no
# Ubuntu Precise Security
Name: ubuntu-precise-security
Method: http://fr.archive.ubuntu.com/ubuntu
Fallback: http://de.archive.ubuntu.com/ubuntu
Suite: precise-security
Components: main restricted universe multiverse
UDebComponents: main restricted universe multiverse
Architectures: amd64 i386
VerifyRelease: 437D05B5
GetInRelease: no
The VerifyRelease lines are GPG key fingerprint to use to check the remote repository. The key needs to be imported in the local keyring:
$ gpg --keyring /usr/share/keyrings/ubuntu-archive-keyring.gpg \
>     --export 437D05B5   GNUPGHOME=gpg gpg --import
Another important point is that we merge three distributions (precise, precise-updates and precise-security) into a single distribution (precise) in our local repository. This may cause some difficulties with tools expecting the three distributions to be available (like the Debian Installer2). Next, you can run reprepro and ask it to update your local mirror:
$ reprepro update
This will take some time on the first run. You can execute this command every night. reprepro is not the fastest mirror solution but it is easy to setup, flexible and reliable.

Repository for local packages Let s configure the repository to accept local packages. For each official distribution (like precise), we will configure two distributions:
  • precise-staging contains packages that have not been fully tested and not ready to go to production.
  • precise-prod contains production packages copied from precise-staging.
In our workflow, packages are introduced in precise-staging where they can be tested and will be copied to precise-prod when we want them to be available for production. You can adopt a more complex workflow if you need. The reprepro part is quite easy. We add the following blocks into conf/distributions:
# Dailymotion Precise packages (staging)
Origin: Dailymotion #  
Label: dm-staging   #  
Suite: precise-staging
Codename: precise-staging
Architectures: i386 amd64 source
Components: main role/dns role/database role/web #  
Description: Dailymotion Precise staging repository
Contents: .gz .bz2
Tracking: keep
SignWith: yes
NotAutomatic: yes #  
Log: packages.dm-precise-staging.log
 --type=dsc email-changes
# Dailymotion Precise packages (prod)
Origin: Dailymotion #  
Label: dm-prod      #  
Suite: precise-prod
Codename: precise-prod
Architectures: i386 amd64 source
Components: main role/dns role/database role/web #  
Description: Dailymotion Precise prod repository
Contents: .gz .bz2
Tracking: keep
SignWith: yes
Log: packages.dm-precise-prod.log
First notice we use several components (in ):
  • main will contain packages that are not specific to a subset of the platform. If you put a package in main, it should work correctly on any host.
  • role/* are components dedicated to a subset of the platform. For example, in role/dns, we ship a custom version of BIND.
The staging distribution has the NotAutomatic flag (in ) which disallows the package manager to install those packages except if the user explicitely requests it. Just below, when a new dsc file is uploaded, the hook email-changes will be executed. It should be in the conf/ directory. The Origin and Label lines (in ) are quite important to be able to define an explicit policy of which packages should be installed. Let s say we use the following /etc/apt/sources.list file:
# Ubuntu packages
deb http://packages.dm.gg/dailymotion precise main restricted universe multiverse
# Dailymotion packages
deb http://packages.dm.gg/dailymotion precise-prod    main role/dns
deb http://packages.dm.gg/dailymotion precise-staging main role/dns
All servers have the precise-staging distribution. We must ensure we won t install those packages by mistake. The NotAutomatic flag is one possible safe-guard. We also use a tailored /etc/apt/preferences:
Explanation: Dailymotion packages of a specific component should be more preferred
Package: *
Pin: release o=Dailymotion, l=dm-prod, c=role/*
Pin-Priority: 950
Explanation: Dailymotion packages should be preferred
Package: *
Pin: release o=Dailymotion, l=dm-prod
Pin-Priority: 900
Explanation: staging should never be preferred
Package: *
Pin: release o=Dailymotion, l=dm-staging
Pin-Priority: -100
By default, packages will have a priority of 500. By setting a priority of -100 to the staging distribution, we ensure the packages cannot be installed at all. This is stronger than NotAutomatic which sets the priority to 1. When a package exists in Ubuntu and in our local repository, we ensure that, if this is a production package, we will use ours by using a priority of 900 (or 950 if we match a specific role component). Have a look at the How APT Interprets Priorities section of apt_preferences(5) manual page for additional information. Keep in mind that version matters only when the priority is the same. To check if everything works as you expect, use apt-cache policy:
$ apt-cache policy php5-memcache
  Installed: 3.0.8-1~precise2~dm1
  Candidate: 3.0.8-1~precise2~dm1
  Version table:
 *** 3.0.8-1~precise2~dm1 0
        950 http://packages.dm.gg/dailymotion/ precise-prod/role/web amd64 Packages
        100 /var/lib/dpkg/status
     3.0.8-1~precise1~dm4 0
        900 http://packages.dm.gg/dailymotion/ precise-prod/main amd64 Packages
       -100 http://packages.dm.gg/dailymotion/ precise-staging/main amd64 Packages
     3.0.6-1 0
        500 http://packages.dm.gg/dailymotion/ precise/universe amd64 Packages
If we want to install a package from the staging distribution, we can use apt-get with the -t precise-staging option to raise the priority of this distribution to 990. Once you have tested your package, you can copy it from the staging distribution to the production distribution:
$ reprepro -C main copysrc precise-prod precise-staging wackadoodle

Local mirror of third-party repositories Sometimes, you want a software published on some third-party repository without to repackage it yourself. A common example is the repositories edited by hardware vendors. Like for an Ubuntu mirror, there are two steps: defining the distribution and defining the source. We chose to put such mirrors into the same distributions as our local packages but with a dedicated component for each mirror. This way, those third-party packages will share the same workflow as our local packages: they will appear in the staging distribution, we validate them and copy them to the production distribution. The first step is to add the components and an appropriate Update line to conf/distributions:
Origin: Dailymotion
Label: dm-staging
Suite: precise-staging
Components: main role/dns role/database role/web vendor/hp
Update: hp
# [...]
Origin: Dailymotion
Label: dm-prod
Suite: precise-prod
Components: main role/dns role/database role/web vendor/hp
# [...]
We added the vendor/hp component to both the staging and the production distributions. However, only the staging distribution gets an Update line (remember, packages will be copied manually into the production distribution). We declare the source in conf/updates:
# HP repository
Name: hp
Method: http://downloads.linux.hp.com/SDR/downloads/ManagementComponentPack/
Suite: precise/current
Components: non-free>vendor/hp
Architectures: i386 amd64
VerifyRelease: 2689B887
GetInRelease: no
Don t forget to add the GPG key to your local keyring. Notice an interesting feature of reprepro: we copy the remote non-free component to our local vendor/hp component. Then, you can synchronize the mirror with reprepro update. Once the packages have been tested, you will have to copy them in the production distribution.

Building Debian packages Our reprepro setup seems complete, but how do we put packages into the staging distribution? You have several options to build Debian packages for your local repository. It really depends on how much time you want to invest in this activity:
  1. Build packages from source by adding a debian/ directory. This is the classic way of building Debian packages. You can start from scratch or use an existing package as a base. In the latest case, the package can be from the official archive but for a more recent distribution or a backport or from an unofficial repository.
  2. Use a tool that will create a binary package from a directory, like fpm. Such a tool will try to guess a lot of things to minimize your work. It can even download everything for you.
There is no universal solution. If you don t have the time budget for building packages from source, have a look at fpm. I would advise you to use the first approach when possible because you will get those perks for free:
  • You keep the sources in your repository. Whenever you need to rebuild something to fix an emergency bug, you won t have to hunt the sources which may be unavailable when you need them the most. Of course, this only works if you build packages that don t download stuff directly from the Internet.
  • You also keep the recipe3 to build the package in your repository. If someone enables some option and rebuild the package, you won t accidently drop this option on the next build. Those changes can be documented in debian/changelog. Moreover, you can use a version control software for the whole debian/ directory.
  • You can propose your package for inclusion into Debian. This will help many people once the package hits the archive.

Builders We chose pbuilder as a builder4. Its setup is quite straightforward. Here is our /etc/pbuilderrc:
DISTRIBUTION=$DIST
NAME="$DIST-$ARCH"
MIRRORSITE=http://packages.dm.gg/dailymotion
COMPONENTS=("main" "restricted" "universe" "multiverse")
OTHERMIRROR="deb http://packages.dm.gg/dailymotion $ DIST -staging main"
HOOKDIR=/etc/pbuilder/hooks.d
BASE=/var/cache/pbuilder/dailymotion
BASETGZ=$BASE/$NAME/base.tgz
BUILDRESULT=$BASE/$NAME/results/
APTCACHE=$BASE/$NAME/aptcache/
DEBBUILDOPTS="-sa"
KEYRING="/usr/share/keyrings/dailymotion-archive.keyring.gpg"
DEBOOTSTRAPOPTS=("--arch" "$ARCH" "--variant=buildd" "$ DEBOOTSTRAPOPTS[@] " "--keyring=$KEYRING")
APTKEYRINGS=("$KEYRING")
EXTRAPACKAGES=("dailymotion-archive-keyring")
pbuilder is expected to be invoked with DIST, ARCH and optionally ROLE environment variables. Building the initial bases can be done like this:
for ARCH in i386 amd64; do
  for DIST in precise; do
    export ARCH
    export DIST
    pbuilder --create
  done
done
We don t create a base for each role. Instead, we use a D hook to add the appropriate source:
#!/bin/bash
[ -z "$ROLE" ]    
  cat >> /etc/apt/sources.list <<EOF
deb http://packages.dm.gg/dailymotion $ DIST -staging role/$ ROLE 
EOF
 
apt-get update
We ensure packages from our staging distribution are preferred over other packages by adding an /etc/apt/preferences file in a E hook:
#!/bin/bash
cat > /etc/apt/preferences <<EOF
Explanation: Dailymotion packages are of higher priority
Package: *
Pin: release o=Dailymotion
Pin-Priority: 900
EOF
We also use a C hook to get a shell in case there is an error. This is convenient to debug a problem:
#!/bin/bash
apt-get install -y --force-yes vim less
cd /tmp/buildd/*/debian/..
/bin/bash < /dev/tty > /dev/tty 2> /dev/tty
A manual build can be run with:
$ ARCH=amd64 DIST=precise ROLE=web pbuilder \
>         --build somepackage.dsc

Version numbering To avoid to apply complex rules to chose a version number for a package, we chose to treat everything as a backport, even in-house software. We use the following scheme: X-Y~preciseZ+dmW.
  • X is the upstream version5.
  • Y is the Debian version. If there is no Debian version, use 0.
  • Z is the Ubuntu backport version. Again, if such a version doesn t exist, use 0.
  • W is our version of the package. We increment it when we make a change to the packaging. This is the only number we are allowed to control. All the others are set by an upstream entity, unless it doesn t exist and in this case, you use 0.
Let s suppose you need to backport wackadoodle. It is available in a more recent version of Ubuntu as 1.4-3. Your first backport will be 1.4-3~precise0+dm1. After a change to the packaging, the version will be 1.4-3~precise0+dm2. A new upstream version 1.5 is available and you need it. You will use 1.5-0~precise0+dm1. Later, this new upstream version will be available in some version of Ubuntu as 1.5-3ubuntu1. You will rebase your changes on this version and get 1.5-3ubuntu1~precise0+dm1. When using Debian instead of Ubuntu, a compatible convention could be : X-Y~bpo70+Z~dm+W.

Uploading To upload a package, a common setup is the following workflow:
  1. Upload the source package to an incoming directory.
  2. reprepro will notice the source package, check its correctness (signature, distribution) and put it in the archive.
  3. The builder will notice a new package needs to be built and build it.
  4. Once the package is built, the builder will upload the result to the incoming directory.
  5. reprepro will notice again the new binary package and integrate it in the archive.
This workflow has the disadvantage to have many moving pieces and to leave the user in the dark while the compilation is in progress. As an alternative, a simple script can be used to execute each step synchronously. The user can follow on their terminal that everything works as expected. Once we have the .changes file, the build script just issues the appropriate command to include the result in the archive:
$ reprepro -C main include precise-staging \
>      wackadoodle_1.4-3~precise0+dm4_amd64.changes
Happy hacking!

  1. The gpg/ directory could be shared by several repositories.
  2. We teached Debian Installer to work with our setup with an appropriate preseed file.
  3. fpm-cookery is a convenient tool to write recipes for fpm, similar to Homebrew or a BSD port tree. It could be used to achieve the same goal.
  4. sbuild is an alternative to pbuilder and is the official builder for both Debian and Ubuntu. Historically, pbuilder was more focused on developers needs.
  5. For a Git snapshot, we use something like 1.4-git20130905+1-ae42dc1 which is a snapshot made after version 1.4 (use 0.0 if no version has ever been released) at the given date. The following 1 is to be able to package different snapshots at the same date while the hash is here in case you need to retrieve the exact snapshot.

26 April 2014

Russell Coker: BTRFS Status April 2014

Since my blog post about BTRFS in March [1] not much has changed for me. Until yesterday I was using 2.13 kernels on all my systems and dealing with the occasional kmail index file corruption problem. Yesterday my main workstation ran out of disk space and went read-only. I started a BTRFS balance which didn t seem to be doing any good because most of the space was actually in use so I deleted a bunch of snapshots. Then my X session aborted (some problem with KDE or the X server I ll never know as logs couldn t be written to disk). I rebooted the system and had kernel threads go into infinite loops with repeated messages about a lack of response for 22 seconds (I should have photographed the screen). When it got into that state the ALT-Fn keys to change a virtual console sometimes worked but nothing else worked the terminal usually didn t respond to input. To try and stop the kernel from entering an infinite loop on every boot that I used rootflags=skip_balance on the kernel command line to stop it from continuing the balance which made the system usable for a little longer, unfortunately the skip_balance mount option doesn t permanently apply, the kernel will keep trying to balance the filesystem on every mount until a btrfs balance cancel operation succeeds. But my attempts to cancel the balance always failed. When I booted my system with skip_balance it would sometimes free some space from the deleted snapshots, after two good runs I got to 17G free. But after that every time I rebooted it would report another Gig or two free (according to btrfs filesystem df ) and then hang without committing the changes to disk. I solved this problem by upgrading my USB rescue image to kernel 3.14 from Debian/Experimental and mounting the filesystem from the rescue image. After letting kernel 3.14 work on the filesystem for a while it was in a stage where I could use it with kernel 3.13 and then boot the system normally to upgrade it to kernel 3.14. I had a minor extra complication due to the fact that I was running apt-get dist-upgrade at the time the filesystem went read-only do the dpkg records of which packages were installed were a bit messed up. But that was easy to fix by running a diff against /var/lib/dpkg/info on a recent snapshot. In retrospect I should have copied from an old snapshot of the root filesystem, but I fixed the problems faster than I could think of better ways to fix them. When running a balance the system had a peak IO rate of about 30MB/s reads and 30MB/s writes. That compares to the maximum contiguous file IO speed of 260MB/s for reads and 320MB/s for writes. During that time it had about 50% CPU time used for my Q8400 quad-core CPU. So far the only tasks that I do regularly which have CPU speed as a significant bottleneck are BTRFS filesystem balancing and recoding MP4 files. Compiling hasn t been an issue because recently I haven t been compiling many programs that are particularly big. Lessons Learned I should photograph the screen regularly when doing things that won t be logged, those kernel error messages might have been useful to me or someone else. The fact that the only kernel that runs BTRFS the way I need comes from the Experimental repository in Debian stands in contrast to the recent kernel patch that stops describing BTRFS as experimental. While I have a high opinion of the people who provide support for the kernel in commercial distributions and their ability to back-port fixes from newer kernels I m concerned about their decision to support BTRFS. I m also dubious about whether we can offer BTRFS support in Debian/Jessie (the next version of Debian) without a significant warning. OTOH if you find yourself with a BTRFS system that isn t working well you could always hire me to fix it. I accept payment via Paypal, bank transfer, or Bitcoin. If you want to pay me in Grange then I assure you I will never forget about it. ;) I thought that I wouldn t have CPU speed issues when I started using the AMD64 architecture, for most tasks that s been the case. But for systems for which storage is important I ll look at getting faster CPUs because of BTRFS. Using faster CPUs for storage isn t that uncommon (I used to work for SGI and dealt with some significant CPU power used for file serving), but needing a fast quad-core CPU to drive a single SSD is a little disappointing. While recovery from file system corner cases isn t going to be particularly common it s something that you want completed quickly, for personal systems you want to be doing something else and for work systems you don t want down-time. The BTRFS problems with running out of disk space are really serious. It seems that even workstations used at home can t survive without monitoring. For any other filesystem used at home you can just let it get full and then delete stuff. Include rootflags=skip_balance in the boot loader configuration for every system with a BTRFS root filesystem and in the /etc/fstab for every non-root BTRFS filesystem. I haven t yet encountered a single situation where continuing the balance did any good or when it didn t do any harm.

25 April 2014

Daniel Pocock: Extra fuel charges when carrying passengers in your car?

Some people seem to be justifying the way mobile networks try to discriminate against tethering. One comment even suggests that multiple devices is a valid justification. Really? Isn't it just as simple as measuring the actual volume of megabytes used, regardless of how many devices are connected? Lets consider a couple of analogies:
  • A petrol station puts some fine print at the bottom of their sign in the street: cars with passengers pay an extra 0.05 per litre (or gallon). It is just a gimmick: the marketing man has worked out that he can put a lower price on his big sign this way and people with passengers may well be splitting the fuel bill between them. There is no technical reason for the charge however - the car actually uses more fuel carrying the extra weight anyway.
  • Or imagine this: the marketing man puts up small print that advises a higher charge of 0.05 per litre for fuelling an exotic car like a Ferrari. The only reason for this, of course, is that the Ferrari driver is probably busy (so he doesn't have time to shop around) and wealthy (so the extra charge won't deter him from making any purchase at all, because he has money to pay it).
Of course, these ideas sound a bit outlandish but the only difference between these crazy ideas and tethering charges is that mobile phone networks usually have the monopoly/duopoly to get away with it. Mobile networks actually dread the idea of becoming commodities like fuel. They thrive on price discrimination and they seem to have an endless number of tricks to get extra money out of people who may have it by deliberately crippling parts of the service for everybody else. This goes against the ideals of innovation and efficiency and in the most extreme cases their practices are widely recognised for the sham that they are. Revealing that you have connected a laptop to your phone may be like revealing you live in an affluent area or that you have an expensive car. Plenty of businesses try to exploit such knowledge when deciding how much they can charge you or other aspects of how they provide you with a service. This is why privacy is important in every feature of the technology we use. This rogue behaviour by "creative" marketing men also helps to understand just what would happen in the world of technology if it wasn't for the use of free licenses, open standards, accessible encryption and net neutrality.

24 April 2014

Daniel Pocock: Android betrays tethering data

When I upgraded an Android device the other day, I found that tethering completely stopped working. The updated CyanogenMod had inherited a new bug from Android, informing the carrier that I was tethering. The carrier, Vodafone Italy, had decided to make my life miserable by blocking that traffic. I had a closer look and managed to find a workaround. There is absolutely no difference, from a technical perspective, between data transmitted from a mobile device on-screen application and data transmitted from tethering. Revealing the use of tethering to the carrier is a massive breach of privacy - yet comments in the Google bug tracker suggest it is a feature rather than a bug. This little table helps put that logic in perspective:
Product Person who carries handset
User Mobile network who wants to discriminate against some types of network traffic to squeeze more money out of the Product
Feature Revealing private information about the way the Product uses his/her Internet so the real User can profit.
It is also bad news for the environment: many people are being tricked into buying un-needed USB dongle modems that will end up in the rubbish in 1-2 years time when their contract expires and the company pushes them to upgrade to the next best thing. Behind the scenes What does it really mean in practice, how does Android tell your carrier which data is from tethering? As my device is rooted and as it is my device and I will do what I want with it, I decided to have a look inside. The ip command revealed that there are now two network devices, rmmnet_usb0 and rmmnet_usb1. The basic ip route command reveals that traffic from different source addresses is handled differently and split over the different network devices:
shell@android:/ # ip route
0.0.0.0/1 dev tun0  scope link
default via 100.66.150.89 dev rmnet_usb0
83.224.66.138 via 100.87.31.214 dev rmnet_usb1
83.224.70.94 via 100.87.31.214 dev rmnet_usb1
100.66.150.88/30 dev rmnet_usb0  proto kernel  scope link  src 100.66.150.90
100.66.150.89 dev rmnet_usb0  scope link
100.87.31.212/30 dev rmnet_usb1  proto kernel  scope link  src 100.87.31.213
100.87.31.214 dev rmnet_usb1  scope link
128.0.0.0/1 dev tun0  scope link
192.168.42.0/24 dev rndis0  proto kernel  scope link  src 192.168.42.129
I then looked more closely and found that there is also an extra routing table, it can be found with ip rule
shell@android:/ # ip rule show
0:      from all lookup local
32765:  from 192.168.42.0/24 lookup 60
32766:  from all lookup main
32767:  from all lookup default
shell@android:/ # ip route show table 60
default via 100.87.51.57 dev rmnet_usb1
100.87.51.57 dev rmnet_usb1
192.168.42.0/24 dev rndis0  scope link
In this routing table, it is obvious that data from the tethering subnet (192.168.42.0/24) is sent over the extra device rmnet_usb1. Manually cleaning it up If the phone is rooted, it is possible to very quickly change the routing table to get all the tethering traffic going through the normal rmnet_usb0 interface again. It is necessary to get rid of that alternative routing table first:
ip rule del pref 32765
and then update the iptables entries that refer to interface names:
iptables -t nat -I natctrl_nat_POSTROUTING -s 192.168.0.0/16 -o rmnet_usb0 -j MASQUERADE
iptables -I natctrl_FORWARD -i rmmnet_usb0 -j RETURN
This immediately resolved the problem for me on the Vodafone network in Italy. Conclusion If Google can be bullied into accepting this kind of discriminatory routing in the stock Android builds and it can even propagate into CyanogenMod, then I'm just glad I'm not running one of those Android builds that has been explicitly "enhanced" by a carrier. It raises big questions about who really is the owner and user of the device and who is receiving the value when a person pays money to "buy" a device.

22 April 2014

Erich Schubert: Kernel-density based outlier detection and the need for customization

Outlier detection (also: anomaly detection, change detection) is an unsupervised data mining task that tries to identify the unexpected.
Most outlier detection methods are based on some notion of density: in an appropriate data representation, "normal" data is expected to cluster, and outliers are expected to be further away from the normal data.
This intuition can be quantified in different ways. Common heuristics include kNN outlier detection and the Local Outlier Factor (which uses a density quotient). One of the directions in my dissertation was to understand (also from a statistical point of view) how the output and the formal structure of these methods can be best understood.
I will present two smaller results of this analysis at the SIAM Data Mining 2014 conference: instead of the very heuristic density estimation found in above methods, we design a method (using the same generalized pattern) that uses a best-practise from statistics: Kernel Density Estimation. We aren't the first to attempt this (c.f. LDF), but we actuall retain the properties of the kernel, whereas the authors of LDF tried to mimic the LOF method too closely, and this way damaged the kernel.
The other result presented in this work is the need to customize. When working with real data, using "library algorithms" will more often than not fail. The reason is that real data isn't as nicely behaved - it's dirty, it seldom is normal distributed. And the problem that we're trying to solve is often much narrower. For best results, we need to integrate our preexisting knowledge of the data into the algorithm. Sometimes we can do so by preprocessing and feature transformation. But sometimes, we can also customize the algorithm easily.
Outlier detection algorithms aren't black magic, or carefully adjusted. They follow a rather simple logic, and this means that we can easily take only parts of these methods, and adjust them as necessary for our problem at hand!
The article persented at SDM will demonstrate such a use case: analyzing 1.2 million traffic accidents in the UK (from data.gov.uk) we are not interested in "classic" density based outliers - this would be a rare traffic accident on a small road somewhere in Scotland. Instead, we're interested in unusual concentrations of traffic accidents, i.e. blackspots.
The generalized pattern can be easily customized for this task. While this data does not allow automatic evaluation, many outliers could be easily verified using Google Earth and search: often, historic imagery on Google Earth showed that the road layout was changed, or that there are many news reports about the dangerous road. The data can also be nicely visualized, and I'd like to share these examples with you. First, here is a screenshot from Google Earth for one of the hotspots (Cherry Lane Roundabout, North of Heathrow airport, which used to be a double cut-through roundabout - one of the cut-throughs was removed since):
Screenshot of Cherry Lane Roundabout hotspot
Google Earth is best for exploring this result, because you can hide and show the density overlay to see the crossroad below; and you can go back in time to access historic imagery. Unfortunately, KML does not allow easy interactions (at least it didn't last time I checked).
I have also put the KML file on Google Drive. It will automatically display it on Google Maps (nice feature of Drive, kudos to Google!), but it should also allow you to download it. I've also explored the data on an Android tablet (but I don't think you can hide elements there, or access historic imagery as in the desktop application).
With a classic outlier detection method, this analysis would not have been possible. However, it was easy to customize the method; and the results are actually more meaningful: instead of relying on some heuristic to choose kernel bandwidth, I opted for choosing the bandwidth by physical arguments: 50 meters is a reasonable bandwidth for a crossroad / roundabout, and for comparison a radius of 2 kilometers is used to model the typical accident density in this region (there should other crossroads within 2 km in Europe).
Since I advocate reproducible science, the source code of the basic method will be in the next ELKI release. For the customization case studies, I plan to share them as a how-to or tutorial type of document in the ELKI wiki; probably also detailing data preprocessing and visualization aspects. The code for the customizations is not really suited for direct inclusion in the ELKI framework, but can serve as an example for advanced usage.
Reference:
E. Schubert, A. Zimek, H.-P. Kriegel
Generalized Outlier Detection with Flexible Kernel Density Estimates
In Proceedings of the 14th SIAM International Conference on Data Mining (SDM), Philadelphia, PA, 2014.
So TLDR of the story: A) try to use more established statistics (such as KDE), and B) don't expect an off-the-shelf solution to do magic, but customize the method for your problem.
P.S. if you happen to know nice post-doc positions in academia:
I'm actively looking for a position to continue my research. I'm working on scaling these methods to larger data and to make them work with various real data that I can find. Open-source, modular and efficient implementations are very important to me, and one of the directions I'd like to investigate is porting these methods to a distributed setting, for example using Spark. In order to get closer to "real" data, I've started to make these approaches work e.g. on textual data, mixed type data, multimedia etc. And of course, I like teaching; which is why I would prefer a position in academia.

Andrew Pollock: [life] Day 84: Kindergarten Term 2 starts, I collapse on the couch for the day

Zoe had her first day back at Kindy this morning for Term 2. I couldn't believe how fast Term 1 flew by, and how little I felt I accomplished on a personal front. The two weeks of school holidays certainly put the brakes on trying to get anything much more done for myself, but Zoe and I had a great time. I was really happy with the variety of activities we were able to do, and it was nice that Zoe and Megan got to spend a reasonable amount of time together too. The weather cooperated for the majority of the time, which was the cherry on top. Zoe only had Kindergarten today this week, but Sarah has the week off, so she's going to be taking Zoe for the next couple of days, which is really convenient timing, as it'll give me time to recover from some minor surgery tomorrow without having to run around after her. I might also manage to finalise my US tax return. I'm hoping to catch a few movies with Anshu too, who also has the week off work. Zoe slept reasonably well last night. Two wake ups, but they were both quickly resolved, so we both got back to sleep quickly. I was absolutely exhausted last night, but felt positively chipper this morning. We biked to Kindergarten, and I decided to leave the trailer there to make things a bit easier for myself in the afternoon. I got home, and just felt like vegging on the couch. Then I remembered Anshu had the day off, so she came over and we hung out and had lunch. It was really nice to have a few hours during the day off. I biked back to Kindergarten, wondering if I'd have to deal with waking up Zoe from a nap, but she hadn't had a nap. We'd had a bit of a talk at breakfast about napping at Kindergarten, and I have no idea if it helped or not, but it meant we could make an orderly departure. Zoe wanted to participate in Megan's tennis class after Kindergarten, and they were down a kid, which made the warm up stuff not work so well, so the teacher was happy for Zoe to take part. I managed to extract her once the real nitty gritty of the class started. I did get a good opportunity to suss out place availability for term 3. We biked home, and I wanted to take my bike in for a service while I'm on lifting restrictions, so we drove over to Cannon Hill to drop it off. The Gold Cross bike shop has now merged into the Super Amart store, so Zoe wanted to look at everything on the way back out. We eventually emerged without buying anything. Next, we went over to Bunnings, because Zoe's been asking if we can grow some veggies and flowers from seed. That ended up being about an hour of trekking around the nursery section trying to find stuff. It was a good way to use up the afternoon. We made it out with a bag of potting mix and a few packets of seeds and a kit for growing stuff that requires a trellis. Unfortunately most of the climbing stuff (like tomatoes) are out of season now, so I'm not sure what we're going to be able to grow with the kit. We got back home, and I put dinner on and we watched a bit of TV together while it cooked. Bath time and bed time went really smoothly, as I think she was pretty tired. Here's hoping she sleeps well tonight.

21 April 2014

Russ Allbery: Review: Hyperbole and a Half

Review: Hyperbole and a Half, by Allie Brosh
Publisher: Touchstone
Copyright: November 2013
ISBN: 1-4767-6459-X
Format: Graphic novel
Pages: 373
It's fundamentally pointless to write this review. Some number of you have already heard of Hyperbole and a Half. Most of that group already own this book and are only reading this review so that you can remember how much fun you had reading it. But, even better, you could just stop reading this review and go read the book again. Some of you who have heard of Hyperbole and a Half didn't know that Allie Brosh had published a book. If that's you, you're probably not reading this review any more, since you're now at your favorite book seller buying a copy of the book, rendering the review somewhat pointless. For those of you who have not heard of Hyperbole and a Half, there's no need to read a review, because it's a blog. So, rather than reading this review, you can just follow that link and read it for yourself. If you find yourself laughing uncontrollably (and occasionally crying) and going "where can I get more of this?", well, there's a book. Which you could have also found out from the blog itself. Like I said, not much point. If you read the blog and don't particularly care for it, well, I greatly respect your position. The diversity of taste in the human race is what leads to our wonderful variety of culture, philosophy, and art, and I appreciate your substantial contribution to that diversity. However, you should now make a mental note to never trust my humor recommendations, and you shouldn't let this review change your mind. But, since I'm here, I may as well write a review anyway. Hyperbole and a Half is, as mentioned, a blog by Allie Brosh. Most of the posts are quite long and substantial and in the form of cartoons mixed with text, usually (but not always) telling some sort of story. The cartoon style is what is often called "MS Paint," meaning that it looks like something drawn in Microsoft Paint with its most basic tools. (Other examples of this style are the Oatmeal and Homestuck, although Brosh stays consistently with a rawer art style than either of those.) It looks extremely simple, like child drawings (and early childhood inspires much of Brosh's material), but once one gets used to it, one realizes that Brosh gets amazing expressiveness and character out of the art style. As an aside, yes, I am wholeheartedly recommending a book that is full of material about young children. Brosh is that good. The subtitle of this book is "unfortunate situations, flawed coping mechanisms, mayhem, and other things that happened," and it's mostly based on (exaggerated) incidents that occurred in Brosh's life. Most of them are hilarious. Some of them are both hilarious and stunning psychological insights. She has an amazing knack for storytelling and for exaggerating just the right moments of the story, or summing up emotions in a wonderful turn of phrase or a picture. She's also amazingly good at telling embarrassing stories about herself in a way that makes you empathize rather than just cringe, and then come away feeling like you understand both her and yourself better. If you've wandered around the Internet much, you have probably run across the phrase "clean ALL the things!" and the corresponding picture. That's from Hyperbole and a Half (specifically, "This is Why I'll Never Be an Adult") and is included in this book (improved; see below). It's even better in context. But, beyond storytelling, the other thing Brosh is amazingly good at is capturing internal mental states and emotions in a way that the reader understands and those who have experienced the emotion immediately go "yes, THAT." The apex of this is her two-part post on depression, which is hands-down the best description of depression that I've ever read. That judgment has been echoed by multiple friends of mine with depression. And yes, both of those posts are included in the book as well, although you can also read them on the web. As you've probably noticed, this book is partly a collection of material that's freely available on the web. Unlike some web comics collections, it's not entirely a reprint collection; there are entirely new stories here (which for me was enough by itself to buy the book). Brosh's post on the book says that it's about 50% new material. But the ones that aren't original have been edited and improved, sometimes substantially. For example, I did a quick comparison of the book version with the web version of "This is Why I'll Never be an Adult" and found Brosh redrew all of the cartoons for the story. The book is also gorgeous, if you care about such things like I do: high quality paper, thick pages, vibrant colors, and a comfortable heft to the hardcover version. It's the sort of book that feels like it will survive for decades, which is good since I intend to re-read it for decades. Brosh has a lot of material on-line, so obviously only a small fraction made it into the book and there's more to discover on the blog. I can particularly recommend "The Alot is Better Than You at Everything" and "Boyfriend Doesn't Have Ebola. Probably.". The latter, which is mostly about creating a better pain scale, is my favorite from the blog that didn't make it into the book. And there are two more examples of the sort of thing that you'll getting here. So ends the probably pointless review. I absolutely adore Hyperbole and a Half. It is my favorite web comic, even surpassing XKCD, which is saying something. Even if there were no material, I would have bought this book in hardcover to support Brosh and to have a high-quality printing of the blog posts. The reworkings and improvements of the republished material and the brand new additions are just bonuses. But there's no need to take my word for it; the best possible advertisement for Brosh's work is on-line, from which you can easily determine if you'll enjoy this book as much as I did. Even the back cover and the inside flaps are awesome. Rating: 10 out of 10

Matthew Garrett: Home entertainment implementations are pretty appalling

I picked up a Panasonic BDT-230 a couple of months ago. Then I discovered that even though it appeared fairly straightforward to make it DVD region free (I have a large pile of PAL region 2 DVDs), the US models refuse to play back PAL content. We live in an era of software-defined functionality. While Panasonic could have designed a separate hardware SKU with a hard block on PAL output, that would seem like unnecessary expense. So, playing with the firmware seemed like a reasonable start.

Panasonic provide a nice download site for firmware updates, so I grabbed the most recent and set to work. Binwalk found a squashfs filesystem, which was a good sign. Less good was the block at the end of the firmware with "RSA" written around it in large letters. The simple approach of hacking the firmware, building a new image and flashing it to the device didn't appear likely to work.

Which left dealing with the installed software. The BDT-230 is based on a Mediatek chipset, and like most (all?) Mediatek systems runs a large binary called "bdpprog" that spawns about eleventy billion threads and does pretty much everything. Runnings strings over that showed, well, rather a lot, but most promisingly included a reference to "/mnt/sda1/vudu/vudu.sh". Other references to /mnt/sda1 made it pretty clear that it was the mount point for USB mass storage. There were a couple of other constraints that had to be satisfied, but soon attempting to run Vudu was actually setting a blank root password and launching telnetd.

/acfg/config_file_global.txt was the next stop. This is a set of tokens and values with useful looking names like "IDX_GB_PTT_COUNTRYCODE". I tried changing the values, but unfortunately made a poor guess - on next reboot, the player had reset itself to DVD region 5, Blu Ray region C and was talking to me in Russian. More inconveniently, the Vudu icon had vanished and I couldn't launch a shell any more.

But where there's one obvious mechanism for running arbitrary code, there's probably another. /usr/local/bin/browser.sh contained the wonderful line:
export LD_PRELOAD=/mnt/sda1/bbb/libSegFault.so
, so then it was just a matter of building a library that hooked open() and launched inetd and dropping that into the right place, and then opening the browser.

This time I set the country code correctly, rebooted and now I can actually watch Monkey Dust again. Hurrah! But, at the same time, concerning. This software has been written without any concern for security, and it listens on the network by default. If it took me this little time to find two entirely independent ways to run arbitrary code on the device, it doesn't seem like a stretch to believe that there are probably other vulnerabilities that can be exploited with less need for physical access.

The depressing part of this is that there's no reason to believe that Panasonic are especially bad here - especially since a large number of vendors are shipping much the same Mediatek code, and so probably have similar (if not identical) issues. The future is made up of network-connected appliances that are using your electricity to mine somebody else's Dogecoin. Our nightmarish dystopia may be stranger than expected.

comment count unavailable comments

19 April 2014

Thomas Goirand: OpenStack 2014.1, aka Icehouse, is out

The new version of OpenStack is out, and I have just finished uploading it all into Debian Sid. With a total of 38 packages that I uploaded yesterday (which was exhausting!), most, if not all, were only moving from Experimental to Sid with only tiny updates, and this represents the achievement of 6 months of packaging work. The new feature list is impressive, and I would like to highlight some part of it: For the moment, I haven t packaged Sahara (eg: Hadoop as a service), but it might come later as a customer of us might require it. There s a lot less unit tests issues in the packages I uploaded to Sid: all SQLAlchemy issues have been dealt with. I wasn t confident with the Havana release that Sid / Testing would be a good environment for OpenStack, but this time with Icehouse, I think it should be much better. Please test this brand new release and report issues on the BTS. As always, the packages are available also as Wheezy backports through the usual channels (see the official install guide).

16 April 2014

Wouter Verhelst: Call for help for DVswitch maintenance

I've taken over "maintaining" DVswitch from Ben Hutchings a few years ago, since Ben realized he didn't have the time anymore to work on it well. After a number of years, I have to admit that I haven't done a very good job. Not becase I didn't want to work on it, but mainly because I don't have enough time to fix DVswitch against the numerous moving targets that it uses; the APIs of libav and of liblivemedia are fluent enough that just making sure everything remains compilable and in working order is quite a job. DVswitch is used by many people; DebConf, FOSDEM, and the CCC are just a few examples, but I know of at least three more. Most of these (apart from DebConf and FOSDEM) maintain local patches which I've been wanting to merge into the upstream version of dvswitch. However, my time is limited, and over the past few years I've not been able to get dvswitch into a state where I confidently felt I could upload it into Debian unstable for a release. One step we took in order to get that closer was to remove the liblivemedia dependency (which implied removing the support for RTSP sources). Unfortunately, the resulting situation wasn't good enough yet, since libav had changed API enough that current versions of DVswitch compiled against current versions of libav will segfault if you try to do anything useful. I must admit to myself that I don't have the time and/or skill set to maintain DVswitch on an acceptable level all by myself. So, this is a call for help: If you're using DVswitch for your conference and want to continue doing so, please talk to us. The first things we'll need to do: See you there?

Andrew Pollock: [life] Day 78: Alginate, dragon boats and relatives

I ordered some alginate the other day, and it arrived yesterday, but we were out, so I had to pick it up from the post office this morning. Anshu and I picked it up before Zoe was dropped off. We had a couple of attempts at making some, but didn't quite get the ratios or the quantity right, and we were too slow, so we'll have to try again. The plan is to try and make a cast of Zoe's hand, since we were messing around with plaster of Paris recently. I've found a good Instructable to try and follow. Nana and her dragon boating team were competing in the Australian Dragon Boat Championships over Easter, and her first race was today. It also ended up that today was the best day to try and go and watch, so when she called to say her first race would be around noon, I quickly decided we should jump in the car and head up to Kawana Waters. We abandoned the alginate, and I slapped together a picnic lunch for Zoe and I, and we bid Anshu farewell and drove up. Zoe's fever seemed to break yesterday afternoon after Sarah picked her up, and she slept well, but despite all that, she napped in the car on the way up, which was highly unusual, but helped pass the time. She woke up when we arrived. I managed to get a car park not too far from the finish line, and we managed to find Nana, whose team was about the enter the marshaling area. Her boat was closest to the shore we were watching from, and her boat came second in their qualifying round for the 200 metre race, meaning they went straight through to the semi-finals. The semi-finals were going to be much later, and I wanted to capitalise on the fact that we were going to have to drive right past my Mum and Dad's place on the way home to try and see my sister and her family, since we missed them on Monday. We headed back after lunch and a little bit of splashing around in the lake, and ended up staying for dinner at Mum and Dad's. Zoe had a great time catching up with her cousin Emma, and fooling around with Grandpa and Uncle Michael. She got to bed a little bit late by the time we got home, but I'm hopeful she'll sleep well tonight.

15 April 2014

Dirk Eddelbuettel: BH release 1.54.0-2

Yesterday's release of RcppBDT 0.2.3 lead to an odd build error. If one used at the same time a 32-bit OS, a compiler as recent as g++ 4.7 and the Boost 1.54.0 headers (directly or via the BH package) then the file lexical_cast.hpp barked and failed to compile for lack of an 128-bit integer (which is not a surprise on a 32-bit OS). After looking at this for a bit, and looking at some related bug report, I came up with a simple fix (which I mentioned in an update to the RcppBDT 0.2.3 release post). Sleeping over it, and comparing to the Boost 1.55 file, showed that the hunch was right, and I have since made a new release 1.54.0-2 of the BH package which contains the fix.
Changes in version 1.54.0-2 (2014-04-14)
  • Bug fix to lexical_cast.hpp which now uses the test for INT128 which the rest of Boost uses, consistent with Boost 1.55 too.
Courtesy of CRANberries, there is also a diffstat report for the most recent release. Comments and suggestions are welcome via the mailing list or issue tracker at the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Next.

Previous.