Search Results: "beh"

2 June 2014

Francesca Ciceri: 100 bugs triaged: achievement unlocked

After two weeks working as OPW intern for Mozilla, it's time for a recap!
What exactly I've been doing in these two weeks? 100 bugs triaged: achievement unlocked! Yes, this is the thing I'm most proud of. I'm a bit cheating here, as strictly speaking, since the beginning of the internship I've triaged only 44 bugs.
But I've decided to count from the beginning of my activity on bugzilla, at the end of March, since I've started work on that as part of the small contribution required for applying to OPW.
Therefore, it's all OPW related :) Here's the grand total. Right now, I've decided to work on an average of 5 bugs a day: it's mostly triage and/or verification, which is quite fun.
It consists in trying to have a more complete and detailed bug report for the developers: asking the right questions to the reporter, ensuring that the bug is filed against the right product or component and all the information about platforms and version are correct. Or verifying that the bug isn't a duplicate, which involves doing some voodoo with Bugzilla quicksearch (I'm not so good with that yet, mostly because I'm not imaginative enough in the queries... but I'm getting better!) Sometimes triaging means reading lots of documentation (to be sure that something is a bug and not a feature) and checking meta-bugs and release notes to be able to pinpoint the time when something was introduced and the reasoning behind it.
That takes a lot of time, but it makes you discover some funny things, like the Mighty Bouncing Unicorn. And while I know it sounds a bit cruel, it's really good when you're verifying a fix and you find it's not totally ok, or that it triggered another bug.
I've been assured that feeling satisfied after that it's an essential part of the sadistic QA work. Writing FAQs for new triager This started as a personal project even before knowing I've been selected for OPW, and it's now part of my internship: I've been writing a first draft of FAQ for those who approach for the first time the Bug Triaging and Verifying work in Mozilla. It meant taking a whole lot of IRC logs and scan them for the most asked questions during bugdays, and you can find here my first draft. I'll send a RFC today about it on dev-quality mailing list and link it to the main Bugdays page. Lessons learned So, what I've learned in these two weeks? That I'm pretty good at figuring things alone, but I like to have feedback on what I'm working on.
That testing things is an art, and perfectionism is a big plus.
That there are such things as stupid questions, but you have to ask them nonetheless.
That people in the Mozilla community are quite friendly and not scary at all. Not even in video! :) Wishlist I've been thinking about this a lot, and I think I'd like to have They will probably become my next pet project.

1 June 2014

Russ Allbery: Spring haul post

Sorry about the lack of posting recently. Work has been rather problematic, and also very distracting, and on my time away from work I've been relaxing rather than writing. Didn't even get reviews written while on vacation. I am still hoping to catch up on reviews, however. In the meantime, I have been reading some, and buying more books. Here's a new haul list. Elizabeth Bear Steles of the Sky (sff)
Katherine Boo Behind the Beautiful Forevers (non-fiction)
John Chu The Water That Falls on You from Nowhere (sff)
Lewis Dartnell The Knowledge (non-fiction)
David Lochbaum, et al. Fukushima (non-fiction)
Ben Goldacre Bad Pharma (non-fiction)
Mira Grant Parasite (sff)
Thomas Olde Heuvelt The Ink Readers of Doi Saket (sff)
Mary Robinette Kowal The Lady Astronaut of Mars (sff)
Madeline Miller The Song of Achilles (historical)
Edmund S. Morgan American Slavery, American Freedom (non-fiction)
Richard Rhodes The Making of the Atomic Bomb (non-fiction)
Lee Smolin The Trouble with Physics (non-fiction)
Charles Stross Equoid (sff)
Charles Stross Neptune's Brood (sff) This is a bit of a grab-bag, including a variety of non-fiction that I had recommended to me by various people. I also picked up the remaining Hugo nominees for best novel that won't be in the voter packet, and the short fiction from Tor.com that was nominated and was available for free from Amazon for the Kindle. I read The Knowledge while I was on vacation, and it's in my large backlog of pending reviews.

28 May 2014

John Goerzen: Review: The Wheel of Time

Five years ago, I opened The Eye of the World, the first book in the Wheel of Time series, and read:
Lord of the Morning, he said, I have come for you.

On the island, the air shimmered and coalesced. The black-clad man stood staring at the fiery mountain rising out of the plain. His face twisted in rage and contempt. You cannot escape so easily, Dragon. It is not done between us. It will not be done until the end of time. Then he was gone, and the mountain and the island stood alone. Waiting.
And then, as often, Robert Jordan began books citing some of the prophecy he invented:
And it came to pass in those days, as it had come before and would come again, that the Dark lay heavy on the land and weighed down the hearts of men, and the green things failed, and hope died. And men cried out to the Creator, saying, O Light of the Heavens, Light of the World, let the Promised One be born of the mountain, according to the prophecies, as he was in ages past and will be in ages to come. Let the Prince of the Morning sing to the land that green things will grow and the valleys give forth lambs. Let the arm of the Lord of the Dawn shelter us from the Dark, and the great sword of justice defend us. Let the Dragon ride again on the winds of time. from Charal Drianaan to Calamon,
The Cycle of the Dragon.
Author unknown, the Fourth Age
And then, the grand opening to each of the books in the 11,000-page series, something like this:
The Wheel of Time turns, and Ages come and pass, leaving memories that become legend. Legend fades to myth, and even myth is long forgotten when the Age that gave it birth comes again. In one Age, called the Third Age by some, an Age yet to come, an Age long past, a wind rose in the Mountains of Mist. The wind was not the beginning. There are neither beginnings nor endings to the turning of the Wheel of Time. But it was a beginning. Born below the ever cloud-capped peaks that gave the mountains their name, the wind blew east, out across the Sand Hills, once the shore of a great ocean, before the Breaking of the World. Down it flailed into the Two Rivers, into the tangled forest called the Westwood, and beat at two men walking with a cart and horse down the rock-strewn track called the Quarry Road.
And yesterday, 11,000 pages later, I have completed the series. Really, how could I have put down a book with such a majestic beginning as that? It hinted at the epic that was to come, the amazing literature I was going to read. I didn t read it straight through; I enjoyed books like War and Peace along the way. But what a trip the Wheel of Time series was. The characters were so detailed, so deep, the setting so elaborate, the plots so grand and also so small everyone from servants to queens were well-represented in the books. Not every section was a page-turner; in fact, around the 4th or 5th book, the series started to plod for a thousand pages or two, reading more like a history book than a novel. Sometimes the prejudices and stereotypes of the characters, particularly the sexism that both the men and women displayed, were frustrating to read. I never quite figured out of this was brilliant writing (making the reader feel frustrated at the plodding pace of the book when it was describing characters being frustrated at the plodding pace of events), or if it could have been trimmed down by 2/3 in some places. But really, the number of brilliant endings, laugh-out-loud situations, and page-turning suspense that somehow managed to build over hundreds, and then even thousands of pages, outweighed the flaws.
Politics is a foggy mire full of snakes. - The Eye of the World
Ah. Furlan washed his hands in one direction, then rubbed them in the other. Ah. Forgive me, Master Andra, but Lady Alys is a fierce sounding woman. Only with those who displease her, Lan said blandly. Her bite is far worse than her bark. Ah. Ah. Ah. Your rooms are this way. . . - The Dragon Reborn
One of the marks of truly good writing is that I leave a book or series with a sense of sadness or nostalgia, leaving characters that seem almost like friends. I felt that after finishing the last book in the series, A Memory of Light. The ending was deep, and satisfying, but ultimately left me wanting even more even more than 11,000 pages, somehow.
Elienda and Briain woke from the dream. She might have been speaking of the weather rather than two deaths among women she knew. We all must wake eventually. - Knife of Dreams
I was often touched by the way the Aiel referred to each other. A desert people, their fond farewall was may you always find water and shade. A fitting metaphor for anyone s life, I think. And they referred to their spouse as shade of my heart. Jordan touched on quite a few themes of religion and philosophy. In that quote above, he has the Aiel referring to death as waking from a dream. An intriguing metaphor, isn t it? He even writes conflicts of philosophy, religion, and culture into the series. I was particularly impressed with his treatment of theodicy and the problem of evil (the argument and religious response relating to the presence of evil in the world in the face of a loving God.) It is the most impressive treatment I have seen in fiction, and does a better job of advancing the theodistic argument than many philosophers have, I suspect. It is hard to really go into that one here because of the length and potential for spoilers. I can, however, comment on his treatment of existential nihilism the theory that life and the universe is without meaning, and the sometimes-related comment that what does anything matter then anyway? Here are a couple of quotes:
In Maradon, I saw what had been done to men who followed me. I saw Light in them, Min. Defying the Dark One no matter the length of his shadow. We will live, that defiance said. We will love and we will hope. And I saw him trying so hard to destroy that. He knows that if he could break them, it would mean something. Something much more than Maradon. Breaking the spirit of men he thirsts for that. He struck far harder than he otherwise would have because he wanted to break my spirit. His voice grew softer and he opened his eyes, looking down at her. And so I stood against him. - Towers of Midnight
What if I think it s all meaningless? he demanded with the loud voice of a king. What if I don t want it to keep turning? We live our lives by the blood of others! And those others become forgotten. What good is it if everything we know will fade? Great deeds or great tragedies, neither means anything! They will become legends, then those legends will be forgotten, then it will all start over again!

Why? Rand thought with wonder. Because each time we live, we get to love again.

It felt relaxing to stare out at that distant light, so welcoming and noble. Storms will soon come, it seemed to say. But for now, I am here. I am here. - The Gathering Storm
I unfortunately can t put in in its proper context due to spoilers, but it is a pivotal turning point in the series, relating to the preservation of the world. By giving what we might normally call a characteristic or behavior evil an almost-human voice and character, it can be discussed in interesting ways.
We are always more afraid than we wish to be, but we can always be braver than we expect. Hold on to your heart, and the Aes Sedai cannot harm what is really you, your heart. They are not nearly so far above us as we believed. May you always find water and shade, Egwene. And always remember your heart. - Lord of Chaos
Eben is dead. Would you want to forget your pain if you lost that hulking giant of yours? Have your feelings for him cut away like some spoiled chunk of flesh in an otherwise good roast? - The Gathering Storm
These quotes are examples of characters in the series dealing with difficult situations. In one, a character is yelled at for offering to use her special powers to numb the pain of another person over a death. It brings up a question that those of us that have known pain might find interesting: would we really have wanted our pain erased? And suggests perhaps not. That there is a benefit to it. That we get through it by embracing it by embracing it while still being true to our heart.
He had four rules concerning action and information. Never make a plan without knowing as much as you can of the enemy. Never be afraid to change your plans when you receive new information. Never believe you know everything. And never wait to know everything. - Lord of Chaos
Sometimes the series treats us to logical thinkers, whether they are generals or innkeepers, and gives us little tidbits like this one.
Androl brushed off his hands, smiling. Children were so adaptable. Before them, centuries of tradition, terror and superstition could melt away like butter left too long in the sun. - Towers of Midnight
And sometimes, things to chew over. Isn t that an interesting comment on prejudice? Children aren t born prejudiced; we can see how easily they make friends with anyone. It is sad that we teach them to be, isn t it? Wheel of Time is a fine story, expertly woven. But it is not just a story. It is true, in the sense that it illuminates truths, just like Romeo and Juliet can. In it, I see lessons and echoes of the civil rights era, comments on what makes good leaders, characters wrestling with decisions without adequate data, people putting on a mask of emotions, sacrifice, and the overall theme of the entire series: the power of love and compassion. The inspiration of the series for me well, it is there, but it may take a few more days to figure it out. I think it has something to do with this: The wind rose high and free, to soar in an open sky with no clouds.

22 May 2014

Martín Ferrari: Yakker, part 4: the client application

This is the fourth post in a series of posts (part 1, part 2, part 3) describing a secure alternative to applications like WhatsApp. I started with the following statement:
I believe it is possible to build a system with the simplicity and functionality of WhatsApp or Viber, which provides end-to-end encryption, is built on free software and open protocols, that supports federation and is almost decentralised, and that would allow interested companies to turn a profit without compromising any of these principles.
Now that most of the infrastructure has been described, in this post I will talk about the user-visible part: a mobile SIP client specially tailored for this architecture.

Features For Yakker to be successful, an application that is visually attractive and simple to use, while providing excellent call quality and stability, is critical. This idea is useless if only geeks adopt it: I want my parents to use it, I want my non-techie friends to use it. I want to tell them to switch from any of the other applications, not only because it is more secure, open and community-based, but also because it is better. For Android, there is already some excellent free applications that could be used as a starting point: Lumicall, CSipSimple and Linphone. I haven't tried it yet, but Linphone has a port to iOS too. Apart from the quality considerations, the following features must be added:
  • Certificate creation and proper storage in the mobile device.
  • An account creation wizard that interacts with the directory service and the account providers. Lumicall currently does something like this already, but only for their own ENUM and SIP server only.
  • Proper DNSSEC validating resolver to securely get the callee's certificate and ENUM record.
  • Optionally, use the directory service API to query records instead of DNS, to enhance privacy.
  • Periodic verification of the user's own ENUM and certificate records in DNS.
  • Use of those certificates to set up the SRTP stream, instead of unauthenticated encryption. Text messages must be encrypted in the same way, but included in the SIP message. SIPs encryption is not enough, as the proxies and service providers can read them.
  • Integration with the system's address book.
  • ENUM lookup, and SIP SRV lookup to detect phones with an associated SIP account.
  • When the called party does not have a SIP account, the client must offer to call using the PSTN gateway at service provider, if it provides one, and to call using the GSM network. It needs to be fast and reliable, so people can use it as the default texting and calling application.
  • Provide an interface to query account's balance, calling rates, and to buy credit. Possibly, offer this by opening a web browser, but negotiating authentication first, so the user does not need to enter an user name or password.
  • Capabilities to migrate to a different service provider.
  • A secure way to share the key pair with another trusted device.
  • A way to import a key pair created by the user manually.
I am probably missing a bunch of other capabilities that need to be implemented. It is a lot of work. Some of these features would only be useful for participants of the Yakker network, but there are others that could be useful for every SIP user, and therefore, implemented first. For example, SIP accounts that are not associated with a phone number could publish certificates under the same domain, and have the client use them to have secure communications with legacy infrastructure.

Funding I am aware that this amount of work is not going to happen overnight. In fact, without a bunch of people from the community interested in the project, it is never going to happen. The good news is that I think that companies might be interested in investing in this. A company could create their own branded version of the client, and use advertisements or service provider preference (the provider the user gets unless they choose one manually) to generate income. As long as the code remains free, and the client is compatible with the whole system, there could many competing clients out there. Their own promotion schemes would work for the benefit of the whole system, by bringing more users.

Risks The biggest threat would be of one client monopolising the network, and then changing the protocols to make the system a walled garden. It has already happened with GTalk, so there is precedent for this. I don't think there is any way to stop a big company from doing this, but one strategy to mitigate the risk a bit would be to create a brand (Yakker or whatever this ends up being called) and have it managed by a trusted community organisation, which can revoke the right to use the brand when a party is not behaving.

Ean Schuessler: Roboneck gets a nod

All the motors start talking to Blender through ROS. Some interesting shaky behavior and a few issues with inverse kinematics are revealed.
http://youtu.be/3vcycClGDso

14 May 2014

Martín Ferrari: Yakker, part 2: the directory service

This is the second post in a series of posts describing a secure alternative to applications like WhatsApp. I started with the following statement:
I believe it is possible to build a system with the simplicity and functionality of WhatsApp or Viber, which provides end-to-end encryption, is built on free software and open protocols, that supports federation and is almost decentralised, and that would allow interested companies to turn a profit without compromising any of these principles.
In this post, I will outline the concepts behind the most critical component of the architecture: the directory service.

Outline I say this is the most critical component, because here lies what makes this architecture different, but also because this is the weakest link in the whole idea. I expect criticism, specially on some security trade-offs, and I hope that people that know better than me can help me improve it. The directory service is what allows the users to register easily, without using passwords, to be able to receive calls and messages from other users, even if they are not part of the network, and to do all this with a reasonable security model.

Let's get technical The directory service is basically a DNSSEC-protected DNS zone serving ENUM records, along with public keys associated with each user identifier. A TLS-enabled API will enable account creation and validation, DNS records publishing, and encrypted records querying. Applications not supporting that API use standard ENUM querying, and the user manually uses traditional web-based methods for account and records management. This allows interoperability with any existing clients and SIP services. The service will authenticate users by using the usual methods to probe ownership of phone numbers: sending an encrypted SMS with a secret that the client application uses then to claim the phone number. This same method can be used to validate ownership of identifiers that are not phone numbers, like pre-existing SIP or email addresses. Once the user is authenticated, the directory service will publish the user's public key and SIP address, both associated with the phone number. When a user wants to place a call or send a message, it uses a DNSSEC-enabled resolver to get securely the other party's public key and SIP address. The user can also perform bulk look-ups, to discover which people on theirs address book is already in the system. These operations disclose an important amount of private data, and I don't think this can be mitigated in an acceptable manner, and therefore the directory service needs to forget the queries as soon as possible, and not to store any logs of these. Also, DNS queries are not encrypted, and are thus vulnerable to snooping by third parties. To mitigate this, the service needs to implement an encrypted but anonymous API to perform queries, and thus offer extra privacy to clients that support the API. The idea is to have more than one service running on different domain names, but not many: they need to keep consistency and replicate among them, and the security and privacy implications of one service not being properly implemented or administered are too big. Therefore, there must not be more than a handful of these, they need to be properly audited, and must not be operated by any for-profit organisation.

Security The Web of Trust is hard. End users don't like hard. Traditional PKI is prohibitely costly, and broken. OTR is good, but is still not hassle-free. Using one of the proposed extensions to DANE, we can solve this problem: using DNSSEC, we can have each users' public key published where everybody can retrieve it securely. This published key can then be used for end-to-end encryption and client authentication with all the components of this architecture. Who creates the key pair? In the simple case, the client application creates the key pair, and stores it in an appropriate container in the mobile device. It then chooses one of many cooperating directory services, and gets the public key associated with the phone number. If the private key is lost (lost your phone?), a new pair is generated and published, after passing the same checks of number ownership, and the old keys are discarded. This opens the door for some attacks, but those can be mitigated by having the client application verify the public records periodically, and the service requiring extra checks for key replacement. For example, by also using a challenge sent by email. What if I want to do things my way? Perfect, you create your keys, and then use the same mechanisms to register with the identity service. Also, you use the service to tell other users to connect to your own SIP server when they want to talk to your phone number. Does this sound like ENUM? It is ENUM, but better. Just by adding DNSSEC, an unified API, and the capability of publishing key material along with the routing information, you got yourself a reasonably secure way of distributing keys and locating users.

Interaction with other components Once the user records are published, the SIP server can use the public key to authenticate the user, and removes the need for passwords. The same principle applies to account creation: if the user has published key material under a phone number's record, the SIP server must accept account creation requests for the same phone number, provided these are authenticated with the private key. If the user wants to switch providers, it is as simple as creating a new account in the new provider, and then updating the ENUM record. The client application queries the service (and possibly other ENUM providers) before placing a call or sending a message. If the peer has keys published, the client can refuse to communicate if the keys don't match, or the peer is not offering call encryption. When public keys are not found, the client can downgrade to traditional unauthenticated encryption, or unencrypted communications.

To be continued In the next posts, I will talk about the overseeing organisation, the SIP service providers, and the client application, and how they all fit together. Stay tuned!

Olivier Berger: Using RDFAlchemy together with RDFLib s SPARQLStore to query DBPedia and process resources in OO way

I ve been searching for interesting ways to manipulate RDF graphs in Python, to create an application that would handle Linked Data Resources in an OO-way, i.e. using Python classes and not tables/sets/lists of triples. The data will be persisted in graphs in a triple store, accessed through a SPARQL enpoint. In this post, I ll illustrate how I managed to tie RDFLib s SPARQLStore plugin and RDFAlchemy to reach a rather nice looking result.
RDFLib provides tools to manipulate graphs, but most of the examples I found didn t load Graph instances from SPARQL, and generally used SPARQLWrapper results (tables) manually. Here comes a first tool to the rescue : SPARQLStore, which allows to dynamically query a remote SPARQL endpoint when navigating an RDFLib graph. I couldn t find a lot of documentation about it, but I found some hints in slide 51 of the excellent presentation by Tatiana Al-Chueyr Linking the world with Python and Semantics. Now, I know how I can manipulate the RDFLib Graphs queried from a remote SPARQL endpoint with the usual methods, I m still not satisfied, because I want OO stuff, not lists of (predicate, object) tuples. Here comes the second tool, RDFAlchemy, which allows to create descriptor classes mapping RDF Resources to Python classes. Here again, there isn t much docs available, but I found the excellent tutorial by Ted Lawless Reading and writing RDF for VIVO with RDFAlchemy. Now we can put the 2 pieces together, modulo a few precautions : the SPARQLStore plugin of RDFLib will generate SPARQL queries everytime a graph is navigated, whereas RDFAlchemy expects the graph to be in memory (at least AFAIU). So we ll have to manually pre-load the contents of the graphs that we need for all the attributes of the descriptor classes. I ve written an example code that illustrates this by trying to query french films in Wikipedia. Here s a copy of the gist. Attention, it will query DBPedia a lot, so pay attention to the bandwidth and memory if you change parameters. It isn t perfect, and I still need to investigate benefits and limitations of the approach. On clear limitation is the number of SPARQL queries made on the endpoint, instead of a smarter pre-loading. Also, on a side note, during tests I could spot a few issues with SPARQLStore, which seems lagging behind probably not used by so many people. The RDFAlchemy project doesn t seem to be in great health, mostly unmaintained from what I can see (and notice that I linked to the GitHub clone/fork that seemed to be the latest maintained while the original author s seems dead), but nevertheless, the code works with a more recent RDFLib, so that s not so bad. Stay tuned for more adventures in Linked Data land in Python.

4 May 2014

Dominique Dumont: Deprecating experience level and preset value in Config::Model

Hello The next releases of Config::Model will deprecate 2 (mis)features: The behavior of cme check will not change. People using cme edit with the GUI will see many more configuration parameters. All the best

1 May 2014

Lisandro Damián Nicanor Pérez Meyer: Call for help from Debian's KDE Team

Hi all!


For quite a while now the KDE team has been severely understaffed. We maintain a lot of packages, with many different kinds of bugs, but we don't have enough people to do all the work that needs to be done. We have tools that help us automate the update to new upstream releases, but that's just the tip of the iceberg of our work and so we are writing to invite more people to get involved in the team and help us get KDE software in Debian into better shape.


Some of the tasks that we need help with are:
  • Bug triaging: there are many many bugs in the BTS. We need people that go through them, understand the problem and how to reproduce it, confirm that they are still present in the latest versions. In particular, there are bugs affecting the version in wheezy, and we need people to go through those as well.
  • Bug forwarding: we are so understaffed that we have been asking users to forward the bugs upstream themselves. Some users do this, but some don't. It would help us a lot to have people in the team in charge of this.
  • Patch forwarding: we have quite a bunch of patches applied in the Debian packages that should be applied upstream. Some need to be generalized instead of being Debian-specific. This work would save us time in the future, so it's very important to get it done.
  • Upgrade-testing: in the past, the upgrade from one Debian stable to the other has been quite traumatic for KDE software users. We need people to try upgrading from wheezy to jessie and report any bugs that they might encounter so that we can fix them ahead of the release.
  • Creating patches: many of the bugs that we have require writing patches, some are easy and some are harder, but any help here would be really appreciated.
  • Packaging other KDE apps: we have packages for the core components of KDE software, but there are many other useful components that still need to get packaged.
  • Updating our welcoming wiki page [1], adding these tasks and any future tasks, and unifying the todo lists [2].
If you are interested in helping with any of these, please join our irc channel #debian-qt-kde in irc.oftc.net, or our mailing list [3]. We are happy to help you get started.


[1]: https://wiki.debian.org/PkgKde
[2]: https://wiki.debian.org/KDETodo
https://wiki.debian.org/KdeDebTasks
http://pkg-kde.alioth.debian.org/todo.html
gobby://gobby.debian.org/Teams/KDE/TODO

[3]: https://lists.debian.org/debian-qt-kde/


--
Regards,
Maximiliano Curia
On behalf of the KDE team

30 April 2014

Russell Coker: Links April 2014

Yves Rossy is the Jetman, he flys with a wing and four jet engines strapped to his body, he gave an interesting TED talk about flying along with some exciting videos [1]. Larry Brilliant gave an informative and inspiring TED talk about stopping pandemics [2]. I thought that Smallpox was the last disease to be eradicated but I was wrong. Michael Shermer gave an interesting TED talk about pattern recognition and self deception [3]. It s a pity that the kissing prank shown at the end only pranked women, they should be less sexist and prank men too. Raffaello D Andrea gave an interesting TED presentation about Athletic quadcopters [4]. It s very impressive and has the potential for several new human/machine sports. Lisa D wrote an insightful article about Prejudice Spillover discussing the way that people who aren t in minority groups only seem to care about injustice when a member of the majority is targetted by mistake [5]. Ron Garret wrote an insightful post about the Divine Right of Billionaires which debunks some stupid arguments by a billionaire [6]. Ron says that it s often instructive to examine incorrect arguments, especially when those arguments are advanced by smart people and demonstrates it in this post. Lisa D wrote an interesting post about her problems with financial aid bureaucracy [7]. She intended the post to be a personal one about her situation, but I think it illustrates problems with the various aid programs. If aid was available to her with less bureaucracy then she would be doing paid work, completing her studies, and heading towards post-graduate studies. Mark Shuttleworth wrote an insightful article about ACPI, security, and device tree [8]. It s the first time I ve seen a good argument for device tree. TED presented an interesting video-conference interview with Edward Snowden [9]. It s unusually long by TED standards but definitely worth watching. Tom Meagher (who s wife was raped and murdered two years ago) wrote an insightful article about rape culture [10]. Key Lay (the Victorian Chief Commissioner of Police) wrote a good article encouraging men to act to stop violence against women [11]. It s particularly noteworthy when a senior police officer speaks out about this given the difficulties women have had in reporting such crimes to police. Emily Baker wrote an insightful article about the lack of support for soldiers who survive war [12]. A lot of attention and money is spent remembering the soldiers who died in the field but little on those who live suffer afterwards, more soldiers die from suicide than enemy fire. Daniel Pocock wrote an informative article about the failings of SMS authentication for online banking [13]. While he has good points I think he s a little extreme. Stopping the least competent attackers is still a significant benefit as most potential attackers aren t that competent. Jess Zimmerman wrote an interesting article for Time about the Not All Men argument that is a current trend in derailing discussions about the treatment of women [14]. The Belle Jar has an insightful article Why Won t You Educate Me About Feminism about some ways that men pretend to care about the treatment of women [15]. Jon Evans wrote an article for Tech Crunch about the Honywell Bubble Count measure of diversity in people you follow on social media [16]. Currently on Twitter I follow 57 accounts of which 15 are companies and organisations, so I follow 42 people. I follow 13 women 31%, for a visible minority group other than my own it s 2/42 or 5%, for people who live in other countries I think it s 8/42 (although it s difficult to determine where some people live) which is 19%. So my Honywell number is 55. The Top Stocks forum has an interesting post by a Coal Seam Gas (CSG) worker [17]. It seems that CSG is even worse than I thought. Ashe Dryden wrote an informative post for Model View Culture about the backlash that members of minority groups (primarily women) receive when they speak out [18].

Mirco Bauer: Smuxi 0.11 "Distractions" Release

And here we go again! We're proud to announce the new version of Smuxi, release 0.11 "Distractions". During the development, 11 bug reports and 2 feature requests in 112 commits were worked on. Notable highlights in this release are:

User Interface Enhancements
  • The chat list can be shrunken. This is especially handy with XMPP/Jabber and long group chat identifiers.
  • The highlight counter is now a separate column. This enhances the vertical alignment with other highlights and guarantees to be visible even if the chat name was truncated.

Multi Identity Support Smuxi cares for user feedback. Multi identity support was the most voted feature and thus it has been implemented! Now you can please your schizo^Wdesire to use different nicks, users and real names depending on the server. Simply edit the server in preferences and change the details.

Message Patterns Everybody knows text can be boring because it is all just text. Nothing can sidetrack you except reading that bare text. Text often has recurring patterns from which something useful and interactive can be created. For example, someone writes:
Hey meebey, do you know RFC2812?
RFCs are a recurring pattern with a distinct number behind it and are real references to something in the internet (collection of protocol specifications). So I would usually fire up a browser tab, copy/paste or type RFC2812 into my favorite search engine and click the first hit. Then I'd reply to the question afterwards. But with Smuxi's message patters, it turns RFC2812 into a link on which you can simply click to launch the relevant document. Wow this is very cool, but isn't this already happening with http URLs and email addresses? Exactly! Why shouldn't more information be used to create useful things from it? Smuxi message patterns allow you to define text patterns that are transformed into clickable links. This can be used for RFCs, CVEs, bug report numbers (#XXX), git commit hashes and much more. Make good use of your creativity! By default Smuxi comes with built-in message patterns for:
  • URLs
  • heuristic URLs (not starting with http:// etc)
  • email addresses
  • RFCs
  • CVEs
  • Debian Security Advisories (DSA)
  • Many popular bug trackers (GNU, GCC, kernel, Launchpad, freedesktop, GNOME, KDE, Xfce, Debian, Redhat, Novell, Xamarin, openSUSE, Mozilla, Samba, SourceForge, CPAN, boost, Claws and Smuxi)
If you know more general patterns useful for others, please submit them. For a full list of built-in message patterns or how to add your own patterns, head over to the message pattern documentation.

Hooks Enhancements
  • A bug was fixed that prevented hooks from issuing more than one command
  • New hook points:
    • engine/session/on-group-chat-person-added
    • engine/session/on-group-chat-person-removed
    • engine/session/on-group-chat-person-updated
  • New hook variables:
    • CMD
    • CMD_PARAMETER
    • CMD_CHARACTER
    • PROTOCOL_MANAGER_PRESENCE_STATUS: Unknown, Offline, Online, Away

Twitter Enhancements As of 14 Jan 2014, Twitter disallows unencrypted HTTP requests which broke Smuxi's Twitter support. Smuxi is now making exclusively encrypted requests (HTTPS) and thus works with Twitter again.

JabbR (Beta) Enhancements
  • Messages now raise Smuxi hooks
  • The Validate certificate setting is now correctly honored.

Updated Translations Smuxi should now be in your language, including:
  • Initial complete Dutch (Jeroen Baten)

Behind the Scenes
  • New Smuxi git repository @ GNOME
  • Cleaner XMPP code (Oliver Schneider)
  • Smuxi's STFL text frontend is doing a graceful shutdown on quit (Calvin B))
  • New sexy website! We hope you like it :)

Contributors Contributors to this release are the following people:
  • Mirco Bauer (98 commits)
  • Oliver Schneider (6 commits)
  • Calvin B (6 commits)
  • Andr s G. Aragoneses (1 commit)
  • Jeroen Baten (translations)
Thank you very much for your contributions to Smuxi! Want it? Go here and grab it right now!

Posted Mon Apr 14 13:23:29 2014

Andrew Pollock: [life] Day 92: Kindergarten, cleaning and more birthday party preparation

I'm mostly back into my normal routine now. Next week should be spot on. Today, after my chiropractic adjustment and another walk to the post office to collect another package, I got stuck into cleaning the house. I got that mostly done before my massage, and knocked out the rest of it before I picked up Zoe from Kindergarten. Zoe seems to have successfully dropped her day time nap at Kindergarten now, which certainly makes a huge difference to getting out of the place on time. I'd underestimated the number of kids coming to Zoe's birthday party, so I needed to get some extra goody bags made up, so we headed directly out to Westfield Carindale to raid the The Reject Shop for some more bits and pieces. After we got home from that, Zoe watched a bit of TV and helped me assemble the extra bags, and then we had dinner. After some epic procrastination (on Zoe's behalf) I managed to get her to bed approximately on time. I really need to try some different approaches to getting through the bath time routine, because it's just taking too long. I gave myself a couple of time outs because I could see the situation deteriorating if I didn't. It's unhelpful that my blog seems to be the third hit in Google at the moment for "positive parent procrastination". I don't have the answers.

Christine Spang: becoming a better speaker

Last week, in the spirit of always getting better, I asked Twitter how to practice saying "um" less while speaking. Here's what I learned. Most people suggested to practice intentionally inserting pauses instead of saying "um". Various strategies include voice acting classes, having a friend aid with a buzzer (presumably to help you notice), and to record practice sessions as well as the real deal. Nagle suggested that meditation helped him, which makes sense to me as I've experienced meditation helping me be more mindful in many different areas of my life, including mundane things like noticing that I've touched something that's probably not that clean and remembering not to touch my face until I wash them. Wilfully directing mindfulness toward noticing how I speak is something I'm really stoked about, though it's easy to get swept away in a default "performance mode" in front of a crowd. Hendrick pointed out that Cognitive Behavioural Therapy is the principle behind changing speech behaviour and is a great starting point for more research. This is going to be really useful next time I'm practicing for a talk! Thanks @zmagg, @hendricklee, @bcrypt, @ebroder, @mscain, and @nagle5000 for playing. :) (original Twitter thread)

29 April 2014

Russell Coker: Autism and the Treatment of Women Again

Background I ve previously written about the claim that people use Autism as an excuse for bad behavior [1]. In summary it doesn t and such claims instead lead to people not being assessed for Autism. I ve also previously written about empathy and Autism in the context of discussions about conference sexual harassment [2]. The main point is that anyone who s going to blame empathy disorders for the widespread mistreatment of women in society and divert the subject from the actions of average men to men in minority groups isn t demonstrating empathy. Discussions of the actions of average men are so often derailed to cover Autism that the Geek Feminism Wiki has a page about the issue of blaming Autism [3]. The Latest Issue Last year Shanley Kane wrote an informative article for Medium titled What Can Men Do about the treatment of women in the IT industry [4]. It s a good article, I recommend reading it. As an aside @shanley s twitter feed is worth reading [5]. In response to Shanley s article Jeff Atwood wrote an article of the same title this year which covered lots of other things [6]. He writes about Autism but doesn t seem to realise that officially Asperger Syndrome is now Autism according to DSM-V (they decided that separate diagnosis of Autism, Asperger Syndrome, and PDD-NOS were too difficult and merged them). Asperger Syndrome is now a term that refers to historic issues (IE research that was published before DSM-V) and slang use. Gender and the Autism Spectrum Jeff claims that autism skews heavily towards males at a 4:1 ratio and cites the Epidemiology of Autism Wikipedia page as a reference. Firstly that page isn t a great reference, I fixed one major error (which was obviously wrong to anyone who knows anything about Autism and also contradicted the cited reference) in the first section while writing this post. The Wikipedia page cites a PDF about the Epidemiology of Autism that claims the 4.3:1 ratio of boys to girls [7]. However that PDF is a summary of other articles and the one which originated the 4.3:1 claim is behind a paywall. One thing that is worth noting in the PDF is that the section containing the 4.3:1 claim also references claims about correlations between race and Autism and studies contradicting such claims it notes the possibility of ascertainment bias . I think that anyone who reads that section should immediately consider the possibility of ascertainment bias in regard to the gender ratio. Most people who are diagnosed with Autism are diagnosed as children. An Autism diagnosis of a child is quite subjective, an important part is an IQ test (where the psychologist interprets the intent of the child in the many cases where answers aren t clear) to compare social skills with IQ. So whether a child is diagnosed is determined by the psychologist s impression of the child s IQ vs the impression of their social skills. Whether a child is even taken for assessment depends on whether they act in a way that s considered to be obviously different. Any child who is suspected of being on the Autism Spectrum will be compared other children who have been diagnosed (IE mostly boys) and this will probably increase the probability that a boy will be assessed. So an Aspie girl might not be assessed because she acts like other Aspie girls not like the Aspie boys her parents and teachers have seen. The way kids act is not solely determined by neuro-type. Our society expects and encourages boys to be louder than girls and take longer and more frequent turns to speak, this is so widespread that I don t think it s possible for parents to avoid it if their kids are exposed to the outside world. Because of this boys who would be diagnosed with Asperger Syndrome by DSM-IV tend to act in ways that are obviously different from other kids. While the combination of Autism and the the social expectations on girls tends to result in girls who are quiet, shy, and apologetic. The fact that girls are less obviously different and that their differences cause fewer difficulties for parents and teachers makes them less likely to be assessed. Note that the differences in behavior of boys and girls who have been diagnosed is noted by the professionals (and was discussed at a conference on AsperGirls that my wife attended) while the idea that this affects assessment rates is my theory. Jeff also cites the book The Essential Difference: Male And Female Brains And The Truth About Autism by Professor Simon Baron-Cohen (who s (in)famous for his Extreme Male Brain theory). The first thing to note about the Extreme Male Brain theory are that it depends almost entirely on the 4.3:1 ratio of males to females on the Autism Spectrum (which is dubious as I noted above). The only other evidence in support of it is subjective studies of children which suffer from the same cultural issues this is why double blind tests should be used whenever possible. The book Delusions of Gender by Cordelia Fine [8] debunks Simon Baron-Cohen s work among other things. The look inside feature of the Amazon page for Delusions of Gender allows you to read about Simon Baron-Cohen s work [9]. Now even if the Extreme Male Brain theory had any merit it would be a really bad idea to cite it (or a book based on it) if you want to make things better for women in the IT industry. Cordelia s book debunks the science and also shows how such claims about supposed essential difference are taken as exclusionary. The Problem with Jeff Atwood Jeff suggests in his post that men should listen to women. Then he and his followers have a huge flame-war with many women over twitter during which which he tweeted Trying to diversify my follows by following any female voices that engaged me in a civil, constructive way recently . If you only listen to women who agree with you then that doesn t really count as listening to women. When you have a stated policy of only listening to women who agree then it seems to be more about limiting what women may feel free to say around you. The Geek Feminism wiki page about the Tone Argument [10] says the following: One way in which the tone argument frequently manifests itself is as a call for civility. A way to gauge whether a request for civility is sincere or not is to ask whether the person asking for civility has more power along whatever axes are contextually relevant (see Intersectionality) than the person being called incivil , less power, or equal power. Often, people who have the privilege of being listened to and taken seriously level accusations of incivility as a silencing tactic, and label as incivil any speech or behavior that questions their privilege. For example, some men label any feminist thought or speech as hostile or impolite; there is no way for anybody to question male power or privilege without being called rude or aggressive. Likewise, some white people label any critical discussion of race, particularly when initiated by people of color, as incivil. Writing about one topic is also a really good idea. A blog post titled What Can Men Do should be about things that men can do. Not about Autism, speculation about supposed inherent differences between men and women which are based on bad research, gender diversity in various occupations, etc. Following up a post on What Can Men Do with discussion (in blog comments and twitter) about what women should do before they are allowed to join the conversation is ridiculous. Jeff s blog post says that men should listen to women, excluding women based on the tone argument is gross hypocrisy. Swearing Jeff makes a big deal of the fact that Shanley uses some profane language in her tweets. This combines a couple of different ways of silencing women. It s quite common for women to be held to a high standard of ladylike behavior, while men get a free pass on doing the same thing. One example of this is the Geek Feminism article about the results of Sarah Sharp s request for civility in the Linux kernel community [11]. That s not an isolated incident, to the best of my recollection in 20+ years my local Linux Users Group has had only one debate about profanity on mailing lists in that case a woman (who is no longer active in the group) was criticised for using lesser profanity than men used both before and after with no comment (as an experiment I used some gratuitous profanity a couple of weeks later and no-one commented). There is also a common difference in interpretation of expressions of emotion, when a woman seems angry then she invariably has men tell her to change her approach (even when there are obvious reasons for her anger) while when a man is angry the possibility that other people shouldn t make him angry will usually be considered. The issues related to the treatment of women have had a large affect on Shanley s life and her friend s lives. It s quite understandable that she is angry about this. Her use of profanity in tweets seems appropriate to the situation. Other Links Newsweek s Gentlemen in Technology article has a section about Jeff [12], it s interesting to note his history of deleting tweets and editing his post. I presume he will change his post in response to mine and not make any note of the differences. Jacob Kaplan-Moss wrote a good rebuttal to Jeff s post [13]. It s a good article and has some other relevant links that are worth reading.

25 April 2014

Daniel Pocock: Extra fuel charges when carrying passengers in your car?

Some people seem to be justifying the way mobile networks try to discriminate against tethering. One comment even suggests that multiple devices is a valid justification. Really? Isn't it just as simple as measuring the actual volume of megabytes used, regardless of how many devices are connected? Lets consider a couple of analogies:
  • A petrol station puts some fine print at the bottom of their sign in the street: cars with passengers pay an extra 0.05 per litre (or gallon). It is just a gimmick: the marketing man has worked out that he can put a lower price on his big sign this way and people with passengers may well be splitting the fuel bill between them. There is no technical reason for the charge however - the car actually uses more fuel carrying the extra weight anyway.
  • Or imagine this: the marketing man puts up small print that advises a higher charge of 0.05 per litre for fuelling an exotic car like a Ferrari. The only reason for this, of course, is that the Ferrari driver is probably busy (so he doesn't have time to shop around) and wealthy (so the extra charge won't deter him from making any purchase at all, because he has money to pay it).
Of course, these ideas sound a bit outlandish but the only difference between these crazy ideas and tethering charges is that mobile phone networks usually have the monopoly/duopoly to get away with it. Mobile networks actually dread the idea of becoming commodities like fuel. They thrive on price discrimination and they seem to have an endless number of tricks to get extra money out of people who may have it by deliberately crippling parts of the service for everybody else. This goes against the ideals of innovation and efficiency and in the most extreme cases their practices are widely recognised for the sham that they are. Revealing that you have connected a laptop to your phone may be like revealing you live in an affluent area or that you have an expensive car. Plenty of businesses try to exploit such knowledge when deciding how much they can charge you or other aspects of how they provide you with a service. This is why privacy is important in every feature of the technology we use. This rogue behaviour by "creative" marketing men also helps to understand just what would happen in the world of technology if it wasn't for the use of free licenses, open standards, accessible encryption and net neutrality.

Gerfried Fuchs: Que[e]rbau

I'm moving. Well, not right here, right now. Rather less than two years. But I already know what my flat will look like and was able to influence that decision. And there will be more to influence, like what to do with common rooms in the builing, or what to put in the garden (voting for climbing facilities for my son of course!). It's this kind of co-housing project where you already know your neighbours beforehand and can find common grounds for decisions like that. The co-housing project I'm moving to is called Que[e]rbau. And it will be living up to its name. It is specificly aimed at people who live tolerance and acceptance, and also potentially live an sorta alternative lifestyle, defining their own identity; but not limited to those. There also will be conventional families living there who specificly don't want to raise their kids in a conservative environment. When I told about this plan to someone they asked me if I really want to do that. Their concerns were with respect to my son and if the house wouldn't become a target. I was puzzled at first, given that we have the Rainbow Parade, the Life Ball and most of all the Rosa Lila Villa since several years in Vienna and I'm not aware of any bigger disturbances it causes, rather the opposite. After thinking a while about it it sounded a bit for the wish of a Don't Ask Don't Tell environment. Recently there was this great documentary done by Vice on youtube about Young and Gay in Putin's Russia (watch all five parts of it, it's worth it). In the light of that I don't think hiding does improve the situation, rather the opposite. Not speaking about it doesn't improve acceptance. And actually, I was approached by at least one person during the Debian Women MiniDebconf about how brave I am considered. I'm not sure if it really is brave, I just don't want to lie to myself anymore, and I very rarely had troubles through that. The more open and natural you behave, the less confrontation area you leave left, and people notice that. Not totally unrelated to that, I created myself a new gpg key. It doesn't carry my official name anymore but just the name I prefer to be addressed with: Rhonda. It also carries a last name you might not have heard yet (it was adopted on the Discworld MUD several years ago, even before I wrote Mermaids; actually in connection with the person who partly triggered the poem), that's the reason I added a plain Rhonda UID to it for those who aren't aware of the last name. I will submit that key to keysigning parties from now on, and it of course is up to you if you feel comfortable with signing it.

/personal permanent link Comments: 0 Flattr this

24 April 2014

Daniel Pocock: Android betrays tethering data

When I upgraded an Android device the other day, I found that tethering completely stopped working. The updated CyanogenMod had inherited a new bug from Android, informing the carrier that I was tethering. The carrier, Vodafone Italy, had decided to make my life miserable by blocking that traffic. I had a closer look and managed to find a workaround. There is absolutely no difference, from a technical perspective, between data transmitted from a mobile device on-screen application and data transmitted from tethering. Revealing the use of tethering to the carrier is a massive breach of privacy - yet comments in the Google bug tracker suggest it is a feature rather than a bug. This little table helps put that logic in perspective:
Product Person who carries handset
User Mobile network who wants to discriminate against some types of network traffic to squeeze more money out of the Product
Feature Revealing private information about the way the Product uses his/her Internet so the real User can profit.
It is also bad news for the environment: many people are being tricked into buying un-needed USB dongle modems that will end up in the rubbish in 1-2 years time when their contract expires and the company pushes them to upgrade to the next best thing. Behind the scenes What does it really mean in practice, how does Android tell your carrier which data is from tethering? As my device is rooted and as it is my device and I will do what I want with it, I decided to have a look inside. The ip command revealed that there are now two network devices, rmmnet_usb0 and rmmnet_usb1. The basic ip route command reveals that traffic from different source addresses is handled differently and split over the different network devices:
shell@android:/ # ip route
0.0.0.0/1 dev tun0  scope link
default via 100.66.150.89 dev rmnet_usb0
83.224.66.138 via 100.87.31.214 dev rmnet_usb1
83.224.70.94 via 100.87.31.214 dev rmnet_usb1
100.66.150.88/30 dev rmnet_usb0  proto kernel  scope link  src 100.66.150.90
100.66.150.89 dev rmnet_usb0  scope link
100.87.31.212/30 dev rmnet_usb1  proto kernel  scope link  src 100.87.31.213
100.87.31.214 dev rmnet_usb1  scope link
128.0.0.0/1 dev tun0  scope link
192.168.42.0/24 dev rndis0  proto kernel  scope link  src 192.168.42.129
I then looked more closely and found that there is also an extra routing table, it can be found with ip rule
shell@android:/ # ip rule show
0:      from all lookup local
32765:  from 192.168.42.0/24 lookup 60
32766:  from all lookup main
32767:  from all lookup default
shell@android:/ # ip route show table 60
default via 100.87.51.57 dev rmnet_usb1
100.87.51.57 dev rmnet_usb1
192.168.42.0/24 dev rndis0  scope link
In this routing table, it is obvious that data from the tethering subnet (192.168.42.0/24) is sent over the extra device rmnet_usb1. Manually cleaning it up If the phone is rooted, it is possible to very quickly change the routing table to get all the tethering traffic going through the normal rmnet_usb0 interface again. It is necessary to get rid of that alternative routing table first:
ip rule del pref 32765
and then update the iptables entries that refer to interface names:
iptables -t nat -I natctrl_nat_POSTROUTING -s 192.168.0.0/16 -o rmnet_usb0 -j MASQUERADE
iptables -I natctrl_FORWARD -i rmmnet_usb0 -j RETURN
This immediately resolved the problem for me on the Vodafone network in Italy. Conclusion If Google can be bullied into accepting this kind of discriminatory routing in the stock Android builds and it can even propagate into CyanogenMod, then I'm just glad I'm not running one of those Android builds that has been explicitly "enhanced" by a carrier. It raises big questions about who really is the owner and user of the device and who is receiving the value when a person pays money to "buy" a device.

22 April 2014

Erich Schubert: Kernel-density based outlier detection and the need for customization

Outlier detection (also: anomaly detection, change detection) is an unsupervised data mining task that tries to identify the unexpected.
Most outlier detection methods are based on some notion of density: in an appropriate data representation, "normal" data is expected to cluster, and outliers are expected to be further away from the normal data.
This intuition can be quantified in different ways. Common heuristics include kNN outlier detection and the Local Outlier Factor (which uses a density quotient). One of the directions in my dissertation was to understand (also from a statistical point of view) how the output and the formal structure of these methods can be best understood.
I will present two smaller results of this analysis at the SIAM Data Mining 2014 conference: instead of the very heuristic density estimation found in above methods, we design a method (using the same generalized pattern) that uses a best-practise from statistics: Kernel Density Estimation. We aren't the first to attempt this (c.f. LDF), but we actuall retain the properties of the kernel, whereas the authors of LDF tried to mimic the LOF method too closely, and this way damaged the kernel.
The other result presented in this work is the need to customize. When working with real data, using "library algorithms" will more often than not fail. The reason is that real data isn't as nicely behaved - it's dirty, it seldom is normal distributed. And the problem that we're trying to solve is often much narrower. For best results, we need to integrate our preexisting knowledge of the data into the algorithm. Sometimes we can do so by preprocessing and feature transformation. But sometimes, we can also customize the algorithm easily.
Outlier detection algorithms aren't black magic, or carefully adjusted. They follow a rather simple logic, and this means that we can easily take only parts of these methods, and adjust them as necessary for our problem at hand!
The article persented at SDM will demonstrate such a use case: analyzing 1.2 million traffic accidents in the UK (from data.gov.uk) we are not interested in "classic" density based outliers - this would be a rare traffic accident on a small road somewhere in Scotland. Instead, we're interested in unusual concentrations of traffic accidents, i.e. blackspots.
The generalized pattern can be easily customized for this task. While this data does not allow automatic evaluation, many outliers could be easily verified using Google Earth and search: often, historic imagery on Google Earth showed that the road layout was changed, or that there are many news reports about the dangerous road. The data can also be nicely visualized, and I'd like to share these examples with you. First, here is a screenshot from Google Earth for one of the hotspots (Cherry Lane Roundabout, North of Heathrow airport, which used to be a double cut-through roundabout - one of the cut-throughs was removed since):
Screenshot of Cherry Lane Roundabout hotspot
Google Earth is best for exploring this result, because you can hide and show the density overlay to see the crossroad below; and you can go back in time to access historic imagery. Unfortunately, KML does not allow easy interactions (at least it didn't last time I checked).
I have also put the KML file on Google Drive. It will automatically display it on Google Maps (nice feature of Drive, kudos to Google!), but it should also allow you to download it. I've also explored the data on an Android tablet (but I don't think you can hide elements there, or access historic imagery as in the desktop application).
With a classic outlier detection method, this analysis would not have been possible. However, it was easy to customize the method; and the results are actually more meaningful: instead of relying on some heuristic to choose kernel bandwidth, I opted for choosing the bandwidth by physical arguments: 50 meters is a reasonable bandwidth for a crossroad / roundabout, and for comparison a radius of 2 kilometers is used to model the typical accident density in this region (there should other crossroads within 2 km in Europe).
Since I advocate reproducible science, the source code of the basic method will be in the next ELKI release. For the customization case studies, I plan to share them as a how-to or tutorial type of document in the ELKI wiki; probably also detailing data preprocessing and visualization aspects. The code for the customizations is not really suited for direct inclusion in the ELKI framework, but can serve as an example for advanced usage.
Reference:
E. Schubert, A. Zimek, H.-P. Kriegel
Generalized Outlier Detection with Flexible Kernel Density Estimates
In Proceedings of the 14th SIAM International Conference on Data Mining (SDM), Philadelphia, PA, 2014.
So TLDR of the story: A) try to use more established statistics (such as KDE), and B) don't expect an off-the-shelf solution to do magic, but customize the method for your problem.
P.S. if you happen to know nice post-doc positions in academia:
I'm actively looking for a position to continue my research. I'm working on scaling these methods to larger data and to make them work with various real data that I can find. Open-source, modular and efficient implementations are very important to me, and one of the directions I'd like to investigate is porting these methods to a distributed setting, for example using Spark. In order to get closer to "real" data, I've started to make these approaches work e.g. on textual data, mixed type data, multimedia etc. And of course, I like teaching; which is why I would prefer a position in academia.

19 April 2014

Russell Coker: Swap Space and SSD

In 2007 I wrote a blog post about swap space [1]. The main point of that article was to debunk the claim that Linux needs a swap space twice as large as main memory (in summary such advice is based on BSD Unix systems and has never applied to Linux and that most storage devices aren t fast enough for large swap). That post was picked up by Barrapunto (Spanish Slashdot) and became one of the most popular posts I ve written [2]. In the past 7 years things have changed. Back then 2G of RAM was still a reasonable amount and 4G was a lot for a desktop system or laptop. Now there are even phones with 3G of RAM, 4G is about the minimum for any new desktop or laptop, and desktop/laptop systems with 16G aren t that uncommon. Another significant development is the use of SSDs which dramatically improve speed for some operations (mainly seeks). As SATA SSDs for desktop use start at about $110 I think it s safe to assume that everyone who wants a fast desktop system has one. As a major limiting factor in swap use is the seek performance of the storage the use of SSDs should allow greater swap use. My main desktop system has 4G of RAM (it s an older Intel 64bit system and doesn t support more) and has 4G of swap space on an Intel SSD. My work flow involves having dozens of Chromium tabs open at the same time, usually performance starts to drop when I get to about 3.5G of swap in use. While SSD generally has excellent random IO performance the contiguous IO performance often isn t much better than hard drives. My Intel SSDSC2CT12 300i 128G can do over 5000 random seeks per second but for sustained contiguous filesystem IO can only do 225M/s for writes and 274M/s for reads. The contiguous IO performance is less than twice as good as a cheap 3TB SATA disk. It also seems that the performance of SSDs aren t as consistent as that of hard drives, when a hard drive delivers a certain level of performance then it can generally do so 24*7 but a SSD will sometimes reduce performance to move blocks around (the erase block size is usually a lot larger than the filesystem block size). It s obvious that SSDs allow significantly better swap performance and therefore make it viable to run a system with more swap in use but that doesn t allow unlimited swap. Even when using programs like Chromium (which seems to allocate huge amounts of RAM that aren t used much) it doesn t seem viable to have swap be much bigger than 4G on a system with 4G of RAM. Now I could buy another SSD and use two swap spaces for double the overall throughput (which would still be cheaper than buying a PC that supports 8G of RAM), but that still wouldn t solve all problems. One issue I have been having on occasion is BTRFS failing to allocate kernel memory when managing snapshots. I m not sure if this would be solved by adding more RAM as it could be an issue of RAM fragmentation I won t file a bug report about this until some of the other BTRFS bugs are fixed. Another problem I have had is when running Minecraft the driver for my ATI video card fails to allocate contiguous kernel memory, this is one that almost certainly wouldn t be solved by just adding more swap but might be solved if I tweaked the kernel to be more aggressive about swapping out data. In 2007 when using hard drives for swap I found that the maximum space that could be used with reasonable performance for typical desktop operations was something less than 2G. Now with a SSD the limit for usable swap seems to be something like 4G on a system with 4G of RAM. On a system with only 2G of RAM that might allow the system to be usable with swap being twice as large as RAM, but with the amounts of RAM in modern PCs it seems that even SSD doesn t allow using a swap space larger than RAM for typical use unless it s being used for hibernation. Conclusion It seems that nothing has significantly changed in the last 7 years. We have more RAM, faster storage, and applications that are more memory hungry. The end result is that swap still isn t very usable for anything other than hibernation if it s larger than RAM. It would be nice if application developers could stop increasing the use of RAM. Currently it seems that the RAM requirements for Linux desktop use are about 3 years behind the RAM requirements for Windows. This is convenient as a PC is fully depreciated according to the tax office after 3 years. This makes it easy to get 3 year old PCs cheaply (or sometimes for free as rubbish) which work really well for Linux. But it would be nice if we could be 4 or 5 years behind Windows in terms of hardware requirements to reduce the hardware requirements for Linux users even further.

Paul Tagliamonte: Hy at PyCon 2014

I gave a talk this year at PyCon 2014, about one of my favorite subjects: Hy. Many of my regular readers will have no doubt explored Hy's thriving GitHub org, played with try-hy, or even installed it locally by pip installing it. I was lucky enough to be able to attend PyCon on behalf of Sunlight, with a solid contingint of my colleagues. We put together a writeup on the Sunlight blog if anyone was interested in our favorite talks.
Tons of really amazing questions, and such an amazingly warm reception from so many of my peers throughout this year's PyCon. Thank you so much to everyone that attended the talk. As always, you should Fork Hy on GitHub, follow @hylang on the twitters, and send in any bugs you find! Hopefully I'll be able to put my talk up in blog-post form soon, but until then feel free to look over the slides or just watch the talk. An extra shout-out to @akaptur for hacking on Hy during the sprints, and giving the exception system quite the workthrough. Thanks, Allison!

Next.

Previous.