Search Results: "barry"

14 December 2016

Antoine Beaupr : Django debates privacy concern

In recent years, privacy issues have become a growing concern among free-software projects and users. As more and more software tasks become web-based, surveillance and tracking of users is also on the rise. While some software may use advertising as a source of revenue, which has the side effect of monitoring users, the Django community recently got into an interesting debate surrounding a proposal to add user tracking actually developer tracking to the popular Python web framework.

Tracking for funding A novel aspect of this debate is that the initiative comes from concerns of the Django Software Foundation (DSF) about funding. The proposal suggests that "relying on the free labor of volunteers is ineffective, unfair, and risky" and states that "the future of Django depends on our ability to fund its development". In fact, the DSF recently hired an engineer to help oversee Django's development, which has been quite successful in helping the project make timely releases with fewer bugs. Various fundraising efforts have resulted in major new Django features, but it is difficult to attract sponsors without some hard data on the usage of Django. The proposed feature tries to count the number of "unique developers" and gather some metrics of their environments by using Google Analytics (GA) in Django. The actual proposal (DEP 8) is done as a pull request, which is part of Django Enhancement Proposal (DEP) process that is similar in spirit to the Python Enhancement Proposal (PEP) process. DEP 8 was brought forward by a longtime Django developer, Jacob Kaplan-Moss. The rationale is that "if we had clear data on the extent of Django's usage, it would be much easier to approach organizations for funding". The proposal is essentially about adding code in Django to send a certain set of metrics when "developer" commands are run. The system would be "opt-out", enabled by default unless turned off, although the developer would be warned the first time the phone-home system is used. The proposal notes that an opt-in system "severely undercounts" and is therefore not considered "substantially better than a community survey" that the DSF is already doing.

Information gathered The pieces of information reported are specifically designed to run only in a developer's environment and not in production. The metrics identified are, at the time of writing:
  • an event category (the developer commands: startproject, startapp, runserver)
  • the HTTP User-Agent string identifying the Django, Python, and OS versions
  • a user-specific unique identifier (a UUID generated on first run)
The proposal mentions the use of the GA aip flag which, according to GA documentation, makes "the IP address of the sender 'anonymized'". It is not quite clear how that is done at Google and, given that it is a proprietary platform, there is no way to verify that claim. The proposal says it means that "we can't see, and Google Analytics doesn't store, your actual IP". But that is not actually what Google does: GA stores IP addresses, the documentation just says they are anonymized, without explaining how. GA is presented as a trade-off, since "Google's track record indicates that they don't value privacy nearly as high" as the DSF does. The alternative, deploying its own analytics software, was presented as making sustainability problems worse. According to the proposal, Google "can't track Django users. [...] The only thing Google could do would be to lie about anonymizing IP addresses, and attempt to match users based on their IPs". The truth is that we don't actually know what Google means when it "anonymizes" data: Jannis Leidel, a Django team member, commented that "Google has previously been subjected to secret US court orders and was required to collaborate in mass surveillance conducted by US intelligence services" that limit even Google's capacity of ensuring its users' anonymity. Leidel also argued that the legal framework of the US may not apply elsewhere in the world: "for example the strict German (and by extension EU) privacy laws would exclude the automatic opt-in as a lawful option". Furthermore, the proposal claims that "if we discovered Google was lying about this, we'd obviously stop using them immediately", but it is unclear exactly how this could be implemented if the software was already deployed. There are also concerns that an implementation could block normal operation, especially in countries (like China) where Google itself may be blocked. Finally, some expressed concerns that the information could constitute a security problem, since it would unduly expose the version number of Django that is running.

In other projects Django is certainly not the first project to consider implementing analytics to get more information about its users. The proposal is largely inspired by a similar system implemented by the OS X Homebrew package manager, which has its own opt-out analytics. Other projects embed GA code directly in their web pages. This is apparently the option chosen by the Oscar Django-based ecommerce solution, but that was seen by the DSF as less useful since it would count Django administrators and wasn't seen as useful as counting developers. Wagtail, a Django-based content-management system, was incorrectly identified as using GA directly, as well. It actually uses referrer information to identify installed domains through the version updates checks, with opt-out. Wagtail didn't use GA because the project wanted only minimal data and it was worried about users' reactions. NPM, the JavaScript package manager, also considered similar tracking extensions. Laurie Voss, the co-founder of NPM, said it decided to completely avoid phoning home, because "users would absolutely hate it". But NPM users are constantly downloading packages to rebuild applications from scratch, so it has more complete usage metrics, which are aggregated and available via a public API. NPM users seem to find this is a "reasonable utility/privacy trade". Some NPM packages do phone home and have seen "very mixed" feedback from users, Voss said. Eric Holscher, co-founder of Read the Docs, said the project is considering using Sentry for centralized reporting, which is a different idea, but interesting considering Sentry is fully open source. So even though it is a commercial service (as opposed to the closed-source Google Analytics), it may be possible to verify any anonymity claims.

Debian's response Since Django is shipped with Debian, one concern was the reaction of the distribution to the change. Indeed, "major distros' positions would be very important for public reception" to the feature, another developer stated. One of the current maintainers of Django in Debian, Rapha l Hertzog, explicitly stated from the start that such a system would "likely be disabled by default in Debian". There were two short discussions on Debian mailing lists where the overall consensus seemed to be that any opt-out tracking code was undesirable in Debian, especially if it was aimed at Google servers. I have done some research to see what, exactly, was acceptable as a phone-home system in the Debian community. My research has revealed ten distinct bug reports against packages that would unexpectedly connect to the network, most of which were not directly about collecting statistics but more often about checking for new versions. In most cases I found, the feature was disabled. In the case of version checks, it seems right for Debian to disable the feature, because the package cannot upgrade itself: that task is delegated to the package manager. One of those issues was the infamous "OK Google" voice activation binary blog controversy that was previously reported here and has since then been fixed (although other issues remain in Chromium). I have also found out that there is no clearly defined policy in Debian regarding tracking software. What I have found, however, is that there seems to be a strong consensus in Debian that any tracking is unacceptable. This is, for example, an extract of a policy that was drafted (but never formally adopted) by Ian Jackson, a longtime Debian developer:
Software in Debian should not communicate over the network except: in order to, and as necessary to, perform their function[...]; or for other purposes with explicit permission from the user.
In other words, opt-in only, period. Jackson explained that "when we originally wrote the core of the policy documents, the DFSG [Debian Free Software Guidelines], the SC [Social Contract], and so on, no-one would have considered this behaviour acceptable", which explains why no explicit formal policy has been adopted yet in the Debian project. One of the concerns with opt-out systems (or even prompts that default to opt-in) was well explained back then by Debian developer Bas Wijnen:
It very much resembles having to click through a license for every package you install. One of the nice things about Debian is that the user doesn't need to worry about such things: Debian makes sure things are fine.
One could argue that Debian has its own tracking systems. For example, by default, Debian will "phone home" through the APT update system (though it only reports the packages requested). However, this is currently not automated by default, although there are plans to do so soon. Furthermore, Debian members do not consider APT as tracking, because it needs to connect to the network to accomplish its primary function. Since there are multiple distributed mirrors (which the user gets to choose when installing), the risk of surveillance and tracking is also greatly reduced. A better parallel could be drawn with Debian's popcon system, which actually tracks Debian installations, including package lists. But as Barry Warsaw pointed out in that discussion, "popcon is 'opt-in' and [...] the overwhelming majority in Debian is in favour of it in contrast to 'opt-out'". It should be noted that popcon, while opt-in, defaults to "yes" if users click through the install process. [Update: As pointed out in the comments, popcon actually defaults to "no" in Debian.] There are around 200,000 submissions at this time, which are tracked with machine-specific unique identifiers that are submitted daily. Ubuntu, which also uses the popcon software, gets around 2.8 million daily submissions, while Canonical estimates there are 40 million desktop users of Ubuntu. This would mean there is about an order of magnitude more installations than what is reported by popcon. Policy aside, Warsaw explained that "Debian has a reputation for taking privacy issues very serious and likes to keep it".

Next steps There are obviously disagreements within the Django project about how to handle this problem. It looks like the phone-home system may end up being implemented as a proxy system "which would allow us to strip IP addresses instead of relying on Google to anonymize them, or to anonymize them ourselves", another Django developer, Aymeric Augustin, said. Augustin also stated that the feature wouldn't "land before Django drops support for Python 2", which is currently estimated to be around 2020. It is unclear, then, how the proposal would resolve the funding issues, considering how long it would take to deploy the change and then collect the information so that it can be used to spur the funding efforts. It also seems the system may explicitly prompt the user, with an opt-out default, instead of just splashing a warning or privacy agreement without a prompt. As Shai Berger, another Django contributor, stated, "you do not get [those] kind of numbers in community surveys". Berger also made the argument that "we trust the community to give back without being forced to do so"; furthermore:
I don't believe the increase we might get in the number of reports by making it harder to opt-out, can be worth the ill-will generated for people who might feel the reporting was "sneaked" upon them, or even those who feel they were nagged into participation rather than choosing to participate.
Other options may also include gathering metrics in pip or PyPI, which was proposed by Donald Stufft. Leidel also proposed that the system could ask to opt-in only after a few times the commands are called. It is encouraging to see that a community can discuss such issues without heating up too much and shows great maturity for the Django project. Every free-software project may be confronted with funding and sustainability issues. Django seems to be trying to address this in a transparent way. The project is willing to engage with the whole spectrum of the community, from the top leaders to downstream distributors, including individual developers. This practice should serve as a model, if not of how to do funding or tracking, at least of how to discuss those issues productively. Everyone seems to agree the point is not to surveil users, but improve the software. As Lars Wirzenius, a Debian developer, commented: "it's a very sad situation if free software projects have to compromise on privacy to get funded". Hopefully, Django will be able to improve its funding without compromising its principles.
Note: this article first appeared in the Linux Weekly News.

6 October 2016

Reproducible builds folks: Reproducible Builds: week 75 in Stretch cycle

What happened in the Reproducible Builds effort between Sunday September 25 and Saturday October 1 2016: Statistics For the first time, we reached 91% reproducible packages in our testing framework on testing/amd64 using a determistic build path. (This is what we recommend to make packages in Stretch reproducible.) For unstable/amd64, where we additionally test for reproducibility across different build paths we are at almost 76% again. IRC meetings We have a poll to set a time for a new regular IRC meeting. If you would like to attend, please input your available times and we will try to accommodate for you. There was a trial IRC meeting on Friday, 2016-09-31 1800 UTC. Unfortunately, we did not activate meetbot. Despite this participants consider the meeting a success as several topics where discussed (eg changes to IRC notifications of tests.r-b.o) and the meeting stayed within one our length. Upcoming events Reproduce and Verify Filesystems - Vincent Batts, Red Hat - Berlin (Germany), 5th October, 14:30 - 15:20 @ LinuxCon + ContainerCon Europe 2016. From Reproducible Debian builds to Reproducible OpenWrt, LEDE & coreboot - Holger "h01ger" Levsen and Alexander "lynxis" Couzens - Berlin (Germany), 13th October, 11:00 - 11:25 @ OpenWrt Summit 2016. Introduction to Reproducible Builds - Vagrant Cascadian will be presenting at the SeaGL.org Conference In Seattle (USA), November 11th-12th, 2016. Previous events GHC Determinism - Bartosz Nitka, Facebook - Nara (Japan), 24th September, ICPF 2016. Toolchain development and fixes Michael Meskes uploaded bsdmainutils/9.0.11 to unstable with a fix for #830259 based on Reiner Herrmann's patch. This fixed locale_dependent_symbol_order_by_lorder issue in the affected packages (freebsd-libs, mmh). devscripts/2.16.8 was uploaded to unstable. It includes a debrepro script by Antonio Terceiro which is similar in purpose to reprotest but more lightweight; specific to Debian packages and without support for virtual servers or configurable variations. Packages reviewed and fixed, and bugs filed The following updated packages have become reproducible in our testing framework after being fixed: The following updated packages appear to be reproducible now for reasons we were not able to figure out. (Relevant changelogs did not mention reproducible builds.) Some uploads have addressed some reproducibility issues, but not all of them: Patches submitted that have not made their way to the archive yet: Reviews of unreproducible packages 77 package reviews have been added, 178 have been updated and 80 have been removed in this week, adding to our knowledge about identified issues. 6 issue types have been updated: Weekly QA work As part of reproducibility testing, FTBFS bugs have been detected and reported by: diffoscope development A new version of diffoscope 61 was uploaded to unstable by Chris Lamb. It included contributions from: Post-release there were further contributions from: reprotest development A new version of reprotest 0.3.2 was uploaded to unstable by Ximin Luo. It included contributions from: Post-release there were further contributions from: tests.reproducible-builds.org Misc. This week's edition was written by Ximin Luo, Holger Levsen & Chris Lamb and reviewed by a bunch of Reproducible Builds folks on IRC.

31 October 2015

Chris Lamb: Free software activities in October 2015

Here is my monthly update covering a large part of what I have been doing in the free software world (previously):
Debian My work in the Reproducible Builds project was also covered in more depth in Lunar's weekly reports (#23, #24, #25, #26).
LTS

This month I have been paid to work 11 hours on Debian Long Term Support (LTS). In that time I did the following:
  • DLA 326-1 for zendframework fixing an SQL injection vulnerability.
  • DLA 332-1 for optipng correcting a use-after-free issue.
  • DLA 333-1 for cakephp preventing a remote Denial of Service attack.
  • DLA 337-1 for busybox fixing a vulnerability when unzipping a specially crafted zip file/
  • DLA 338-1 for xscreensaver preventing a crash when hot-swapping monitors.

Uploads
  • redis New upstream release as well as changing the default UNIX socket location and correctly supporting "cluster" mode config file hardening and redis-sentinel's runtime directory handling under systemd. An update for jessie was also uploaded.
  • python-redis Attempting to get the autopkgtest tests to finally pass.
  • debian-timeline Making the build reproducible.
  • gunicorn New upstream release.


4 October 2015

Lunar: Reproducible builds: week 23 in Stretch cycle

What happened in the reproducible builds effort this week: Toolchain fixes Andreas Metzler uploaded autogen/1:5.18.6-1 in experimental with several patches for reproducibility issues written by Valentin Lorentz. Groovy upstream has merged a change proposed by Emmanuel Bourg to remove timestamps generated by groovydoc. Ben Hutchings submitted a patch to add support for SOURCE_DATE_EPOCH in linux-kbuild as an alternate way to specify the build timestamp. Reiner Herrman has sent a patch adding support for SOURCE_DATE_EPOCH in docbook-utils. Packages fixed The following packages became reproducible due to changes in their build dependencies: commons-csv. fest-reflect, sunxi-tools, xfce4-terminal, The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues but not all of them: Patches submitted which have not made their way to the archive yet: Tomasz Rybak uploaded pycuda/2015.1.3-1 which should fix reproducibility issues. The package has not been tested as it is in contrib. akira found an embedded code copy of texi2html in fftw. reproducible.debian.net Email notifications are now only sent once a day per package, instead of on each status change. (h01ger) disorderfs has been temporarily disabled to see if it had any impact on the disk space issues. (h01ger) When running out of disk space, build nodes will now automatically detect the problem. This means test results will not be recorded as FTBFS and the problem will be reported to Jenkins maintainers. (h01ger) The navigation menu of package pages has been improved. (h01ger) The two amd64 builders now use two different kernel versions: 3.16 from stable and 4.1 from backports on the other. (h01ger) We now graph the number of packages which needs to be fixed. (h01ger) Munin now creates graphs on how many builds were performed by build nodes (example). (h01ger) A migration plan has been agreed with DSA on how to turn Jenkins into an official Debian service. A backport of jenkins-job-builder for Jessie is currently missing. (h01ger) Package reviews 119 reviews have been removed, 103 added and 45 updated this week. 16 fail to build from source issues were reported by Chris Lamb and Mattia Rizzolo. New issue this week: timestamps_in_manpages_generated_by_docbook_utils. Misc. Allan McRae has submitted a patch to make ArchLinux pacman record a .BUILDINFO file.

17 September 2015

Julien Danjou: My interview in le Journal du Hacker

A few days ago, the French equivalent of Hacker News, called "Le Journal du Hacker", interviewed me about my work on OpenStack, my job at Red Hat and my self-published book The Hacker's Guide to Python. I've spent some time translating it into English so you can read it if you don't understand French! I hope you'll enjoy it.
Hi Julien, and thanks for participating in this interview for the Journal du Hacker. For our readers who don't know you, can you introduce you briefly?
You're welcome! My name is Julien, I'm 31 years old, and I live in Paris. I now have been developing free software for around fifteen years. I had the pleasure to work (among other things) on Debian, Emacs and awesome these last years, and more recently on OpenStack. Since a few months now, I work at Red Hat, as a Principal Software Engineer on OpenStack. I am in charge of doing upstream development for that cloud-computing platform, mainly around the Ceilometer, Aodh and Gnocchi projects.
Being myself a system architect, I follow your work in OpenStack since a while. It's uncommon to have the point of view of someone as implied as you are. Can you give us a summary of the state of the project, and then detail your activities in this project?
The OpenStack project has grown and changed a lot since I started 4 years ago. It started as a few projects providing the basics, like Nova (compute), Swift (object storage), Cinder (volume), Keystone (identity) or even Neutron (network) who are basis for a cloud-computing platform, and finally became composed of a lot more projects. For a while, the inclusion of projects was the subject of a strict review from the technical committee. But since a few months, the rules have been relaxed, and we see a lot more projects connected to cloud-computing joining us. As far as I'm concerned, I've started with a few others people the Ceilometer project in 2012, devoted to handling metrics of OpenStack platforms. Our goal is to be able to collect all the metrics and record them to analyze them later. We also have a module providing the ability to trigger actions on threshold crossing (alarm). The project grew in a monolithic way, and in a linear way for the number of contributors, during the first two years. I was the PTL (Project Technical Leader) for a year. This leader position asks for a lot of time for bureaucratic things and people management, so I decided to leave my spot in order to be able to spend more time solving the technical challenges that Ceilometer offered. I've started the Gnocchi project in 2014. The first stable version (1.0.0) was released a few months ago. It's a timeseries database offering a REST API and a strong ability to scale. It was a necessary development to solve the problems tied to the large amount of metrics created by a cloud-computing platform, where tens of thousands of virtual machines have to be metered as often as possible. This project works as a standalone deployment or with the rest of OpenStack. More recently, I've started Aodh, the result of moving out the code and features of Ceilometer related to threshold action triggering (alarming). That's the logical suite to what we started with Gnocchi. It means Ceilometer is to be split into independent modules that can work together with or without OpenStack. It seems to me that the features provided by Ceilometer, Aodh and Gnocchi can also be interesting for operators running more classical infrastructures. That's why I've pushed the projects into that direction, and also to have a more service-oriented architecture (SOA)
I'd like to stop for a moment on Ceilometer. I think that this solution was very expected, especially by the cloud-computing providers using OpenStack for billing resources sold to their customers. I remember reading a blog post where you were talking about the high-speed construction of this brick, and features that were not supposed to be there. Nowadays, with Gnocchi and Aodh, what is the quality of the brick Ceilometer and the programs it relies on?
Indeed, one of the first use-case for Ceilometer was tied to the ability to get metrics to feed a billing tool. That's now a reached goal since we have billing tools for OpenStack using Ceilometer, such as CloudKitty. However, other use-cases appeared rapidly, such as the ability to trigger alarms. This feature was necessary, for example, to implement the auto-scaling feature that Heat needed. At the time, for technical and political reasons, it was not possible to implement this feature in a new project, and the functionality ended up in Ceilometer, since it was using the metrics collected and stored by Ceilometer itself. Though, like I said, this feature is now in its own project, Aodh. The alarm feature is used since a few cycles in production, and the Aodh project brings new features on the table. It allows to trigger threshold actions and is one of the few solutions able to work at high scale with several thousands of alarms. It's impossible to make Nagios run with millions of instances to fetch metrics and triggers alarms. Ceilometer and Aodh can do that easily on a few tens of nodes automatically. On the other side, Ceilometer has been for a long time painted as slow and complicated to use, because its metrics storage system was by default using MongoDB. Clearly, the data structure model picked was not optimal for what the users were doing with the data. That's why I started Gnocchi last year, which is perfectly designed for this use case. It allows linear access time to metrics (O(1) complexity) and fast access time to the resources data via an index. Today, with 3 projects having their own perimeter of features defined and which can work together Ceilometer, Aodh and Gnocchi finally erased the biggest problems and defects of the initial project.
To end with OpenStack, one last question. You're a Python developer for a long time and a fervent user of software testing and test-driven development. Several of your blogs posts point how important their usage are. Can you tell us more about the usage of tests in OpenStack, and the test prerequisites to contribute to OpenStack?
I don't know any project that is as tested on every layer as OpenStack is. At the start of the project, there was a vague test coverage, made of a few unit tests. For each release, a bunch of new features were provided, and you had to keep your fingers crossed to have them working. That's already almost unacceptable. But the big issue was that there was also a lot of regressions, et things that were working were not anymore. It was often corner cases that developers forgot about that stopped working. Then the project decided to change its policy and started to refuse all patches new features or bug fix that would not implement a minimal set of unit tests, proving the patch would work. Quickly, regressions were history, and the number of bugs largely reduced months after months. Then came the functional tests, with the Tempest project, which runs a test battery on a complete OpenStack deployment. OpenStack now possesses a complete test infrastructure, with operators hired full-time to maintain them. The developers have to write the test, and the operators maintain an architecture based on Gerrit, Zuul, and Jenkins, which runs the test battery of each project for each patch sent. Indeed, for each version of a patch sent, a full OpenStack is deployed into a virtual machine, and a battery of thousands of unit and functional tests is run to check that no regressions are possible. To contribute to OpenStack, you need to know how to write a unit test the policy on functional tests is laxer. The tools used are standard Python tools, unittest for the framework and tox to run a virtual environment (venv) and run them. It's also possible to use DevStack to deploy an OpenStack platform on a virtual machine and run functional tests. However, since the project infrastructure also do that when a patch is submitted, it's not mandatory to do that yourself locally.
The tools and tests you write for OpenStack are written in Python, a language which is very popular today. You seem to like it more than you have to, since you wrote a book about it, The Hacker's Guide to Python, that I really enjoyed. Can you explain what brought you to Python, the main strong points you attribute to this language (quickly) and how you went from developer to author?
I stumbled upon Python by chance, around 2005. I don't remember how I hear about it, but I bought a first book to discover it and started toying with that language. At that time, I didn't find any project to contribute to or to start. My first project with Python was rebuildd for Debian in 2007, a bit later. I like Python for its simplicity, its object orientation rather clean, its easiness to be deployed and its rich open source ecosystem. Once you get the basics, it's very easy to evolve and to use it for anything, because the ecosystem makes it easy to find libraries to solve any kind of problem. I became an author by chance, writing blog posts from time to time about Python. I finally realized that after a few years studying Python internals (CPython), I learned a lot of things. While writing a post about the differences between method types in Python which is still one of the most read post on my blog I realized that a lot of things that seemed obvious to me where not for other developers. I wrote that initial post after thousands of hours spent doing code reviews on OpenStack. I, therefore, decided to note all the developers pain points and to write a book about that. A compilation of what years of experience taught me and taught to the other developers I decided to interview in the book.
I've been very interested by the publication of your book, for the subject itself, but also the process you chose. You self-published the book, which seems very relevant nowadays. Is that a choice from the start? Did you look for an editor? Can you tell use more about that?
I've been lucky to find out about others self-published authors, such as Nathan Barry who even wrote a book on that subject, called Authority. That's what convinced me it was possible and gave me hints for that project. I've started to write in August 2013, and I ran the firs interviews with other developers at that time. I started to write the table of contents and then filled the pages with what I knew and what I wanted to share. I manage to finish the book around January 2014. The proof-reading took more time than I expected, so the book was only released in March 2014. I wrote a complete report on that on my blog, where I explain the full process in detail, from writing to launching. I did not look for editors though I've been proposed some. The idea of self-publishing really convince me, so I decided to go on my own, and I have no regret. It's true that you have to wear two hats at the same time and handle a lot more things, but with a minimal audience and some help from the Internet, anything's possible! I've been reached by two editors since then, a Chinese and Korean one. I gave them rights to translate and publish the books in their countries, so you can buy the Chinese and Korean version of the first edition of the book out there. Seeing how successful it was, I decided to launch a second edition in Mai 2015, and it's likely that a third edition will be released in 2016.
Nowadays, you work for Red Hat, a company that represents the success of using Free Software as a commercial business model. This company fascinates a lot in our community. What can you say about your employer from your point of view?
It only has been a year since I joined Red Hat (when they bought eNovance), so my experience is quite recent. Though, Red Hat is really a special company on every level. It's hard to see from the outside how open it is, and how it works. It's really close to and it really looks like an open source project. For more details, you should read The Open Organization, a book wrote by Jim Whitehurst (CEO of Red Hat), which he just published. It describes perfectly how Red Hat works. To summarize, meritocracy and the lack of organization in silos is what makes Red Hat a strong organization and puts them as one of the most innovative company. In the end, I'm lucky enough to be autonomous for the project I work on with my team around OpenStack, and I can spend 100% working upstream and enhance the Python ecosystem.

15 June 2015

Lunar: Reproducible builds: week 7 in Stretch cycle

What happened about the reproducible builds effort for this week: Presentations On June 7th, Reiner Herrmann presented the project at the Gulaschprogrammiernacht 15 in Karlsruhe, Germany. Video and audio recordings in German are available, and so are the slides in English. Toolchain fixes Daniel Kahn Gillmor's report on help2man started a discussion with Brendan O'Dea and Ximin Luo about standardizing a common environment variable that would provide a replacement for an embedded build date. After various proposals and research by Ximin about date handling in several programming languages, the best solution seems to define SOURCE_DATE_EPOCH with a value suitable for gmtime(3).
  1. Martin Borgert wondered if Sphinx could be changed in a way that would avoid having to tweak debian/rules in packages using it to produce HTML documentation.
Daniel Kahn Gillmor opened a new report about icont producing unreproducible binaries. Packages fixed The following 32 packages became reproducible due to changes in their build dependencies: agda, alex, c2hs, clutter-1.0, colorediffs-extension, cpphs, darcs-monitor, dispmua, haskell-curl, haskell-glfw, haskell-glib, haskell-gluraw, haskell-glut, haskell-gnutls, haskell-gsasl, haskell-hfuse, haskell-hledger-interest, haskell-hslua, haskell-hsqml, haskell-hssyck, haskell-libxml-sax, haskell-openglraw, haskell-readline, haskell-terminfo, haskell-x11, jarjar-maven-plugin, kxml2, libcgi-struct-xs-perl, libobject-id-perl, maven-docck-plugin, parboiled, pegdown. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues but not all of them: Patches submitted which did not make their way to the archive yet: reproducible.debian.net A new variation to better notice when a package captures the environment has been introduced. (h01ger) The test on Debian packages works by building the package twice in a short time frame. But sometimes, a mirror push can happen between the first and the second build, resulting in a package built in a different build environment. This situation is now properly detected and will run a third build automatically. (h01ger) OpenWrt, the distribution specialized in embedded devices like small routers, is now being tested for reproducibility. The situation looks very good for their packages which seems mostly affected by timestamps in the tarball. System images will require more work on debbindiff to be better understood. (h01ger) debbindiff development Reiner Herrmann added support for decompling Java .class file and .ipk package files (used by OpenWrt). This is now available in version 22 released on 2015-06-14. Documentation update Stephen Kitt documented the new --insert-timestamp available since binutils-mingw-w64 version 6.2 available to insert a ready-made date in PE binaries built with mingw-w64. Package reviews 195 obsolete reviews have been removed, 65 added and 126 updated this week. New identified issues: Misc. Holger Levsen reported an issue with the locales-all package that Provides: locales but is actually missing some of the files provided by locales. Coreboot upstream has been quick to react after the announcement of the tests set up the week before. Patrick Georgi has fixed all issues in a couple of days and all Coreboot images are now reproducible (without a payload). SeaBIOS is one of the most frequently used payload on PC hardware and can now be made reproducible too. Paul Kocialkowski wrote to the mailing list asking for help on getting U-Boot tested for reproducibility. Lunar had a chat with maintainers of Open Build Service to better understand the difference between their system and what we are doing for Debian.

4 May 2015

Lunar: Reproducible builds: first week in Stretch cycle

Debian Jessie has been released on April 25th, 2015. This has opened the Stretch development cycle. Reactions to the idea of making Debian build reproducibly have been pretty enthusiastic. As the pace is now likely to be even faster, let's see if we can keep everyone up-to-date on the developments. Before the release of Jessie The story goes back a long way but a formal announcement to the project has only been sent in February 2015. Since then, too much work has happened to make a complete report, but to give some highlights: Lunar did a pretty improvised lightning talk during the Mini-DebConf in Lyon. This past week It seems changes were pilling behind the curtains given the amount of activity that happened in just one week. Toolchain fixes We also rebased the experimental version of debhelper twice to merge the latest set of changes. Lunar submitted a patch to add a -creation-date to genisoimage. Reiner Herrmann opened #783938 to request making -notimestamp the default behavior for javadoc. Juan Picca submitted a patch to add a --use-date flag to texi2html. Packages fixed The following packages became reproducible due to changes of their build dependencies: apport, batctl, cil, commons-math3, devscripts, disruptor, ehcache, ftphs, gtk2hs-buildtools, haskell-abstract-deque, haskell-abstract-par, haskell-acid-state, haskell-adjunctions, haskell-aeson, haskell-aeson-pretty, haskell-alut, haskell-ansi-terminal, haskell-async, haskell-attoparsec, haskell-augeas, haskell-auto-update, haskell-binary-conduit, haskell-hscurses, jsch, ledgersmb, libapache2-mod-auth-mellon, libarchive-tar-wrapper-perl, libbusiness-onlinepayment-payflowpro-perl, libcapture-tiny-perl, libchi-perl, libcommons-codec-java, libconfig-model-itself-perl, libconfig-model-tester-perl, libcpan-perl-releases-perl, libcrypt-unixcrypt-perl, libdatetime-timezone-perl, libdbd-firebird-perl, libdbix-class-resultset-recursiveupdate-perl, libdbix-profile-perl, libdevel-cover-perl, libdevel-ptkdb-perl, libfile-tail-perl, libfinance-quote-perl, libformat-human-bytes-perl, libgtk2-perl, libhibernate-validator-java, libimage-exiftool-perl, libjson-perl, liblinux-prctl-perl, liblog-any-perl, libmail-imapclient-perl, libmocked-perl, libmodule-build-xsutil-perl, libmodule-extractuse-perl, libmodule-signature-perl, libmoosex-simpleconfig-perl, libmoox-handlesvia-perl, libnet-frame-layer-ipv6-perl, libnet-openssh-perl, libnumber-format-perl, libobject-id-perl, libpackage-pkg-perl, libpdf-fdf-simple-perl, libpod-webserver-perl, libpoe-component-pubsub-perl, libregexp-grammars-perl, libreply-perl, libscalar-defer-perl, libsereal-encoder-perl, libspreadsheet-read-perl, libspring-java, libsql-abstract-more-perl, libsvn-class-perl, libtemplate-plugin-gravatar-perl, libterm-progressbar-perl, libterm-shellui-perl, libtest-dir-perl, libtest-log4perl-perl, libtext-context-eitherside-perl, libtime-warp-perl, libtree-simple-perl, libwww-shorten-simple-perl, libwx-perl-processstream-perl, libxml-filter-xslt-perl, libxml-writer-string-perl, libyaml-tiny-perl, mupen64plus-core, nmap, openssl, pkg-perl-tools, quodlibet, r-cran-rjags, r-cran-rjson, r-cran-sn, r-cran-statmod, ruby-nokogiri, sezpoz, skksearch, slurm-llnl, stellarium. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues but not all of them: Patches submitted which did not make their way to the archive yet: Improvements to reproducible.debian.net Mattia Rizzolo has been working on compressing logs using gzip to save disk space. The web server would uncompress them on-the-fly for clients which does not accept gzip content. Mattia Rizzolo worked on a new page listing various breakage: missing or bad debbindiff output, missing build logs, unavailable build dependencies. Holger Levsen added a new execution environment to run debbindiff using dependencies from testing. This is required for packages built with GHC as the compiler only understands interfaces built by the same version. debbindiff development Version 17 has been uploaded to unstable. It now supports comparing ISO9660 images, dictzip files and should compare identical files much faster. Documentation update Various small updates and fixes to the pages about PDF produced by LaTeX, DVI produced by LaTeX, static libraries, Javadoc, PE binaries, and Epydoc. Package reviews Known issues have been tagged when known to be deterministic as some might unfortunately not show up on every single build. For example, two new issues have been identified by building with one timezone in April and one in May. RD and help2man add current month and year to the documentation they are producing. 1162 packages have been removed and 774 have been added in the past week. Most of them are the work of proper automated investigation done by Chris West. Summer of code Finally, we learned that both akira and Dhole were accepted for this Google Summer of Code. Let's welcome them! They have until May 25th before coding officialy begins. Now is the good time to help them feel more comfortable by sharing all these little bits of knowledge on how Debian works.

2 September 2014

Raphaël Hertzog: My Free Software Activities in August 2014

This is my monthly summary of my free software related activities. If you re among the people who made a donation to support my work (65.55 , thanks everybody!), then you can learn how I spent your money. Otherwise it s just an interesting status update on my various projects. Distro Tracker Even though I was officially in vacation during 3 of the 4 weeks of August, I spent many nights working on Distro Tracker. I m pleased to have managed to bring back Python 3 compatibility over all the (tested) code base. The full test suite now passes with Python 3.4 and Django 1.6 (or 1.7). From now on, I ll run tox on all code submitted to make sure that we won t regress on this point. tox also runs flake8 for me so that I can easily detect when the submitted code doesn t respect the PEP8 coding style. It also catches other interesting mistakes (like unused variable or too complex functions). Getting the code to pass flake8 was also a major effort, it resulted in a huge commit (89 files changed, 1763 insertions, 1176 deletions). Thanks to the extensive test suite, all those refactoring only resulted in two regressions that I fixed rather quickly. Some statistics: 51 commits over the last month, 41 by me, 3 by Andrew Starr-Bochicchio, 3 by Christophe Siraut, 3 by Joseph Herlant and 1 by Simon Kainz. Thanks to all of them! Their contributions ported some features that were already available on the old PTS. The new PTS is now warning of upcoming auto-removals, is displaying problems with uptream URLs, includes a short package description in the page title, and provides a link to screenshots (if they exist on screenshots.debian.net). We still have plenty of bugs to handle, so you can help too: check out https://tracker.debian.org/docs/contributing.html. I always leave easy bugs for others to handle, so grab one and get started! I ll review your patch with pleasure. :-) Tryton After my last batch of contributions to Tryton s French Chart of Accounts (#4108, #4109, #4110, #4111) C dric Krier granted me commit rights to the account_fr mercurial module. Debconf 14 I wasn t able to attend this year but thanks to awesome work of the video team, I watched some videos (and I still have a bunch that I want to see). Some of them were put online the day after they had been recorded. Really amazing work! Django 1.7 After the initial bug reports, I got some feedback of maintainers who feared that it would be difficult to get their packages working with Django 1.7. I helped them as best as I can by providing some patches (for horizon, for django-restricted-resource, for django-testscenarios). Since I expected many maintainers to be not very pro-active, I rebuilt all packages with Django 1.7 to detect at least those that would fail to build. I tagged as confirmed all the corresponding bug reports. Looking at https://bugs.debian.org/cgi-bin/pkgreport.cgi?users=python-django@packages.debian.org;tag=django17, one can see that some progress has been made with 25 packages fixed. Still there are at least 25 others that are still problematic in sid and 35 that have not been investigated at all (except for the automatic rebuild that passed). Again your help is more than welcome! It s easy to install python-django 1.7 from experimental and they try to use/rebuild the packages from the above list. Dpkg translation With the freeze approaching, I wanted to ensure that dpkg was fully translated in French. I thus pinged debian-l10n-french@lists.debian.org and merged some translations that were done by volunteers. Unfortunately it looks like nobody really stepped up to maintain it in the long run so I did myself the required update when dpkg 1.17.12 got uploaded. Is there anyone willing to manage dpkg s French translation? With the latest changes in 1.17.13, we have again a few untranslated strings:
$ for i in $(find . -name fr.po); do echo $i; msgfmt -c -o /dev/null --statistics $i; done
./po/fr.po
1083 translated messages, 4 fuzzy translations, 1 untranslated message.
./dselect/po/fr.po
268 translated messages, 3 fuzzy translations.
./scripts/po/fr.po
545 translated messages.
./man/po/fr.po
2277 translated messages, 8 fuzzy translations, 3 untranslated messages.
Misc stuff I made an xsane QA upload (it s currently orphaned) to drop the (build-)dependency on liblcms1 and avoid getting it removed from Debian testing (see #745524). For the record, how-can-i-help warned me of this after one dist-upgrade. With the Django 1.7 work and the need to open up an experimental branch, I decided to switch python-django s packaging to git even though the current team policy is to use subversion. This triggered (once more) the discussion about a possible switch to git and I was pleased to see more enthusiasm this time around. Barry Warsaw tested a few workflows, shared his feeling and pushed toward a live discussion of the switch during Debconf. It looks like it might happen for good this time. I contributed my share in the discussions on the mailing list. Thanks See you next month for a new summary of my activities.

2 comments Liked this article? Click here. My blog is Flattr-enabled.

7 May 2014

Julien Danjou: Making of The Hacker's Guide to Python

As promised, today I would like to write a bit about the making of The Hacker's Guide to Python. It has been a very interesting experimentation, and I think it is worth sharing it with you. The inspiration All started out at the beginning of August 2013. I was spending my summer, as the rest of the year, hacking on OpenStack. As years passed, I got more and more deeply involved in the various tools that we either built or contributed to within the OpenStack community. And I somehow got the feeling that my experience with Python, the way we used it inside OpenStack and other applications during these last years was worth sharing. Worth writing something bigger than a few blog posts. The OpenStack project is doing code reviews, and therefore so did I for almost two years. That inspired a lot of topics, like the definitive guide to method decorators that I wrote at the time I started the hacker's guide. Stumbling upon the same mistakes or misunderstanding over and over is, somehow, inspiring. I also stumbled upon Nathan Barry's blog and book Authority which were very helpful to get started and some sort of guideline. All of that brought me enough ideas to start writing a book about Python software development for people already familiar with the language. The writing The first thing I started to do is to list all the topics I wanted to write about. The list turned out to have subjects that had no direct interest for a practical guide. For example, on one hand, very few developers know in details how metaclasses work, but on the other hand, I never had to write a metaclass during these last years. That's the kind of subject I decided not to write about, dropped all subjects that I felt were not going to help my reader to be more productive. Even if they could be technically interesting.
Then, I gathered all problems I saw during the code reviews I did during these last two years. Some of them I only recalled in the days following the beginning of that project. But I kept adding them to the table of contents, reorganizing stuff as needed. After a couple of weeks, I had a pretty good overview of the contents that there I will write about. All I had to do was to fill in the blank (that sounds so simple now). The entire writing of the took hundred hours spread from August to November, during my spare time. I had to stop all my other side projects for that. The interviews While writing the book, I tried to parallelize every thing I could. That included asking people for interviews to be included in the book. I already had a pretty good list of the people I wanted to feature in the book, so I took some time as soon as possible to ask them, and send them detailed questions. I discovered two categories of interviewees. Some of them were very fast to answer ( 1 week), and others were much, much slower. A couple of them even set up Git repositories to answer the questions, because that probably looked like an entire project to them. :-) So I had to not lose sight and kindly ask from time to time if everything was alright, and at some point started to kindly set some deadline. In the end, the quality of the answers was awesome, and I like to think that was because I picked the right people! The proof-reading Once the book was finished, I somehow needed to have people proof-reading it. This was probably the hardest part of this experiment. I needed two different types of reviews: technical reviews, to check that the content was correct and interesting, and language review. That one is even more important since English is not my native language. Finding technical reviewers seemed easy at first, as I had ton of contacts that I identified as being able to review the book. I started by asking a few people if they would be comfortable reading a simple chapter and giving me feedbacks. I started to do that in September: having the writing and the reviews done in parallel was important to me in order to minimize latency and the book's release delay. All people I contacted answered positively that they would be interested in doing a technical review of a chapter. So I started to send chapters to them. But in the end, only 20% replied back. And even after that, a large portion stopped reviewing after a couple of chapters. Don't get me wrong: you can't be mad at people not wanting to spend their spare time in book edition like you do. However, from the few people that gave their time to review a few chapters, I got tremendous feedback, at all level. That's something that was very important and that helped a lot getting confident. Writing a book alone for months without having anyone taking a look upon your shoulder can make you doubt that you are creating something worth it. As far as English proof-reading, I went ahead and used ODesk to recruit a professional proof-reader. I looked for people with the right skills: a good English level (being a native English speaker at least), be able to understand what the book was about, and being able to work with correct delays. I had mixed results from the people I hired, but I guess that's normal. The only error I made was not to parallelize those reviews enough, so I probably lost a couple of months on that. The toolchain
While writing the book, I did a few breaks to build a toolchain. What I call a toolchain is set of tools used to render the final PDF, EPUB and MOBI files of the guide. After some research, I decided to settle on AsciiDoc, using the DocBook output, which is then being transformed to LaTeX, and then to PDF, or either to EPUB directly. I realy on Calibre to convert the EPUB file to MOBI. It took me a few hours to do what I wanted, using some magic LaTeX tricks to have a proper render, but it was worth it and I'm particularly happy with the result. For the cover design, I asked my talented friend Nicolas to do something for me, and he designed the wonderful cover and its little snake! The publishing Publishing in an interesting topic people kept asking me about. This is what I had to answer a few dozens of time: I never had any plan for asking an editor to publish this book. Nowadays, asking an editor to publish a book feels to me like asking a major company to publish a CD. It feels awkward. However, don't get me wrong: there can be a few upsides of having an editor. They will find reviewers and review your book for you. Having the book review handled for you is probably a very good thing, considering how it was hard to me to get that in place. It can be especially important for a technical book. Also, your book may end up in brick and mortar stores and be part of a collection, both improving visibility. That may improve your book's selling, though the editor and all the intermediaries are going to keep the largest amount of the money anyway. I've heard good stories about people using Gumroad to sell electronic contents, so after looking for competitors in that market, I picked them. I also had the idea to sell the book with Bitcoins, so I settled on Coinbase, because they have a nice API to do that. Setting up everything was quite straight-forward, especially with Gumroad. It only took me a few hours to do so. Writing the Coinbase application took a few hours too. My initial plan was to only sell online an electronic version. On the other hand, since I kept hearing that a printed version should exist, I decided to give it a try. I chose to work with Lulu because I knew people using it, and it was pretty simple to set up. The launch Once I had everything ready, I built the selling page and connected everything between Mailchimp, Gumroad, Coinbase, Google Analytics, etc. Writing the launch email was really exciting. I used Mailchimp feature to send the launch mail in several batches, just to have some margin in case of a sudden last minute problem. But everything went fine. Hurrah! I distributed around 200 copies of the ebook in the first 48 hours, for about $5000. That covered all the cost I had from the writing the book, and even more, so I was already pretty happy with the launch.
Retrospective In retrospect, something that I didn't do the best way possible is probably to build a solid mailing list of people interested, and to build an important anticipation and incentive to buy the book at launch date. My mailing list counted around 1500 people subscribed because they were interested in the launch of the book subscribed; in the end, probably only 10-15% of them bought the book during the launch, which is probably a bit lower than what I could expect. But more than a month later, I distributed in total almost 500 copies of the book (including physical units) for more than $10000, so I tend to think that this was a success. I still sell a few copies of the book each weeks, but the number are small compared to the launch. I sold less than 10 copies of the ebook using Bitcoins, and I admit I'm a bit disappointed and surprised about that. Physical copies represent 10% of the book distribution. It's probably a lot lower than most people that pushed me to do it thought it would be. But it is still higher of what I thought it would be. So I still would advise to have a paperback version of your book. At least because it's nice to have it in your library.
I only got positive feedbacks, a few typo notices, and absolutely no refund demand, which I really find amazing. The good news is also that I've been contacted with a couple of Korean and Chinese editors to get the book translated and published in those countries. If everything goes well, the book should be translated in the upcoming months and be available on these markets in 2015! If you didn't get a copy, it's still time to do so!

1 July 2013

Russ Allbery: Review: Dark Lies the Island

Review: Dark Lies the Island, by Kevin Barry
Publisher: Greywolf
Copyright: 2012
Printing: 2013
ISBN: 1-55597-651-4
Format: ARC
Pages: 185
This is a literary short story collection by a mainstream Irish writer, rather far from my normal fare. It was the second book in a shipment of Powell's Indiespensable, which is generally (when there is one) a proof or Advanced Reading Copy of something relatively obscure. I'm not really the target audience, not being much of a short story aficionado, but the whole point of doing this Indiespensable experiment is to stretch my reading range. First off, as I would generally expect by mainstream fiction, the quality of the writing is excellent. One thing I particularly like about Barry is that he's refreshingly and unhesitatingly profane. His characters curse like I expect people to curse, some of them at the drop of a hat, and for me it gives a feel of solid realism to his dialogue. He's also very good with eye dialect, giving the impression of accents without making the text hard to read. Some of these stories I liked a great deal, rather more than I had expected. But if there was a general undermining flaw for me, it was that I prefer stories that go somewhere, that have a plot or lead up to a definite point. Some of Barry's do, but a lot of them seem more vignettes or just scenes, sketches of characters, that drift off without resolution. I can admire those as writing exercises, but for me they don't fill the desire I have when I sit down with a story. I'm not sure how useful these reviews will be, given that I don't have a lot of mainstream fiction background to compare them with, but here are some reactions, for whatever they're worth. "Across the Rooftops": This is one of those sketches, but it's one of the best of them. It's a single moment on the rooftops, an attempt to capture the moment of possibility around the first romantic overture. It's sadly quite conventional in gender roles (nervous man initiating, woman as relationship gatekeeper), but apart from that it's very well-written. This kind of thing is normally not to my taste, but Barry does such a great job capturing the sense of uncertainty and the sharp focus on detail that I couldn't help but like it. (7) "Wifey Redux": This is by far my favorite story of this book. I think it's both hilarious and brilliant, right from the starting line.
This is the story of a happy marriage but before you throw up and turn the page let me say that it will end with my face pressed hard into the cold metal of the Volvo's bonnet, my hands cuffed behind my back, and my rights droned into my ear this will occur in the car park of a big-box retail unit on the Naas Road in Dublin.
It's the story of a marriage and how it changes over time, but also the story of being a father and his baffled, flailing reactions to his daughter's sexual explorations. But that's not what makes the story. What makes the story is what leads up to the arrest at the end, which had me laughing out-loud in delight. Barry is great at writing a sort of devilish humor; I wish more of the collection was like this. (9) "Fjord of Killary": This is another one of the ones I enjoyed. It's about an author who buys a hotel (with a bar) on the fjord of Killary thinking that the local culture and contact along with the wild beauty would inspire his poetry. At the start of the story, he's thoroughly sick of both the climate and the people, and the collision between his idyllic dreams and the reality, as well as his acerbic first-person complaints, is quite entertaining. A huge storm and resulting flood provides the conflict and impetus for change, and a few moments of humor in the understated local reactions. I thought the closing epiphany was a bit too obvious and expected, and the ending didn't quite work for me, but it has some good moments. (7) "A Cruelty": Here's where the collection started losing me. This is still well-written and closely observed, but in the case of this story, I wish it weren't, since I found it extremely uncomfortable. The viewpoint character is vulnerable to an encounter of unalloyed nastiness. I'm sure that's the point, but, similar to how I avoid horror, it's the sort of story I'd rather not read. (3) "Beer Trip to Llandudno": This was my second-favorite story of the collection. It's about a group of men who take periodic beer-tasting trips and rate beers. They're just normal men, from a variety of backgrounds and professions, and I thought Barry did a great job capturing the sort of camaraderie that comes from long association with a hobby. That includes, here, the awkward limits on what one talks about and the tendency to use the hobby as a shield from anything that drifts into less certain territory. I thought the story suffered somewhat from not having much of a plot or conclusion, but that's also necessary from the setup. This group of people isn't actually going to solve problems, but will be there for each other in their own way. (8) "Ernestine and Kit": Another very dark story, although that's not quite as obvious at the start. As with "A Cruelty," it's well-written but disturbing and lacks any sort of positive resolution. Here, I think Barry went a bit far into portraying people who are a large part of the popular imagination but who are exceptionally rare in reality. It's the kind of story that, at least for me, plays badly with my brain's threat analysis: too memorable for the level of actual risk, and feeling like it was playing a bit too much with popular boogeymen. (4) "The Mainland Campaign": This one was just too subtle for me. Some of the summaries for this book hint at a meaning for the story that I didn't pick up at all on my first reading. On re-reading, I suspect that analysis is right, but even re-reading the story with that interpretation in mind (this is the sort of story where I'm not sure the average American would pick it up on first reading, but others might), I still don't understand exactly what happened. I think some mainstream writing, and some short stories, tends to err on the side of subtle. (4) "Wistful England": Another one that's way over on the sketch of a moment side of stories. Unlike "Across the Rooftops," it failed to make any impression on me whatsoever; I had to skim it just now to remind myself of any details at all, and by the time I edited this review for posting, I'd forgotten it again. I think it was aiming at capturing a mood, but the mopey viewpoint character didn't mean anything to me. (3) "Doctor Sot": This is another odd one, and I'm not quite sure what I think of it. Either it's another character sketch without much of a plot, or the plot was too subtle. The protagonist is a rather unlikable drunk who doubles as a small town's incompetent doctor. The story involves his encounter with a local sort of hippie encampment, which is moderately interesting even though I couldn't stand the man. Then it leads to an encounter that clearly meant something, but which was entirely lost on me. (4) "The Girls and the Dogs": And from an absence of plot back to the deeply disturbing. This story is about a moderately unlikable drug dealer who gets refuge from an extremely unlikable crazy person in a weird relationship with a woman and her sister. (As is sadly typical, polyamory only makes an appearance when the author wants to paint a scenario that's utterly fucked up and psychotic.) There's a pseudo-fantasy element here in that the characters talk about spells, but I think the only spell involved is authorial fiat to create a deeply broken and sick game that plays into a host of negative stereotypes. Despite myself, I did get drawn into the suspense of the story (Barry's undoubtedly a good writer), but it left me feeling dirty on several levels. (4) "White Hitachi": Finally, another story that I liked, although not as good as the ones earlier in the collection. This one follows a con man and two-bit thief as he gets his brother out of lockup and tries to resolve, or at least stay ahead of, various problems in his life. Again, no real ending, but I thought Barry did a good job at taking a snapshot of a life from a perspective that I don't read much in literature (along with all the sexism and racism that comes along with it). It sort of drifts off rather than ends (although the last sentence is a great bit of characterization), but I found it oddly enjoyable. (6) "Dark Lies the Island": I'm not sure what to make of this. It's another character sketch, a moment of deep psychological significance for a character, floating somewhat alone but described in detail. But the character is a girl struggling with cutting, from a male author whose other stories did not fill me with confidence in his handling of female characters. Barry makes this psychologically believable, but I'm a little nervous about trusting his portrayal of that mindset. It's also, again, just a bit too subtle; I had a hard time getting an emotional feel for how the events of the story connect. (5) "Berlin Arkonaplatz My Lesbian Summer": The final story of the collection is fairly typical of the problem I had with most of it. It seems reasonably well-written, I had a hard time connecting with the protagonist or understanding his motivations. It looks at a slice of life with which I have no familiarity at all (and is interesting for that reason), but I don't quite trust the story enough to submerge in it. Barry seems to write a lot of protagonists who seem aimless, befuddled, or just largely out of control, and I like a bit more agency in the primary character. Silvija is an interesting character, even shown from another's perspective, but the story is almost purely observational: showing her without comment, without obvious motive, and mostly without plot. It's strangely compelling, but not quite compelling enough that I'd actually seek it out. Which, really, is the whole book in a nutshell. (6) Rating: 6 out of 10

3 November 2012

Russ Allbery: Welcome, Planet Debian

As part of a discussion about something else on debian-project, it seemed clear that people were interested in more than just Debian-relevant posts on the Planet Debian aggregator. I'd previously only pushed Debian-related posts there, but I asked explicitly and people want the full feed. So this is the first untagged post going there (and something of a test). The difference, for Planet Debian readers, is that you'll get a lot more book reviews (of non-technical books), and a smattering of software announcements. You'll also get posts like the below whenever I buy more books. I got into the habit of making haul posts from enjoying a friend's version of that, and now I use them as a way of keeping track of what I've already bought until I get around to putting stuff into a real book database. (I'm currently leaning towards Tellico, for what it's worth.) I'll try to add at least a bit of commentary so that it won't just be a list of books. If I can get back into the habit, you'll also get regular postings of photographs. I've been out of the habit for about a year, though, so we'll see how that goes. Anyway, a small haul: Kevin Barry Dark Lies the Island (mainstream)
J. Robert Lennon Familiar (mainstream)
Ken & Jo Walton GURPS: Celtic Myth (RPG) This is notable because it's my first Indiespensible shipment. Powell's Books, from which I buy nearly all of my books, has a subscription club called Indiespensible, which sends a new novel, usually from independent presses, every six weeks. I've been eyeing this for a long time and finally decided to go ahead and try it for a while. I mostly read science fiction and fantasy, so this is a good way to try to read more mainstream fiction, and since I know absolutely nothing about what's good in mainstream fiction, something curated is a good idea. I haven't cracked open either of the books yet, but I quite enjoyed the chocolate toffee. GURPS: Celtic Myth was because I discovered that there was a role-playing supplement by one of my favorite authors, so that had to be acquired, even if I'm not a GURPS player.

31 October 2012

Russell Coker: Links October 2012

The F Word has an informative post about men commenting on Feminist blogs [1]. Most of it applies to any situation where a member of a powerful group comments on an issue related to a minority group. Near the end they say: I ll also paraphrase and flesh out the most useful piece of advice I ever read, when I made the effort to research white privilege: don t expect the minority to trust you. Trust is earned, and you re just another commenter who they can t tell apart from any other commenter. You re entering someone else s space, where different rules apply. You get to have the rest of the world for people to assume you re a wonderful person. Here, you re just another one of them , and given the track record of them , it s up to you to listen, learn and prove that you re being thoughtful and honestly trying to examine your privilege. Sociological Images has an interesting article on people s perception of the sky colour it seems that the blue sky meme is modern [2]. Charles Stross wrote an interesting article about the possible uses for future low power computers [3]. He gets a bit over-excited about the possibilities for sensing making a tiny computer that can sense so many things isn t going to be easy (DSLRs are big for a reason). But having lots of powerful computers everywhere does provide lots of interesting and potentially bad opportunities. Martin Bekkelund has an interesting article about Amazon wiping DRM infected books that it had sold to a customer without giving a refund or an explanation [4]. If you want to buy ebooks it seems that the sensible thing to do would be to immediately crack them or download them from The Pirate Bay so Amazon can t steal them back. Rebecca Saxe gave an interesting talk about trying to develop methods for conflict resolution through neuroscience [5]. Barry Eisler has written an interesting article about the corruption of journalists [6]. It s really worth reading, some of the methods of corruption apply to even the more casual bloggers. Krebs on Security has an informative article about the Microsoft Tech Support phone scams [7]

12 February 2012

Gregor Herrmann: RC bugs 2012/05-06

I was at FOSDEM over the last weekend, so here's my report for two weeks now:

31 December 2011

Russell Coker: Links December 2011

Barry Ritholtz wrote an insightful post quoting Federal Reserve Bank of Kansas City President Thomas Hoenig, who warns that the nation s biggest banks are putting the U.S. capitalist society at risk [1]. Big banks oppose capitalism. Glenn Greenwald has written an insightful article for Salon about the modern definition of American excellence being the killing of supposedly bad people without any due process [2]. Mazuma Mobile buys used mobile phones [3]. They can send a post-pack to ship your old mobile to them. This is good for the environment and also saves some money. Sam Varghese has written an informative article about the Trans Pacific Partnership Agreement that will probably end up benefiting US corporations at the expense of Australian citizens [4]. Cory Doctorow has written an informative article for The Guardian about the BBC DRM plans[5]. He received information that was denied in a FOI request which shows how poor the BBC case is and how bad the Ofcom oversight is. Sam Harris has written an insightful blog post about self-defense [6]. He also has many other posts that are worth reading. Aparna Roa gave an interesting TED presentation about her robotic art [7].

10 June 2011

Gunnar Wolf: Reading revolutions: Online digital text and implications for reading in academe A (very informal) review

It's been a long time since I last took some time to read First Monday A great online publication, if you are not familiar with it, that I would categorize (and no, I'm not probably well-informed in it to be authoritative) as dealing with social, psychological aspects of the cultural shifts the online world has brought upon us (often dealing with topics related to Free Software communities, the reason I first met the publication). Firstmonday is an Open Access champion from early on. It follows an approachable but academic format (this means, it is peer-reviewed, its articles give extensive lists of references, and the articles are not the short reads we have got used to finding on the net, but, quoting from their audience profile, English is not the first language of many First Monday readers; A large percentage of First Monday readers are not a part of academia; Cultures, educational backgrounds, and fields of study vary greatly among First Monday readers.) This means, it's at least a great publication for me to follow :) Anyway After a long time not following it, I have just read Reading revolutions: Online digital text and implications for reading in academe, by Barry W. Cull; First Monday, Volume 16, Number 6 - 6 June 2011 Nice, interesting read. As I was planning on telling about the article to a couple of friends more into the subjects than myself, I'll comment+quote some bits on it. Before going any further: The article makes several references to Maryanne Wolf. No relation to her I'm not lulling my (two? are you both still reading?) readers towards her work ;-) The article talks about the differences social, even dips into some physiological aspects that the activity of reading is sustaining due to the shift from an activity done mainly from books (or other similar printed material) to the computer screen. Of course, we all know from our own experience many of the basic traits Shorter attention spans, a different reading pattern (skimming instead of reading; browsing through several related items instead of in-depth reading a single text as a knowledge unit). The article begins with an overview of reading and humankind. Cull quotes Maryanne Wolf's phrase, despite the fact that it took our ancestors about 2,000 years to develop an alphabetic code, children are regularly expected to crack this code in about 2,000 days . An interesting point I never thaught of is the start of reading as a purely mental activity, detaching reason from verbalization, ~1200 years ago:
Interestingly, these early scribes first did their work by reading out loud to themselves. Not until the ninth century did monastic regulations begin requiring silent reading. By the thirteenth century the practice of men reading silently and alone became commonplace. This shift to silent reading was a profound change, one that Darnton suggested involved a greater mental adjustment than the shift to printed text
. One last important point in this older history, that I'm quoting because I know I'll need the reference later on for one of my texts is about reading as a social activity Yes, also related to the quietness I just mentioned:
Interestingly, with the democratization of the printed text, there was a return to reading aloud. Reading was a solitary silent process only for the educated elite who could afford to buy books. For the rest of the population, as Darnton pointed out, reading was a social activity which took place in workshops, barns, and taverns [and] while children played, women sewed, and men repaired tools.
After this introduction (obviously one of the most interesting parts to me), Cull gives numbers showing how reading is evolving (in the USA and Canada), and quotes some prediction on how the future will end up adapting. Of course, I live in a place with a very different society, so I cannot comment much. Then he confronts some studies regarding specifically leisure reading, as it is a much more trustable factor than just literacy (in a world as highly literate as ours is, many people only read when they have to and have never or very seldom experienced the pleasure of reading just for the sake of it), bringing into the discussion the Internet (and computers in general) usage patterns. I found also very interesting the next section, regarding the pattern changes many libraries are facing now, specially academic/research-oriented libraries:
For several millennia, right up until just two decades ago, the central role of a library was to collect and house physical texts: from clay tablets, to scrolls, to printed books (Battles, 2003; Manguel, 2006). While printed text remains essential to most academic libraries, today s libraries have also become a core conduit via which researchers access scholarly texts online. Just within the last few years, Canadian academic libraries, in a situation similar to libraries throughout the Western world, have reached an interesting tipping point librarians now spend the majority of their collections budgets on electronic instead of printed texts.
( )
Libraries are convinced that digital text, now in its infancy, is likely to have a long future. Not only do they purchase electronic texts, but most academic libraries have also become publishers of electronic texts, whether they are digitizing large portions of their book holdings, or focusing on scanning a relatively small number of archival documents from their unique special collections.
And yes, doing some work with our Institute's library, I can confirm this trend. About e-books: I have got quite into that topic since the Kindle won my heart (and my money!) half a year ago. The little device completly changed my reading habits, I have read lots more since I carry it. And yes, I have never considered a full tablet-like device The article talks long about the difference, about the disadvantage that multitasking means to the human brain (oh, do I suffer from it!) I liked this snippet, quoting Steve Jobs a couple of years ago, regarding an Apple e-book reader question:
When Apple was rumoured to be working on an e book reader a few years ago, CEO Steve Jobs expressed his lack of interest: It doesn t matter how good or bad the product is, the fact is that people don t read anymore, he reportedly said, continuing by stating that 40 percent of the people in the U.S. read one book or less last year. The whole conception is flawed at the top because people don t read anymore (Markoff, 2008). Nicholas Carr (2010) has summed up Apple s involvement in the tablet phenomenon this way: Jobs is no dummy. As a text delivery system, the iPad is perfectly suited to readers who don t read anymore .
The issue for me is, I do enjoy reading, but I am an information addict. I know that if I have parallel information flows, my attention will surely dilute between them. Of course, as I read this article on-screen, it was hard for me to take the needed discipline not to be distracted by IMs or IRC highlights during the whole reading (which was also as an excercise for myself ;-) ). I am surprised to see this on student preferences (and even more surprised to see this data comes from my university):
In a study of students at the Universidad Nacional Aut noma de M xico (UNAM), the majority of students preferred print, and 63 percent reported that they could bear reading a document on a computer screen for no more than one hour (Ram rez Leyva, 2003). When it comes to course textbooks, a marked student preference for paper over e books has recently been found (Woody, 2010).
As we approach the end, it talks about another important topics I have often tried (and often failed) to communicate to my users: That of paratext, the meaning of the different texts, covers, items, layouts, etc. that are not part of the text itself but do shape the way we face it. To some of us, this seems obvious. To others, it is so hard to understand It closes with two more topics I will refer to. One is the permanent connectivity. All the time, more people are connected virtually all of their waking time. This affects not only learning habits but priorities. Will this near constant access to information interfere with students desire to comprehend and remember information, necessary to the educational process of turning it into knowledge? Author and university business school lecturer Don Tapscott recently suggested that students might not have to stress about the details those you can check Finally, regarding the continuity and ellaboration found in texts that are each time more common He quotes Maryanne Wolf:
I am worried about kids who are immersed in digital culture. They will get to college and they will have been Twittering so much that they won t have the patience to read those really long cognitively convoluted and complex sentences. They may not have developed those rich networks which are required in order to read at a high level of sophistication. The effort is what we are going to lose. They are becoming not so much a lazy reader, but an atrophied reader
As you can see, not only this post is meant to tell my couple-of-interested-friends to read the article, but it's mainly meant as a mental placeholder for myself. I will be surely refering to some of these items.

18 November 2010

John Goerzen: The TSA: Stupid, Owned, or Complicit?

I have long been in Bruce Schneier s camp, thinking that the TSA is a joke: nothing but security theater. A few recent examples come to mind: I don t get it. They have been completely reactionary since they began. They have a complete failure of institutional imagination. Something happens, and then a new rule comes out to prevent the thing that everybody is now expecting. And what happens about the thing that people aren t expecting yet? Nothing. So we now have to take off our shoes because one guy tried to use them for something nefarious. OK, fine, but the next guy is probably going to try something other than shoes. Which is why, I m sure, many people are pointing out that the TSA is over-reliant on technology and device detection and completely underemphasizing evildoer detection as, we are repeatedly reminded, the Israelis excel at. The TSA s attempt to remedy that was foolish at best, and, according to a recent report, not grounded in science. Which is why I am heartened that, almost a decade after 9/11, Americans are starting to let go of their fear and be ready to reclaim some sense of intelligence at the security line. The fact that politicians think there is something to be gained by being tough on TSA s invasive screening procedures, rather than risk looking soft on terrorism, is evidence of this. So, what I haven t yet worked out is this: What gives, TSA? Are they: (Note: this criticism is directed mostly at the upper levels of TSA management; I do not believe the people most of us see have the ability to change the system, even if they wanted to.) One final word: I also get annoyed at all the people that grouse at the TSA checking 80-year-olds as thoroughly as everyone else. An 80-year-old could be wearing a hidden device just as much as anyone else could, and if we don t check them, then someday they probably will. The key is to be smart about who we check carefully. Use data, behavioral analysis, simple questioning, etc it works, and is a lot better than exempting people under 13 and over 80 from screening on arbitrary grounds. Also, it might help anyone with a blurry groin. And it might just save a bunch of us from getting cancer.

25 June 2010

Anand Kumria: Barry Munday

This is a fairly classic coming of age story involving an unwanted pregnancy. Like the others in this genre, it has a twist (in this case neither of the parents are teenagers, like Juno) but revealing it would give away much of the plot. Oddly this film was finished in 2008 but has taken two years to surface. Apparently the first cuts just did not work. So it was redone a few times. I am glad that the extra time was done, since this film feels "right". Patrick Wilson, from Watchmen, is perfecly cast as Barry. The other big names make this a delight to watch, as they put in a comedic turn. There are quite a few cringe worthy moments, especially when Barry is being his pick-up artist self. But he evolves and makes the film warm-hearted. If you end up seeing this film, I think you'll enjoy it. It won't challenge you but it will entertain.

3 February 2010

Barry Hawkins: The Agile Business Analyst

Agile Business Analysts? I have recently been fleshing out my thoughts on the role of business analysts in an Agile team. This is something I have historically addressed on a case-by-case basis with clients, but when thinking about it last week at a client site, I realized that I do have a general take on how the role and responsibilities of a business analyst change when a group moves to using an Agile process. Per the usual, I ll be using Scrum as the example process where needed. The Classical Roles Historically the business analyst has had two primary roles. In a process where cross-functional communication and collaboration are minimized, these roles are essential. What little communication that takes place between the business and the software development teams purporting to serve them passes almost exclusively through one or more business analysts. I view the business analyst in a phased development approach as having two roles; Translator and Gatekeeper. The Translator The business analyst is primarily tasked with taking the needs of the business and translating them into a written document that is then handed to the software developers and testers. This role s necessity is based upon some fundamental assumptions. First, programmers and testers are on the whole, too socially retarded to actually talk to the business persons and find out what they want. Second, the business is primarily viewed as being unable to focus its thought long enough to tell the programmers and testers what they need. Since a key value of an Agile approach to software development is to have individuals and their interactions be the driving force rather than processes an tools (see The Agile Manifesto), the obliteration of this role is a primary objective. The Gatekeeper Business analysts are also viewed the gatekeepers for requirements in classical, phased software process approaches. This is a somewhat unfair role, since they rarely have the authority to prevent business or technology changes, like a meter maid trying to stop an armed terrorist. It s even more unnatural since change always happens, so it s like the meter maid being assigned to an anti-terrorist unit with no additional training, authority, or weapons. The Agile Roles Some of the more extreme practitioners of Agile methodologies say that business analysts have no role in an Agile team. I disagree, provided the role of the business analyst is allowed to evolve. The roles I ve seen them take on as valuable members of an Agile team are Facilitator, Historian, and Journalist. The Facilitator A high degree of collaboration and interaction take place between business and technology in Agile Software Development. The business analyst can serve a critical need in these interactions, facilitating communication between business and technology and making sure that critical areas are being covered in an interaction. This can be a more fulfilling experience for an analyst, since they often talk to one side, then go to other to pass on the information, all the while thinking, Man, this would be simpler if you were both in the room with me at the same time. The Historian The business analyst s documentation skill is excellent for capturing significant information that often gets lost in a team of purely technical people. I have seen some great uses of business analysts in this area. One is to document the system that the team actually builds (not the big, up-front imagined system that is covered in a phased design stage). Another is capturing key technological and architectural decisions and the context in which they were made, so that when a group revisits certain items asking, Why in the world did we decide to do that? , they have the means to be reminded or informed why a particular path was chosen. The Journalist The business analyst can also be the team s journalist, making sure the latest information makes it out to all interested parties. One creative approach I ve seen at a client site is a business analyst who has a project blog. He posts entries after each meeting between the business and technology, documenting what new user stories were created (even scans in the story cards!), a summary of the discussions held, and documents any key decisions that might have been made. Oh, and he provides an excellent executive summary at the top that s suitable for anyone s review, right up to the CEO. Tell me you wouldn t love to have that guy in your Agile team.

8 October 2009

Barry Hawkins: More Expensive and Complicated Equals Better: Carseats and Software

So I finally got around to checking out the TED site; I ve quickly become a fan. One of the first talks I watched was Steven Levitt s child carseats talk. Both the talk and the feedback comments on the TED site reminded me of things I see in software development. Here s the video if you haven t seen it.
<object height="326" width="334"><param name="movie" value="http://video.ted.com/assets/player/swf/EmbedPlayer.swf"><param name="allowFullScreen" value="true"><param name="wmode" value="transparent"><param name="bgColor" value="#ffffff"><param name="flashvars" value="vu=http://video.ted.com/talks/dynamic/StevenLevitt_2005G-medium.flv&amp;su=http://images.ted.com/images/ted/tedindex/embed-posters/StevenLevitt-2005G.embed_thumbnail.jpg&amp;vw=320&amp;vh=240&amp;ap=0&amp;ti=30&amp;introDuration=16500&amp;adDuration=4000&amp;postAdDuration=2000&amp;adKeys=talk=steven_levitt_on_child_carseats;year=2005;theme=unconventional_explanations;theme=presentation_innovation;theme=how_the_mind_works;theme=not_business_as_usual;event=TEDGlobal+2005;&amp;preAdTag=tconf.ted/embed;tile=1;sz=512x288;"><embed allowfullscreen="true" bgcolor="#ffffff" flashvars="vu=http://video.ted.com/talks/dynamic/StevenLevitt_2005G-medium.flv&amp;su=http://images.ted.com/images/ted/tedindex/embed-posters/StevenLevitt-2005G.embed_thumbnail.jpg&amp;vw=320&amp;vh=240&amp;ap=0&amp;ti=30&amp;introDuration=16500&amp;adDuration=4000&amp;postAdDuration=2000&amp;adKeys=talk=steven_levitt_on_child_carseats;year=2005;theme=unconventional_explanations;theme=presentation_innovation;theme=how_the_mind_works;theme=not_business_as_usual;event=TEDGlobal+2005;" height="326" pluginspace="http://www.macromedia.com/go/getflashplayer" src="http://video.ted.com/assets/player/swf/EmbedPlayer.swf" type="application/x-shockwave-flash" width="334" wmode="transparent"></embed></object>
Steven Levitt shows in his talk how 30 years of data on car crash fatalities imply that carseats do not outperform regular seatbelts for children ages 2 and up. Anyone who has a child or grandchildren will probably bristle at hearing that; as the father of a pre-schooler, it certainly gave me pause. We spent no small amount of time and decent chunk of money in selecting carseats for our child, thinking we had done our best to ensure our child s safety. For that matter, it would be illegal for us not to have done so. To see a decent-sized data sample suggesting my child would be better off in a seatbelt at her age is rather unsettling. By the end of the talk, I took away these observations: So, software professionals, does any of that sound familiar? It reminds me of numerous initiatives over the years that have led us inexorably to the software productivity morass we have slogged through for years now. I suppose the most heinous case study in my own experience would be J2EE and the insistence that it was the solution that any reasonable business application would choose for a platform. (Before you .Net folks jump on that one, DCOM and later the .Net enterprise stack was much the same.) And who hasn t read a benchmark or white paper with seemingly incontrovertible data depicted in highly-polished graphics insisting that Product Y is the one solution to address them all? I noticed that there were comments below the video on its page at the TED site, largely because the negative verbiage of the topmost comment jumped out at me. I took the time to read through them all (there were 36 at the time of this writing), and to my amusement, I saw parallels between them and how people react to questioning and examining our existing practices and means of developing software. See if any of these strike a chord: In recent years I ve been greatly encouraged by the willingness of companies to question whether or not the heavyweight frameworks and technology stacks are what they should be using. Helping companies slough off high-ceremony processes in favor of right-sized process that focuses on delivering the right software in a timely manner has been some of the more rewarding work I ve done the past several years. I think we have an encouraging number of people in the industry who are challenging the more expensive and complicated always equals better mantra. For the brave individuals willing to put these questions to the community at large, I hope you find some comfort in knowing that the resistance and rejection you will encounter is a thing to be expected; you are not alone in that respect. Here s hoping we continue to be open to self-examination, no matter what emotional responses it might provoke or what fear of the future it may stir up within us.

2 October 2009

Barry Hawkins: Just When a Wizard Would Have Been Most Useful: Coaching versus Contracting

Then they stopped, and Thorin muttered something about supper, and where shall we get a dry patch to sleep on? Not until then did they notice Gandalf was missing. So far he had come all the way with them, never saying if he was in the adventure or merely keeping them company for a while. He had eaten most, talked most, and laughed most. But now he simply was not there at all! Just when a wizard would have been most useful, too, groaned Dori and Nori (who shared the hobbit s views about regular meals, plenty and often).

- J.R.R. Tokien, The Hobbit

I am very fond of the works of J.R.R. Tolkien; it would not surprise me to find that many of you recognize these words from the second chapter of The Hobbit titled Roast Mutton . It occurred to me recently that there are parallels between Gandalf s role in The Hobbit and that of an Agile coach. Now, before my fellow Tolkien enthusiasts leap on their keyboards, bear with me on this. Know that I am not saying an Agile coach is on par with a wizard (OK, with one of the Istari sent by the Valar, but let s table that so as not to scare off the normal folk, alright?); that should be enough to calm you down. In the excerpt from The Hobbit at the top of the page, Bilbo and the dwarves have run into their first predicament. Note that it s not a particularly difficult situation; they just need to find shelter and partake of some food. Fire would be nice, too, if it could be managed. (Mind you, it s the first day of their journey; they set off mere hours ago fully provisioned and riding on ponies.) It isn t very long before the fledgling group finds itself not only without shelter and food, but in the hands of three rather nasty trolls who are deciding how best to eat the entire group. Gandalf returns once things have gotten out of control, and with some subtle adjustments to the situation, the crisis is averted. Could Gandalf have come back sooner and spared them this entire experience? Perhaps, but in their struggle a few key things happened. First, the group had to work out how to assess tasks at hand and appropriately delegate. To their credit, that effort was partially successful. The most skilled firestarters were assigned to that task, one of the keen-eyed dwarves was assigned to be the lookout. Second, they gathered some field experience that led to the establishment of improved practices, i.e. don t leave the ponies laden with packs when you make camp, particularly if it s all your food. (They lost most of their food that night when the pony carrying it bolted and ran straight into a nearby river later that night.) Third, Bilbo Baggins was called upon to perform his first task as burglar, albeit a fool s errand that landed them in the troll predicament. While Bilbo was wildly under-equipped for his job, he managed to work through it. That experience began a developmental journey that would prepare him for the great things that would be expected of him later on. This was not Gandalf s first adventure, nor was it the first group of people he needed to equip and challenge in order to develop them to a point that they could accomplish their goal. At this point, he s been in Middle-Earth just shy of 2,000 years. He would have been more than capable of walking them through their entire journey, but to what end? Agile coaching is a discipline that aims to help teams develop their own use of methodologies like Scrum and cross-methodology practices like testing, user stories, etc. This means equipping teams with just enough information to strike out on their own for a bit, then letting them run with it rather than dazzle them with one s own mastery so as to appear like the indispensable demigod. Until people struggle with the terse maxims of Agile Software Development, they cannot internalize them. And when the wizard is always around, why bother struggling? One of my greatest frustrations as an Agile coach is how few companies are willing to take a coaching approach with their Agile adoption. They want you to come in and be the demigod as a full-time contractor. Sure, there are fiscal, political, and seemingly practical reasons that they will cite; I chalk most of them up to being excuses for a lack of willingness to embrace what it would take to face the hard task of nurturing what you have. It s seemingly easier to just throw more money at more bodies and hope that somehow things will all work out, and surely if you can stumble upon some superstar to wrangle the mess, you ll eventually be able to browbeat them into becoming a full-time employee. Don t get me wrong; in some ways, I benefit from this dysfunction. From a selfish business standpoint, having a single full-time client is certainly easier than juggling multiple concurrent clients and their schedules. As far as actually accomplishing the aim of my business, however, I think it hinders the mission. One of my aims in working with companies is to be a coach despite being brought in as a contractor. It s certainly possible, though it is more challenging. There s not that natural separation of the coach from the team that forces them to take up the mantle on their own. Few things in my work are more rewarding than having a developer come to me and say, I wasn t really sure this Agile stuff could work, but now, after going through all this, I wouldn t want to go back to the old way of doing things. Much later in the journey of the hobbit and his dwarf companions, Bilbo is again called upon for a challenging task. His response makes me think Gandalf s approach has worked:
Perhaps I have begun to trust my luck more than I used to in the old days he meant last spring before he left his own house, but it seemed centuries ago but anyway I think I will go and have a peep at once and get it over. Now who is coming with me?

- J.R.R. Tokien, The Hobbit

Here s hoping more people will be willing to embrace the challenging, messy, and altogether beneficial task of equipping teams and allowing them to struggle when necessary, even if it means the occasional scuffle with trolls.

Next.