Search Results: "pm"

6 February 2014

Johannes Schauer: botch updates

My last update about ongoing development of botch, the bootstrap/build ordering tool chain, was three months ago with the announcement of bootstrap.debian.net. Since then a number of things happened, so I thought an update was due.

New graphs for port metrics By default, a dependency graph is created by arbitrarily choosing an installation set for binary package installation or source package compilation. Installation set vertices and source vertices are connected according to this arbitrary selection. Niels Thykier approached me at Debconf13 about the possibility of using this graph to create a metric which would be able to tell for each source package, how many other source packages would become uncompilable or how many binary packages would become uninstallable, if that source package was removed from the archive. This could help deciding about the importance of packages. More about this can be found at the thread on debian-devel. For botch, this meant that two new graph graphs can now be generated. Instead of picking an arbitrary installation set for compiling a source package or installing a binary package, botch can now create a minimum graph which is created by letting dose3 calculate strong dependencies and a maximum graph by using the dependency closure.

Build profile syntax in dpkg With dpkg 1.17.2 we now have experimental build profile support in unstable. The syntax which ended up being added was:
Build-Depends: large (>= 1.0), small <!profile.stage1>
But until packages with that syntax can hit the archive, a few more tools need to understand the syntax. The patch we have for sbuild is very simple because sbuild relies on libdpkg for dependency parsing. We have a patch for apt too, but we have to rebase it for the current apt version and have to adapt it so that it works exactly like the functionality dpkg implements. But before we can do that we have to decide how to handle mixed positive and negative qualifiers or whether to remove this feature altogether because it causes too much confusion. The relevant thread on debian-dpkg starts here.

Update to latest dose3 git Botch heavily depends on libdose3 and unfortunately requires features which are only available in the current git HEAD. The latest version packaged in Debian is 3.1.3 from October 2012. Unfortunately the current dose3 git HEAD also relies on unreleased features from libcudf. On top of that, the GraphML output of the latest ocamlgraph version (1.8.3) is also broken and only fixed in the git. For now everything is set up as git submodules but this is the major blocker preventing any packaging of botch as a Debian package. Hopefully new releases will be done soon for all involved components.

Writing and reading GraphML Botch is a collection of several utilities which are connected together in a shell script. The advantage of this is, that one does not need to understand or hack the OCaml code to use botch for different purposes. In theory it also allows to insert 3rd party tools into a pipe to further modify the data. Until recently this ability was seriously hampered by the fact that many botch tools communicated with each other through marshaled OCaml binary files which prevent everything which is not written in OCaml from modifying them. The data that was passed around like this were the dependency graphs and I initially implemented it like that because I didnt want to write a GraphML parser. I now ended up writing an xmlm based GraphML parser so as of now, botch only reads and writes ASCII text files in XML (for the graphs) and in rfc822 packages format (for Packages and Sources files) which can both easily be modified by 3rd party tools. The ./tools directory contains many Python scripts using the networkx module to modify GraphML and the apt_pkg module to modify rfc822 files.

Splitting of tools To further increase the ability to modify program execution without having to know OCaml, I split up some big tools into multiple smaller ones. Some of the smaller tools are now even written in Python which is probably much more hackable for the general crowd. I converted those tools to Python which did not need any dose3 functionality and which were simple enough so that writing them didnt take much time. I could convert more tools but that might introduce bugs and takes time which I currently dont have much of (who does?).

Gzip instead of bz2 Since around January 14, snapshot.debian.org doesnt offer bzip2 compressed Packages and Sources files anymore but uses xz instead. This is awesome for must purposes but unfortunately I had to discover that there exist no OCaml bindings for libxz. Thus, botch is now using gzip instead of bz2 until either myself or anybody else finds some time to write a libxz OCaml binding.

Self hosting Fedora Paul Wise made me aware of Harald Hoyer's attempts to bootstrap Fedora. I reproduced his steps for the Debian dependency graph and it turns out that they are a little bit bigger. I'm exchanging emails with Harald Hoyer because it might not be too hard to use botch for rpm based distributions as well because dose3 supports rpm. The article also made me aware of the tred tool which is part of graphviz and allows to calculate the transitive reduction of a graph. This can help making horrible situations much better.

Dose3 bugs I planned to generate such simplified graphs for the neighborhood of each source package on bootstrap.debian.net but then binutils stopped building binutils-gold and instead provided binutils-gold while libc6-dev breaks binutils-gold (<< 2.20.1-11). This unfortunately triggered a dose3 bug and thus bootstrap.debian.net will not generate any new results until this is fixed in dose3. Another dose3 bug affects packages which Conflicts/Replaces/Provides:bar while bar is fully virtual and are Multi-Arch:same. Binaries of different architecture with this property can currently not be co-installed with dose3. Unfortunately linux-libc-dev has this property and thus botch cannot be used to analyze cross builds until that bug is fixed in dose3. I hope I get some free time soon to be able to look at these dose3 issues myself.

More documentation Since I started to like the current set of tools and how they work together I ended up writing over 2600 words of documentation in the past few days. You can start setting up and running botch by reading the first steps and get more detailed information by reading about the 28 tools that botch makes use of as of now. All existing articles, thesis and talks are linked from the wiki home.

Andrew Pollock: [life] Day 10, Playgroup again

I was bracing myself for a rough night, but Zoe slept through until almost 6am. Today was playgroup again, so we just took a slow start to the day and headed out on the bike a bit after 9am. There was a mix of new faces and people from last week. I packed a lunchbox this time instead of relying on the snack collection in my "Dad Bag" backpack. We worked on trying to approach other girls to get them to play. Unfortunately they're all that little bit younger and more shy, so we haven't had a huge success. Zoe's been pretty happy just playing with the various toys and playing with me. Parachute time was a big hit again. I think there were about 25 adults there (I was the only Dad this week), with many with multiple children. The youngest baby I saw was a 7 week old. Zoe had been asking me what a honeydew was, so after Playgroup we dropped by the Hawthorne Garage and grabbed half a honeydew and a couple of tomatoes and I made some fritters for lunch with leftover corned beef from dinner the other night. Zoe wasn't too impressed with them. I think she didn't like the diced tomato. She declared that the honeydew was just like rockmelon. We read a story in the hammock and she went down pretty easily for nap after that. I'd just started doing the dishes when my mate Steve gave me a call. He was in the area, and he dropped in. That worked out well, because I'd been meaning to pick his brain about my little electronics project. We geeked it up for a bit. Zoe's mini lab coat arrived in the mail, and when she finally woke up after 3pm, I asked her if she wanted to go out to Bunnings to get a pair of kids-sized safety glasses, but she said she wanted to do something "fun" instead. When I asked her what she wanted to do, she initially wanted to go to Wet and Wild, but that's an all day outing and we're going next Wednesday, so she settled on going to Colmslie pool again instead. So we quickly got organised and jumped on the bike and biked over to the pool. We started out in the kid's play area, which had a couple of small slides and some fountains and whatnot, and then we gravitated over to the 50 metre pool, where Zoe practised her diving for a bit and generally quite confidently navigated the 1.5m deep end of the pool. It was good, because at least I could just stand up instead of tread water while she practised diving in off the blocks. We had a post-swim ice cream and biked home, and then I whipped up dinner and it was bathtime and bedtime. I'm going to have to start tracking my weekly kilometres with Zoe on the bike, because I think it'll be a reasonable distance.

5 February 2014

Thomas Koch: shared infrastructure coop

I'm working in a very small web agency with 4 employees, one of them part time and our boss who doesn't do programming. It shouldn't come as a surprise, that our development infrastructure is not perfect. We have many ideas and dreams how we could improve it, but not the time.
Now we have two obvious choices: Either we just do nothing or we buy services from specialized vendors like github, atlassian, travis-ci, heroku, google and others.
Doing nothing does not work for me. But just buying all this stuff doesn't please me either. We'd depend on proprietary software, lock-in effects or one-size-fits-all offerings.
Another option would be to find other small web shops like us, form a cooperative and share essential services. There are thousands of web shops in the same situation like us and we all need the same things: As I said, all of the above is available as commercial offerings. But I'd prefer the following to be satisfied:
Does something like that already exists? There already is the German cooperative hostsharing which is kind of similar but does provide mainly hosting, not services. But I'll ask them next after writing this blog post.
Is your company interested in joining such an effort? Does it sound silly?




Wouter Verhelst: "Nuclear precision weapons"

Now that the madness is mostly over, I have some time to catch up on reading my newspaper. Last week, one article in De Standaard talks about the nuclear weapons on Belgian soil which officially didn't exist (until wikileaks proved otherwise) and which are now apparently going to be modernized. The newer model would be "nuclear precision weapons". Ignoring the question of whether today's world still requires nuclear bombs (this may or may not be true, I don't care), I question the logic which leads to that phrase. A nuclear weapon is a weapon of mass destruction. By definition, a weapon of mass destruction causes collateral damage. By definition, a precision weapon is a weapon that does not cause collateral damage or, at the very lease, where every effort is made to limit the amount of collateral damage. Even the very first nuclear bombs were capable of destroying entire cities. Today's nuclear weapons, even the smaller ones, are far more powerful than those. Don't get me wrong: I'm not a peace activist. In fact, I have been contracted by companies who produce military equipment, and don't feel bad about that. But to claim that it is possible to create "nuclear precision weapons" is to deceive oneself. A nuclear weapon is not very precise.

Andrew Pollock: [life] Day 9, Kindergarten and cleaning

Another busy day today. I saw my girlfriend off in the morning, and then got stuck into cleaning the bathrooms. I ducked out to Jaycar to have another attempt at getting components. Got home, did some more cleaning, and had my chiropractic adjustment. Then I had my exit interview for work via video conference, and did some administrivia while I still had my work computer. Then the guy from Pack & Send came to take all the work equipment away. Now I feel very disconnected. Had some lunch, did more cleaning. Mum and Dad came around to go with me for pick up today, so they could see the new Kindergarten. Zoe was again fast asleep at pick up, and the kids who were awake were all in the other room eating birthday cake, so we let Zoe wake up as gently as possible and then collected Megan and squeezed everyone into the car and headed home. Zoe and Megan had a great time playing with each other again, and Megan's Dad picked her up at about 4:30pm and I started on dinner. We all had dinner and then Mum and Dad headed off, and I gave Zoe a bath and put her to bed. She was a bit emotional at bedtime, so it took a while to get her settled. Here's hoping she sleeps okay tonight.

4 February 2014

Enrico Zini: debops

Debops What I like the most about being a Developer is building tools to (hopefully) make someone's life better. I like it when my software gets used, and people thank me for it, because there was a need they had that wasn't met before, and thanks to my software now it is being met. I am maintaining software for meteorological research that is soon going to be 10 years old, and is still evolving and getting Real Work done. I like to develop software as if it is going to become a part of human cultural heritage, developing beyond my capacity, eventually surviving me, allowing society to declare that the need, small as it was, is now met, and move on to worry about some other problem. I feel that if I'm not thinking of my software in that way, then I am not being serious. Then I am not developing something fit for other people to use and rely on. This involves Development as much as it involves Operations: tracking security updates for all the components that make up a system. Testing. Quality assurance. Scalability. Stability. Hardening. Monitoring. Maintenance requirements. Deployment and upgrade workflows. Security. I came to learn that the requirements put forward by sysadmins are to be taken seriously, because they are the ones whose phone will ring in the middle of the night when your software breaks. I am also involved in more than one software project. I am responsible for about a dozen web applications deployed out there in the wild, and possibly another dozen of non-web projects, from terabyte-sized specialised archival tools to little utilities that are essential links in someone's complex toolchain. I build my software targeting Debian Stable + Backports. At FOSDEM I noticed that some people consider it uncool. I was perplexed. It provides me with a vast and reasonably recent set of parts to use to build my systems. It provides me with a single bug tracking system for all of them, and tools to track known issues in the systems I deployed. It provides me with a stable platform, with a well documented upgrade path to the next version. It gives me a release rhythm that allows me to enjoy the sweet hum of spinning fans thinking about my next mischief, instead of spending my waking time chasing configuration file changes and API changes deep down in my dependency chain. It allows me to rely on Debian for security updates, so I don't have to track upstream activity for each one of the building blocks of the systems I deploy. It allows me not to worry about a lot of obscure domain specific integration issues. Coinstallability of libraries with different ABI versions. Flawless support for different versions of Python, or Lua, or for different versions of C++ compilers. It has often happened to me to hear someone rant about a frustrating situation, wonder how come it had never happened to me, and realise that someone in Debian, who happens to be more expert than I can possibly be, had thought hard about how to deal with that issue, years before. I know I cannot be an expert of the entire stack from bare iron all the way up, and I have learnt to stand on the shoulders of giants. 'Devops' makes sense for me in that it hints at this cooperation between developers and operators, having constructive communication, knowing that each side has their own needs, trying their best to meet them all. It hints at a perfect world where developers and operators finally come to understand and trust each other's judgement. I don't know that perfect world, but I, a developer, do like to try to understand and trust the judgement of sysadmins. I sympathise with my sysadmin friends who feel that devops is turning into a trend of developers thinking they can do without sysadmins. Reinventing package managers. Bundling dependencies. Building "apps" instead of components. I wish that people who deploy a system built on such premises, have it become so successful that they end up being paid to maintain them for their whole career. That is certainly what I wish and strive for, for me and my own projects. In my experience, a sustainable and maintainable system won't come out of the startup mindset of building something quick&dirty, then sell it and move on to something else. In my experience, the basis for having sustainable and maintainable systems have been well known and tested in Debian, and several other distributions, for over two decades. At FOSDEM, we thought that we need a name for such a mindset. Between beers, that name came to be "debops". (It's not just Debian, though: many other distributions get it right, too)

3 February 2014

Wouter Verhelst: Waking up

Today is the monday after FOSDEM 2014, and I'm slowly waking up. Being in charge of the video team, and after having spent a whole week preparing followed by a whole weekend stressing about, today is a day of slowly waking up from hibernation and returning gear to various rental companies. At FOSDEM 2013, we had recorded five rooms. Even though we did lose a few talks, I felt pretty good about the whole thing, and I think we did pretty well then. It had been my plan to increase the number of rooms to be recorded from 5 to a reasonably higher number say, 10 or so but someone convinced me to do all of them. I'll readily admit I was afraid. Scratch that, I was terrified. Petrified, that we wouldn't be able to pull it off. For previous editions, we'd kept the number conservative, since video work is a high-risk business: if something goes wrong with volunteers work (say, the heralding volunteer doesn't show up), you find a replacement, or you have a staff member introduce a speaker, and the show will go on. Not so with video work: if something minor goes wrong, you will most likely lose content. If something minor goes wrong in one devroom, I'll deal with it. If something minor goes wrong in 20% of devrooms, that's 4 to 5 devrooms, and I can't deal with it. Moreover, finding enough volunteers to manage the video for five rooms was a challenge last year; I couldn't even begin to imagine how to do so for 22 of them. We did work out a way that didn't seem too far-fetched, and that might work: why not look for volunteers from the devrooms themselves? But then, they couldn't be expected to have the required experience, so we'd have to do a lot of handholding. The week before FOSDEM, four of us sat down in an office in Mechelen, and started preparing. We had a lot to do; and when you do, more jobs keep being added to the pile; TODO lists tend to grow longer, not shorter. Installing servers, testing cameras, setting twinpact dipswitches, configuring laptops. Changing a few tests in the config management system, and having to go over the laptops again. Buying extra gear. Buying yet more extra gear, several times. Buying cardboard boxes when it appeared that the plastic foldable ones we'd ordered wouldn't make it in time, and unloading the pallet of them when it did, after all, so we could load them into the van with all of the other gear. Laminating instruction sheets. Receiving shipments of rented gear. Calling another supplier last minute when one of our other suppliers had contacted us to let us know one of their cameras that they'd promised us had broken down, and couldn't be repaired in time. Calling that last-minute supplier again when we found out that some of the cameras we'd rented could do HDV only (and not DV), and we needed to find more than the ones we'd already asked for. Discovering that his cameras were of the same problematic model and so also couldn't be used. The feeling of relief when he called me a few hours later and told me he'd managed to scrounge up enough cameras to help us out after all. And then it was time to go to the ULB and start setting up everything. On friday, we had a little bit of help, but did have to do most of it ourselves. After setting up everything in every room and testing as much as possible within the time constraints that we had, we picked up the most expensive bits of the gear, brought it down to a safe place, and went home. On saturday and sunday, for me it was mostly a matter of handing out gear to various devrooms, running around to fix various issues, and then in the evening receiving boxes back, using a checklist to ensure everything expected was in the box (and that things not needed would remain in the room, or be sent back). It was exhausting, to the extent that on saturday afternoon, I had to take a bit of time off so I could go and rest. I think it's safe to say that something like this hasn't been done before. FOSDEM is a huge conference; most conferences with multiple rooms don't have more somewhere between 5 and 10 of them. The fact that we have so many more, and the fact that we record all of them, puts FOSDEM video work in a class of its own, to the extent that the professional cameramen I talked to in order to get the right cameras would incredulously ask "what do you need that for?" when I gave them the numbers we needed. As such, I didn't expect, nor was aiming for, perfection. Unfortunately, with video work it is close to impossible to attain perfection; I have told multiple people during the event that I would be happy if we reached 85% of talks recorded at acceptable quality; and though it is much too soon to say for certain, my gut feeling tells m that we've probably achieved that. One thing we hadn't planned for was streaming. During the past three years, FOSDEM enlisted the help and sponsorship of flumotion to get streaming going. Unfortunately, that did not work out this year; and since we already had far too much on our plates with everything else, we decided to forego streaming and focus on recording, only. A few days before the event, however, Steinar Gunderson took it upon himself to fix that. While we couldn't support him a great deal, we could give him access to the secondary laptops (which weren't otherwise doing much anyway) on which he could then do his thing. This was mostly transparent to us; we did communicate to some extent (e.g., when a machine that had been down had fixed and was working again), but he mostly did his thing while we did ours. Full details, for those who want it, in the linked blog post. Today, then, I spent most of my time handling hardware: waiting for other members of FOSDEM to drive the van up to my office; waiting for the laptop rental company to retrieve most of the laptops; waiting for two of the camera rental companies to retrieve their cameras and tripods; driving over to the final camera rental company to return their cameras and tripods; and finally, driving up to the ULB to retrieve the last laptops who had been copying files all night. As I write this, some video files have already been uploaded to the FOSDEM video archive. However, these are extremely low-quality renditions of video snippets that need to be reviewed; they are not ready yet for public consumption. When those files exist, expect an announcement through the main FOSDEM website.

Kees Cook: compiler hardening in Ubuntu and Debian

Back in 2006, the compiler in Ubuntu was patched to enable most build-time security-hardening features (relro, stack protector, fortify source). I wasn t able to convince Debian to do the same, so Debian went the route of other distributions, adding security hardening flags during package builds only. I remain disappointed in this approach, because it means that someone who builds software without using the packaging tools on a non-Ubuntu system won t get those hardening features. Think of a sysadmin trying the latest nginx, or a vendor like Valve building games for distribution. On Ubuntu, when you do that ./configure && make you ll get the features automatically. Debian, at the time, didn t have a good way forward even for package builds since it lacked a concept of global package build flags . Happily, a solution (via dh) was developed about 2 years ago, and Debian package maintainers have been working to adopt it ever since. So, while I don t think any distro can match Ubuntu s method of security hardening compiler defaults, it is valuable to see the results of global package build flags in Debian on the package archive. I ve had an on-going graph of the state of build hardening on both Ubuntu and Debian for a while, but only recently did I put together a comparison of a default install. Very few people have all the packages in the archive installed, so it s a bit silly to only look at the archive statistics. But let s start there, just to describe what s being measured. Here s today s snapshot of Ubuntu s development archive for the past year (you can see development opening after a release every 6 months with an influx of new packages):
Here s today s snapshot of Debian s unstable archive for the past year (at the start of May you can see the archive unfreezing after the Wheezy release; the gaps were my analysis tool failing):
Ubuntu s lines are relatively flat because everything that can be built with hardening already is. Debian s graph is on a slow upward trend as more packages get migrated to dh to gain knowledge of the global flags. Each line in the graphs represents the count of source packages that contain binary packages that have at least 1 hit for a given category. ELF is just that: a source package that ultimately produces at least 1 binary package with at least 1 ELF binary in it (i.e. produces a compiled output). The Read-only Relocations ( relro ) hardening feature is almost always done for an ELF, excepting uncommon situations. As a result, the count of ELF and relro are close on Ubuntu. In fact, examining relro is a good indication of whether or not a source package got built with hardening of any kind. So, in Ubuntu, 91.5% of the archive is built with hardening, with Debian at 55.2%. The stack protector and fortify source features depend on characteristics of the source itself, and may not always be present in package s binaries even when hardening is enabled for the build (e.g. no functions got selected for stack protection, or no fortified glibc functions were used). Really these lines mostly indicate the count of packages that have a sufficiently high level of complexity that would trigger such protections. The PIE and immediate binding ( bind_now ) features are specifically enabled by a package maintainer. PIE can have a noticeable performance impact on CPU-register-starved architectures like i386 (ia32), so it is neither patched on in Ubuntu, nor part of the default flags in Debian. (And bind_now doesn t make much sense without PIE, so they usually go together.) It s worth noting, however, that it probably should be the default on amd64 (x86_64), which has plenty of available registers. Here is a comparison of default installed packages between the most recent stable releases of Ubuntu (13.10) and Debian (Wheezy). It s clear that what the average user gets with a default fresh install is better than what the archive-to-archive comparison shows. Debian s showing is better (74% built with hardening), though it is still clearly lagging behind Ubuntu (99%):

2014, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

Petter Reinholdtsen: Testing sysvinit from experimental in Debian Hurd

A few days ago I decided to try to help the Hurd people to get their changes into sysvinit, to allow them to use the normal sysvinit boot system instead of their old one. This follow up on the great Google Summer of Code work done last summer by Justus Winter to get Debian on Hurd working more like Debian on Linux. To get started, I downloaded a prebuilt hard disk image from http://ftp.debian-ports.org/debian-cd/hurd-i386/current/debian-hurd.img.tar.gz, and started it using virt-manager. The first think I had to do after logging in (root without any password) was to get the network operational. I followed the instructions on the Debian GNU/Hurd ports page and ran these commands as root to get the machine to accept a IP address from the kvm internal DHCP server:
settrans -fgap /dev/netdde /hurd/netdde 
kill $(ps -ef awk '/[p]finet/   print $2 ')
kill $(ps -ef awk '/[d]evnode/   print $2 ')
dhclient /dev/eth0
After this, the machine had internet connectivity, and I could upgrade it and install the sysvinit packages from experimental and enable it as the default boot system in Hurd. But before I did that, I set a password on the root user, as ssh is running on the machine it for ssh login to work a password need to be set. Also, note that a bug somewhere in openssh on Hurd block compression from working. Remember to turn that off on the client side. Run these commands as root to upgrade and test the new sysvinit stuff:
cat > /etc/apt/sources.list.d/experimental.list <<EOF
deb http://http.debian.net/debian/ experimental main
EOF
apt-get update
apt-get dist-upgrade
apt-get install -t experimental initscripts sysv-rc sysvinit \
    sysvinit-core sysvinit-utils 
update-alternatives --config runsystem
To reboot after switching boot system, you have to use reboot-hurd instead of just reboot, as there is not yet a sysvinit process able to receive the signals from the normal 'reboot' command. After switching to sysvinit as the boot system, upgrading every package and rebooting, the network come up with DHCP after boot as it should, and the settrans/pkill hack mentioned at the start is no longer needed. But for some strange reason, there are no longer any login prompt in the virtual console, so I logged in using ssh instead. Note that there are some race conditions in Hurd making the boot fail some times. No idea what the cause is, but hope the Hurd porters figure it out. At least Justus said on IRC (#debian-hurd on irc.debian.org) that they are aware of the problem. A way to reduce the impact is to upgrade to the Hurd packages built by Justus by adding this repository to the machine:
cat > /etc/apt/sources.list.d/hurd-ci.list <<EOF
deb http://darnassus.sceen.net/~teythoon/hurd-ci/ sid main
EOF
At the moment the prebuilt virtual machine get some packages from http://ftp.debian-ports.org/debian, because some of the packages in unstable do not yet include the required patches that are lingering in BTS. This is the completely list of "unofficial" packages installed:
# aptitude search '?narrow(?version(CURRENT),?origin(Debian Ports))'
i   emacs                   - GNU Emacs editor (metapackage)
i   gdb                     - GNU Debugger
i   hurd-recommended        - Miscellaneous translators
i   isc-dhcp-client         - ISC DHCP client
i   isc-dhcp-common         - common files used by all the isc-dhcp* packages
i   libc-bin                - Embedded GNU C Library: Binaries
i   libc-dev-bin            - Embedded GNU C Library: Development binaries
i   libc0.3                 - Embedded GNU C Library: Shared libraries
i A libc0.3-dbg             - Embedded GNU C Library: detached debugging symbols
i   libc0.3-dev             - Embedded GNU C Library: Development Libraries and Hea
i   multiarch-support       - Transitional package to ensure multiarch compatibilit
i A x11-common              - X Window System (X.Org) infrastructure
i   xorg                    - X.Org X Window System
i A xserver-xorg            - X.Org X server
i A xserver-xorg-input-all  - X.Org X server -- input driver metapackage
#
All in all, testing hurd has been an interesting experience. :) X.org did not work out of the box and I never took the time to follow the porters instructions to fix it. This time I was interested in the command line stuff.

Andrew Pollock: [life] Day 7, Kindergarten again

Today was a pretty quiet day, because Zoe had Kindergarten again. Nominally, I'm supposed to be doing Debian stuff today, but I had some work presentations I wanted to watch while I still had the chance, so I mostly watched them for four hours while blowing the cobwebs out of my development environment. Zoe slept well last night and slept in a little bit to a whopping 6:30am. We got out the door at about 8:05am and by the time I'd got the bike trailer on the bike and made it to Kindergarten it was about 8:30am. Drop off went pretty smoothly. I decided to go for a run after I got back home, so I banged out 5km. My running fitness has been a bit off of late, partially because of the heat, and partially because I just haven't been doing it enough. I need to get back up to 10km as soon as possible so I can try increasing the distance. Both of Megan's parents work, and so I'm helping them out with picking her up on Mondays and Wednesdays. Today was the day of doing that. It was certainly an interesting experience having to wrangle two kids instead of one. Megan's a good kid, so it was no trouble. To kill time, we went to the car wash, where I bought the girls babyccinos while we waited, and then we headed into the Valley so I could check my post office box. After we got home, I attempted to do something I'd seen on Mister Maker. He basically squirted some PVA glue, some paint and some detergent into a cup and then mixed it all up and then blew into it with a straw until the bubbles rose over the top of the cup and pressed down on it with some paper. I used bubble mix instead of detergent because I figured it would have to be as good, but only Zoe's cup really worked as intended. Then, despite much discussion about it, Zoe decided to suck on the straw instead of blow it and so I had to race her off to the bathroom to rinse her mouth out. After that the girls played a bit more on their own, then Zoe wanted to watch TV but Megan didn't, so we painted a butterfly together and then Megan's Dad arrived to pick her up, and Sarah arrived not long after that. I killed some time doing a spontaneous electronics project before yoga and that was my day.

31 January 2014

Andrew Pollock: [life] Day 4, Brazilian jiu jitsu, Science Friday and the Lunar New Year

I want Zoe to do one "extra curricular" activity per term this year. Something dance-related, something gymnastics-related and maybe some other form of sport (I'm thinking soccer). My girlfriend and I were wandering through Westfield Carindale on Saturday, and we happened upon a guy from Infinity Martial Arts touting his wares. I thought I'd suss it out, and they had an introductory offer where the sign-up fee and uniform fee were significantly reduced, and they had a pretty flexible timetable. They were just starting up their East Brisbane location, and the close proximity to home along with the reduced price sealed the deal. Long-term, I'd like for Zoe to learn Tae Kwon Do, for self-defense, but BJJ seemed as good as anything to get her introduced to the idea of martial arts. The class for 2-4 year olds was billed as "Fun and Fitness 4 Kids" so it's really a combination of listening to the instructor, some basic gymnastics-style stuff and a little bit of martial arts. We biked over this morning (going up Hawthorne Road is a slog) and got there in about 15 minutes via the direct route. It's in the upstairs of a gym in the middle of an industrial area, but it was pretty easily accessible by bike. They're still waiting on some of the equipment, so the space was a little spartan. It was just Zoe and I and a mother of two not quite 2 year old twin girls taking a trial class. First up, we got her uniform, and she looked so cute. There were pants and a jacket and a belt. I'm going to have to video the instructor tying up the belt next week so I can learn how to do it the right way. The class started with the kids standing on these flat coloured circles ("mushrooms") and effectively playing "Simon Says" ("instructor says") without being caught out. It was a pretty sneaky way of doing a bunch of warm up exercises like rotating the knees and ankles. Zoe did very well, but there were a few that she just point blank refused to do. Next the instructor set up a bunch of "stations" around the room. The first station involved me crouching in a fetal position with a football and Zoe had to try and tip me over to get the football. That was a load of fun. The next station was a few steps and foam-filled vinyl ramp, and Zoe just had to do a somersault down that. The next station was pretty much the same but taller, and Zoe had to do a "sausage roll" on her side down that. The next station was just a small exercise ball and Zoe had to do some "donkey kicks" on it. The final station involved me waving a couple of cut-off pool noodles at arm's length, and Zoe had to run in covering her head and give me a bear hug. We did a few rotations of these stations. It was heaps of fun. Next, the instructor got a whole bunch of ball pit balls of different colours, and scattered them over the floor, and put a basket at each end of the room. The kids were then instructed to retrieve specific colours as fast as possible. Zoe started out trying to get as many as possible in her arms before returning to the basket, but the idea was to do it one ball at a time. A lot of running back and forth. Finally, we did some actual BJJ (I think). It was called the "sleeping crocodile hold" or something like that. I had to lie on my back, and Zoe had to sneak up to me from my side and grapple me with one arm behind my head and the other around my waist and a knee in my side. I have no interest in Zoe learning mixed martial arts, but this class was so much fun. I was feeling a bit tired this morning before we headed out, but by the end of it I was so pumped. It was just the right combination of daddy/daughter rough and tumble, with a bit of gymnastics and following instruction. I'm pretty certain Zoe enjoyed it. I liked that the instructor stopped for a water break between each activity, so the kids were kept well hydrated throughout. We took the "scenic route" home, because we had no particular time constraints. It was more like 25 minutes and involved the Norman Park Greenway. I was so glad we went that way. It was a beautiful ride that I didn't know existed. Very indirect, it involved going through Woolloongabba, Coorparoo and Norman Park, around the back of Coorparoo State High School along the side of Norman Creek. It was semi-wetland conditions.The only part that was a bit annoying was where Norman Avenue met Wynnum Road. It was quite steep and the green light didn't last very long. I had wanted to do story time at Bulimba Library at 10:30am on Fridays, but I'd rather bike home via the Norman Park Greenway instead, because it's a nice ride. I've since decided that I'll just use the story time at the library during wet weather, when we'd be driving to BJJ anyway, and be able to make it to the library in time after class. I've also got to figure out where I'm going to fit doing some Science into the schedule. Fridays are going to be busy I think. I managed to get Zoe down for a nap by a bit after 12:30pm today. She was pretty knackered after the class (as was I from biking home) so I let her watch a bit of TV while I prepared lunch. She was funny, she saw how sweaty I was when we got home, and suggested I take a shower while she watch some TV. I was a bit unprepared for my first Science Friday, though. I'd been considering going to the Sir Thomas Brisbane Planetarium after BJJ, but I discovered it's closed until February 7 for maintenance and an upgrade, so that cunning plan was thwarted. I used her nap time to order 365 Science Experiments and 50 Dangerous Things (you Should Let Your Children Do). I should get plenty of inspiration out of those two. After she woke up, we did the old "vinegar and sodium bicarbonate" trick as our science experiment. I've finally got a use for my Google Labs Founders' Award lab coat. I need to try and find some child-sized safety glasses. The adult ones barely stay on her little nose. After that, we went for a walk to our local toy store to see if they sold child-sized safety glasses (they didn't) and then Zoe watched a little bit of TV and then we walked to the CityCat to go to Teneriffe to catch a bus to Chinatown. I couldn't have timed it better if I tried. Just as we got to Chinatown, they started doing a thing with the Chinese dragons, and I hoisted Zoe up on my shoulders so she could watch. I wasn't sure how she was going to take it, with all the noise from the drums and the dragons themselves, but she was enthralled. We watched other acts and then my girlfriend joined us, and we went to a Chinese restaurant for dinner. I bought Zoe these training chopsticks a while ago, and she's taken to them like a duck to water. I brought them with us so she could use them at the restaurant, and she ate the biggest dinner I've ever seen her eat. After dinner, we caught the tail end of a procession and had some ice cream. We were starting to leave, and there was another dragon, and Zoe was brave enough to go and touch it on her own, for good luck. We then made our way back to the bus. By the time the bus arrived and got us back to Teneriffe, and a CityCat finally arrived, it was very late, but Zoe was pretty good the whole time. It was probably the latest night she's ever had with me while we've been out on the go, and she only got a bit ratty once we were home. It didn't help that she'd forgotten that she'd put Cowie in a cupboard this morning and we burned some more time tracking her down. Today was a fantastic (and very full) day. I enjoyed it a lot, and I think Zoe did too. Fortunately, Fridays won't always be this full on.

30 January 2014

Rudy Godoy: What have we done?

Couple of weeks ago I was in the situation of having to setup up a new laptop. I decided to go with wheezy s DVD installer. So far so good. I didn t expect (somewhere in the lies-I-tell-myself dept.) to have GNOME as the default Debian desktop. However after install I ve figured it was the new GNOME3 everybody was talking about. Before that I ve seen it in Ubuntu systems used by my classmates. I thought yeah, OK GNOME3, I think that s fine for them as a Linux desktop. Turns out that I ve started using the Debian desktop aka GNOME3 and noticed that it was not as bloated as the Ubuntu desktop I ve seen before looked, so I sticked with it, (I thought for a while). Turns out that I did like this new so called GNOME3, the non-window based but an application-based system (that is something that sticks in my head). I liked the way it makes sense as a desktop system, like when you look for applications, documents, connect to networks, use pluggable devices or just configure stuff every time with less and less effort. Good practices and concepts learned from Mac OS X-like environments and for sure taking advantage of the new features the Linux kernel and user-space environment got over the years. So, like one month later I stick with it and it makes sense for me to keep it. I had no chance to try the latest XFCE or KDE, my default choices before this experience. Kudos GNOME team, even after the depictions you guys had on GNOME Shell; as I learned. This whole situation got me into some pondering about the past of the Linux user experience and how we in the community lead people into joining. I remember that when I guy asked: how do I configure this new monitor/VGA card/network card/Etc? the answer was in the lines of: what is the exact chipset model and specific whole product code number that your device has? Putting myself in the shoes of such people or today s people I d say: what are you talking about? what it is a chipset? I mean, like it was too technical that only one guy with more than average knowledge could grasp. From a product perspective this is similar for a car manufacturer tell to a customer to look for the exact layout or design your car s engine has, so that they are able to tell whether is the 82 s model A or 83 s model C. Simplicity on naming and identification was not in the mindset of most of us. This is funny because as technology advances it also becomes more transparent to the customer. So, for instance, today s kids can become really power users of any new technology as if they had, indeed, many pre-set chipsets in their brain. But when going into the details we had to grasp few years ago they have some hard time figuring out the complexity of the product that presents itself on this clean and simple interface. New times, interesting times. Always good to look back. No, I m not getting old.

28 January 2014

Andrew Pollock: [life] Day 1, first day of Kindergarten

Yesterday was the first day of my new routine, except I haven't really established the new routine yet. I think that's going to take a few weeks. It was, however, the first day of Zoe's new routine, she started Kindergarten. "Kindergarten" is a bit of an ambiguous term, even within Australia, it seems. Back when I went to school, there was "pre-school", which was, as best I can determine, what "Kindergarten" is now, except a year later. It was optional, half a week, and the year preceding the start of primary ("elementary") school. I think every other state was already doing an extra year of school (which I think they called "Kindergarten", "Kinder", or something starting with "K") so Queensland blew up "pre-school", created "Prep" and now that's the first year of primary school. Zoe will start "Prep" next year. Then the Queensland education boffins realised that kids were missing out on the stuff that "pre-school" used to offer, so they reintroduced it in the form of "Kindergarten" (by Queensland standards). It's supposed to be play-based, and isn't really "long day care". Zoe does Monday, Tuesday and every other Wednesday, from 8:30am until 2:20pm. I'd only heard about C&K from a friend after moving back to Brisbane, so we were very fortunate to secure a place without having camped out on a waiting list from birth. We got very lucky, and ended up getting multiple offers, but we're pretty thrilled with the one that Zoe got into. It's a relatively new facility, right next to a state school and some bushland. They take the kids on excursions into the bushland, and up to the state school where there's some chickens. The Kindergarten facility itself is really lovely. Anyway, yesterday was Zoe's first day. I biked there yesterday morning (without Zoe) and it took me about 11 minutes. It took me more like 15 minutes to pick her up with the bike trailer. It's uphill on the way there, but not too much of a slog. It's been really great that Zoe's BFF from day care, Megan, also scored a place in the same Kindergarten, as I think this will help the transition for both girls greatly. I was able to stay for an extended period of time to help settle Zoe in, and I was able to leave without her being particularly distressed. I then biked home, running a few errands along the way. I then got stuck into a woodworking project (I'm trying to make a little wooden clothesline for Zoe). It turns out trying to cut a halved joint with a hand saw and chisel is a lot of work. I've managed to aggravate an old rotator cuff injury and the two pieces are a tad too tight, so I'm thinking I'll redo it and cut the joint out with an electric router instead. I probably spent at least a couple of hours faffing around with that. So I banged away with a hammer and chisel for a bit, got a haircut, had a late lunch and next thing I knew it was 2pm and time to head back on the bike to pick up Zoe. She was still asleep on her stretcher when I arrived to pick her up, which wasn't that surprising, since she tends to sleep later and longer at day care. I got a good opportunity to chat with her teacher about her first day and then gently woke her up with a kiss, and we headed home. I introduced Zoe to one of our new neighbours, and after a snack, we drove into Fortitude Valley where I had to pick up some mail from my post office box. When we got back home, I started on dinner while Zoe watched a little bit of TV. I spontaneously invited my cousin over to join us for dinner, and we walked down to the Hawthorne Garage for a babyccino after dinner. It was a good day. I realised that I'm going to have to be super regimented with my time on the days Zoe is in Kindergarten, because 2pm is going to roll around very quickly. "Scarcity brings clarity", as Sergey says...

27 January 2014

Richard Hartmann: You at FOSDEM?

Behind the scenes, all the pieces are falling together to make this FOSDEM the best ever. As every year ;) Part of making things even more awesome is a great effort: To tape ALL the things. As in, literally, every single talk, discussion, and lightning talk; devrooms included. We have the extra equipment, we have the extra storage space, but we don't have all the extra manpower, yet. Why not sign up as a volunteer? There's a self-checkout system where you simply sign up for tasks. Heralding and video-taping are especially simple to do if you plan to attend the respective track/devroom, anyway. If you prefer talking over listening, cloak room and infodesk are available asa well; they are a lot more social and people always tell us it's great fun. Finally, if your schedule is full to the brim, you can help us with build-up and teardown. As an aside, there's a little experiment with IPv6 in the works.

26 January 2014

Daniel Pocock: Beyond init systems: other opportunities for simplification

One thing that everybody agrees on in the debate about Debian's init system replacement is that the manifests used by most of the new schemes are far more concise than traditional init scripts. Simplification need not stop there however. Another opportunity for simplification exists in the creation of users at package installation. Every packaging tool has a different approach and all of them involve multiple lines of boilerplate script code duplicated between hundreds of packages that all need exactly the same thing. What is worse, embedding this logic into the postinst scripts on Debian or %post logic of an RPM makes it far more awkard for a system administrator to audit the user account requirements of packages (when installing many of them in a hurry) or to customise the logic for a whole site without rebuilding the whole packages. Potential improvement How to improve this? Well, packages (on any distribution) could be invited to declare the user and group accounts they depend on. A central configuration parameter in /etc/dpkg/dpkg.cfg or /etc/rpm would then control how the package manager behaves, with several possible options:
  • Create users/groups on demand without prompting
  • Abort installation/configuration of any package if the pre-requisite users do not exist on the system
  • Prompt: at the beginning of any package install, ask the administrator to create the users himself before proceeding, with an option to abort and potentially displaying some convenient LDIF fragments and other samples that the administrator can cut and paste.
Running a command like apt-get --simulate would give a system administrator an immediate view of all users/groups required by the packages they propose to install.

25 January 2014

Ben Hutchings: Looking back on my time at Solarflare, part 1

Friday was my last day at Solarflare Communications, which I joined in 2006 (originally as Level 5 Networks). When I started, Level 5 Networks (L5N) had a 2-port gigabit Ethernet controller called EF1 and a user-level network stack called EtherFabric which ran on Linux. It had nearly completed the Falcon project to develop a 10-gigabit controller (also supporing a 2-port gigabit mode) and was porting EtherFabric to Windows. Later that year, however, L5N merged with Solarflare Communications (SF), which was developing a 10GBASE-T PHY (for 10-gigabit Ethernet over twisted-pair cables). The investors and management saw the opportunity to create an integrated 10GBASE-T LAN-on-motherboard chip that would be attractive to OEMs and so had the potential for mass sales. EtherFabric was put on the back-burner, but regular net drivers for Linux, Windows and other operating systems became more important. In a change from my previous jobs, my initial role at L5N was to maintain and extend the in-house test automation software, Runbench, written in Python. I worked with Dickon Reed who had written most of Runbench up to then. One of my first major tasks was to extend it to cover DUTs running Windows. Aside from changing Runbench itself, I wrote a simple SSH server to be installed on DUTs, using the Twisted.Python framework. (Cygwin's port of OpenSSH wasn't suitable for running a lot of native Windows commands.) I rewrote the test-installation script for the Windows driver itself, converting it to Python (with a C extension to get around the WOW64 nonsense). As Windows does not allow loading a driver just once (like insmod in Linux), except by using the kernel debugger, I added a time check in the driver so the test-installation script can make it fail early when reloaded after a reboot, so that a crasher bug in the probe path would usually be recoverable. In 2007 I persuaded management in Cambridge to loan several older servers to DebConf. I took them there and back by car, and they were used to convert video for live streams and recordings. (This was repeated in 2009.) I was also asked to work on other small development tasks, including some cleanup on the Linux net driver (sfc). In mid-2007, when SF decided it was time to get sfc upstream (in-tree), I joined Steve Hodgson and Robert Stonehouse in working on that. Out-of-tree (separately distributed) drivers for Linux normally need a large number of preprocessor conditions for compatibility with Linux's kernel module API across a range of supported versions. They may also implement some features with a driver-specific interface that should be replaced with a standard interface (possibly including the work to define that standard). They may not comply with the kernel coding style or unwritten conventions for kernel code. All of these problems were present and had to be dealt with. As part of this work, I extended unifdef so that it could be used to remove all of the backward-compatibility and not-for-upstream code without introducing extra blank lines. I could then export the out-of-tree driver from CVS(!) into a kernel source tree automatically and turn it into a patch or patch series that would stand a chance of being acceptable upstream. Our first few attempts got little response, but eventually we newbies got the message that the driver was simply too big for a single submission and that putting patches on a web site because they're too big for the list was not a solution. In the next week, I removed about half of the code (on a git branch), resulting in a relatively lean driver that could be sent to the netdev list. And finally the 9th (I think) submission was accepted. Following this, I continued to work on sfc but was also brought into the Siena project. Siena (SFL9021) was the LAN-on-motherboard chip that had been planned following the merger. It combined a dual-port controller based on Falcon with a 10GBASE-T PHY and a management controller (MC) to support Lights-Out Management (LOM). (Since 10GBASE-T has not taken off, it is now mostly sold as the SFC9020 variant in which the PHY is disabled.) I worked on some of the test framework and test cases for validating the controller design in software simulation and FPGA. In the course of this I learned to read and (somewhat) understand the chip design written in Verilog, but only made a single trivial fix to it. In August 2009, the first Siena ASICs came back from the fab. There were a few ASIC-specific bugs but all of them were quickly worked around in firmware, so it would soon be ready for production. But there was much work still to be done on the driver and firmware. The firmware had until then been running in a smaller block of RAM with no peripherals to manage, while sfc had earlier been modified just enough to configure a sim or FPGA for running a userland test application. sfc now had many special cases scattered around it, and would tell the MC firmware to peek and poke registers that were no longer directly accessible from the host. The software and firmware developers had agreed that, in order to support LOM, the MC would be responsible for managing the ports and all peripherals and drivers would send higher-level requests to the MC. As a bonus, this removed the need for the driver to know the details of each new board, so that it would not be necessary to backport sfc into distribution kernels very often. Over the next few months, the firmware team did an excellent job of defining and implementing those operations, including writing the driver-side functions to invoke them. Meanwhile, Steve and I concentrated on refactoring sfc so most of the Falcon/Siena differences could be abstracted through a few structures with function pointers. The refactoring and new code for Siena were completed and submitted upstream in several large patch series in October and November. In fact, there were so many patches that Solarflare appeared in LWN's table of who wrote Linux 2.6.33 (as did I, thanks also to another patch series adding firmware metadata to many drivers). Sadly we had missed Linux 2.6.32 which was a longterm stable branch and used in many distributions, but I was able to get this version backported into all the major distributions over the next year. (To be continued.)

Russell Coker: Links January 2014

Fast Coexist has an interesting article about the art that Simon Beck creates by walking in snow [1]. If you are an artist you can create art in any way, even by walking in patterns in the snow. Russ Altman gave an interesting TED talk about using DNA testing before prescribing drugs [2]. I was surprised by the amount of variation in effects of codeine based on genetics, presumably many other drugs have a similar range. Helen Epstein wrote an interesting article about Dr. Sara Josephine Baker who revolutionised child care and saved the lives of a huge number of children [3]. Her tenacity is inspiring. Also it s interesting to note that the US Republican party was awful even before the Southern Strategy . The part about some doctors opposing child care because it s the will of God for children to die and keep them in employment is chilling. Jonathan Weiler wrote an insightful article about the problems with American journalism in defending the government [4]. He criticises the media for paying more attention to policing decorum than to content. Tobias Buckell wrote an interesting post about the so-called socialised health-care in the US [5]. He suggests that Ronald Reagan socialised health-care by preventing hospitals from dumping dying people on the street. I guess if doing nothing for people until they have a medical emergency counts as socialised health-care then the US has it. Kelvin Thomson MP made some insightful comments about climate change, the recent heat-wave in Australia, and renewable energy [6]. Iwan Baan gave an interesting TED talk about ways that people have built cheap homes in unexpected places [7], lots of good pictures. Racialicious has an interesting article by Arturo R. Garc a about research into the effects of concussion and the way the NFL in the US tried to prevent Dr. Bennet Omalu publicising the results of his research [8]. Stani (Jan Schmidt) wrote an interesting post about how they won a competition to design a commemerative Dutch 5 Euro coin [9]. The coin design is really good (a candidate for the geekiest coin ever), I want one! Seriously if anyone knows how to get one at a reasonable price (IE close to face value for circulated or not unreasonably expensive for uncirculated) then please let me know. When writing about Edward Snowden, Nathan says Imagine how great a country would be if if it were governed entirely by people who Dick Cheney would call Traitor [10]. That s so right, that might make the US a country I d be prepared to live in. Andrew Solomon gave an interesting TED talk Love No Matter What about raising different children [11]. Aditi Shankardass gave an interesting TED talk about using an ECG to analyse people diagnosed wit severe Autism and other developmental disorders [12]. Apparently some severe cases of Autism have a root cause that can be treated with anti-seizure medication. George Monbiot wrote an insightful article about the way that Bono and Bob Geldoff promote G8 government intervention in Africa and steal air-time that might be given to allow Africans to represent themselves in public debates [13]. Daniel Pocock wrote an informative article about racism in Australian politics and how it is bad for job-seekers and the economy (in addition to being horribly wrong) [14]. Aeon Magazine has an interesting article by Anne Buchanan about the difference between scientists and farmers [15]. She has some interesting points about the way that the lack of general knowledge impacts research, but misses the point that in most fields of study there is a huge problem of people not knowing about recent developments in their own field. I don t think it s a pipe dream to be well educated in humanities and science, but I guess that depends on the definition of well educated . Brian Cox gave an interesting TED talk titled Why We Need the Explorers about the benefits of scientific research [16]. Yupu Zhang, Abhishek Rajimwale, Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau from the University of Wisconsin-Madison wrote an interesting paper about ZFS corruption in the face of disk and memory errors [17]. One thing to note is that turning off atime can reduce the probability of a memory error leading to corrupt data being written to disk, run zfs set atime=off tank to fix this. The comedian Solomon Georgio celebrated Martin Luther King day by tweeting I love you to racists [18]. It s an interesting approach and appears to have worked well.

24 January 2014

Erich Schubert: Definition of Data Science

Everything is "big data" now, and everything is "data science". Because these terms lack a proper, falsifiable definition.
A number of attempts to define them exist, but they usually only consist of a number of "nice to haves" strung together. For Big Data, it's the 3+ V's, and for Data Science, this diagram on Wikipedia is a typical example.
This is not surprising: effectively these term are all marketing, not scientific attempts at definiting a research domain.
Actually, my favorite definition is this, except that it should maybe read pink pony in the middle, instead of unicorn.
Data science has been called "the sexiest job" so often, this has recently led to an integer overflow.
The problem with these definitions is that they are open-ended. They name some examples (like "volume") but they essentially leave it open to call anything "big data" or "data science" that you would like to. This is, of course, a marketers dream buzzword. There is nothing saying that "picking my nose" is not big data science.

If we ever want to get to a usable definition and get rid of all the hype, we should consider a more precise definition; even when this means making it more exclusive (funnily enough, some people already called above open-ended definitions "elitist" ...).
Big data:
  • Must involve distributed computation on multiple servers
  • Must intermix computation and data management
  • Must advance over the state-of-the-art of relational databases, data warehousing and cloud computing in 2005
  • Must enable results that were unavailable with earlier approaches, or that would take substantially longer (runtime or latency)
  • Must be disruptively more data-driven
Data science:
  • Must incorporate domain knowledge (e.g. business, geology, etc.).
  • Must take computational aspects into account (scalability etc.).
  • Must involve scientific techniques such as hypothesis testing and result validation.
  • Results must be falsifiable.
  • Should involve more mathematics and statistics than earlier approaches.
  • Should involve more data management than earlier approaches (indexing, sketching&hashing etc.).
  • Should involve machine learning, AI or knowledge discovery algorithms.
  • Should involve visualization and rapid prototyping for software development.
  • Must satisfy at least one of these shoulds in a disruptive level.
But this is all far from a proper definition. Partially because these fields are so much in flux; but largely because they're just too ill-defined.
There is a lot of overlap, that we should try to flesh out. For example, data science is not just statistics. Because it is much more concerned with how data is organized and how the computations can be made efficiently. Yet often, statistics is much better at integrating domain knowledge. People coming from computation, on the other hand, usually care too little about the domain knowledge and falsifiability of their results - they're happy if they can compute anything.
Last but not least, nobody will be in favor of such a rigid definition and requirements. Because most likely, you will have to strip that "data scientist" label off your business card - and why bite the hand that feeds? Most of what I do certainly would not qualify as data science or big data anymore with an "elitist" definition. While this doesn't lessen my scientific results, it makes them less marketable.
Essentially, this is like a global "gentlemans agreement". Buzz these words while they last, then move on to the next similar "trending topic".

Maybe we should just leave these terms to the marketing folks, and let them bubble them till it bursts. Instead, we should just stick to the established and better defined terms...
Thank you.
Of course, sometimes you will have to play Buzzword Bingo. Nobody is going to stop you. But I will understand that you are doing "playing buzzword bingo", unless you get more precise.
Once you then have results that are so massively better, and really disrupted science, then you can still call it "data science" later on.
You have been seeing, I've been picking on the word "disruptive" a lot. As long as you are doing "business as usual", and focusing on off-the-shelf solution, it will not be disruptive. And it then won't be big data science, or a big data approach that yields major gains. It will be just "business as usual" with different labels, and return results as usual.
Let's face it. We don't just want big data or data science. What everybody is looking for is disruptive results, which will require a radical approach, not a slight modification involving slightly more computers of what you have been doing all along.

Julien Danjou: OpenStack Ceilometer Icehouse-2 milestone released

Yesterday, the second milestone of the Icehouse development branch of Ceilometer has been released and is now available for testing and download. This means the first half of the OpenStack Icehouse development has passed! New features For the Icehouse-1 milestone, we barely had enough time to implement 2 blueprints. We almost did a better job this time, but finally only 2 blueprints were implemented again. This is really far from what we planned initially. The infrastructure slowdown issues and the lower number of reviews is probably the root cause here. Anyway, Ceilometer now offers a REST API to accesses the stored event. The initial work to replace the /v2/meters endpoint with something more RESTy has started with the implementation of /v2/samples. Bug fixes Thirty-one bugs were fixed, though most of them might not interest you so I won't elaborate too much on that. Go read the list if you are curious. Toward Icehouse 3 We now have 29 blueprints targeting the Ceilometer's third Icehouse milestone, with some of them are already started and ready to merge. However, it's likely that we won't make all of them. As usual, the priority should indicate how confident we are that we want and need a feature. Still, it's likely the roadmap will be adjusted in the upcoming weeks. I'll try to make sure we'll get there without too much trouble for the 6th March 2013. Stay tuned!

22 January 2014

Erich Schubert: The init wars

The init wars have recently caught a lot of media attention (e.g. heise, prolinux, phoronix). However, one detail that is often overlooked: Debian is debating over the default, while all of them are already supported to a large extend, actually. Most likely, at least two of them will be made mandatory to support IMHO.
The discussion seems to be quite heated, with lots of people trying to evangelize for their preferred system. This actually only highlights that we need to support more than one, as Debian has always been about choice. This may mean some extra work for the debian-installer developers, because choosing the init system at install time (instead of switching later) will be much easier. More often than not, when switching from one init system to another you will have to perform a hard reset.
If you want to learn about the options, please go to the formal discussion page, which does a good job at presenting the positions in a neutral way.
Here is my subjective view of the init systems:
  • SysV init is the current default, and thus deserves to be mentioned first. It is slow, because it is based on a huge series of shell scripts. It can often be fragile, but at the same time it is very transparent. For a UNIX system administrator, SysV init is probably the preferred choice. You only reboot your servers every year anyway.
  • upstart seems to be a typical Canonical project. It solves a great deal of problems, but apparently isn't good enough at it for everybody, and they fail at including anyone in their efforts. Other examples of these fails include Unity and Mir, where they also announced the project as-is, instead of trying to get other supporters on board early (AFAICT). The key problem to widespread upstart acceptance seems to be the Canonical Contributor License Agreement that many would-be contributors are unwilling to accept. The only alternative would be to fork upstart completely, to make it independent of Canonical. (Note that upstart nevertheless is GPL, which is why it can be used by Debian just fine. The CLA only makes getting patches and enhancements included in the official version hard.)
  • systemd is the rising star in the init world. It probably has the best set of features, and it has started to incorporate/replace a number of existing projects such as ConsoleKit. I.e. it not only manages services, but also user sessions. It can be loosely tied to the GNOME project which has started to rely on it more and more (much to the unhappyness of Canonical, who used to be a key player for GNOME; note that officially, GNOME chose to not depend on systemd, yet I see this as the only reliable combination to get a complete GNOME system running, and since "systemd can eventually replace gnome-session" I foresee this tie to become closer). As the main drawback, systemd as is will (apparently) only work with the Linux kernel, whereas Debian has to also support kFreeBSD, NetBSD, Hurd and the OpenSolaris kernels (some aren't officially supported by Debian, but by separate projects).
So my take: I believe the only reasonable default is systemd. It has the most active development community and widest set of features. But as it cannot support all architectures, we need mandatory support for an alternative init system, probably SysV. Getting both working reliably will be a pain, in particular since more and more projects (e.g. GNOME) tie themselves closely to systemd, and would then become Linux-only or require major patches.
I have tried only systemd on a number of machines, and unfortunately I cannot report it as "prime time ready" yet. You do have the occasional upgrade problems and incompatibilities, as it is quite invasive. From screensavers activating during movies to double suspends, to being unable to shutdown my system when logged in (systemd would treat the login manager as separate session, and not being the sole user it would not allow me to shut down), I have seen quite a lot of annoyances happen. This is an obvious consequence of the active development on systemd. This means that we should make the decision early, because we will need a lot of time to resolve all these bugs for the release.
There are more disruptions coming on the way. Nobody seems to have talked about kDBUS yet, the integration of an IPC mechanism like DBUS into the Linux kernel. It IMHO has a good chance of making it into the Linux kernel rather soon, and I wouldn't be surprised if it became mandatory for systemd soon after. Which then implies that only a recent kernel (say, mid-2014) version might be fully supported by systemd soon.
I would also like to see less GNOME influence in systemd. I have pretty much given up on the GNOME community, which is moving into a UI direction that I hate: they seem to only care about tablet and mobile phones for dumb users, and slowly turn GNOME into an android UI; selling black background as major UI improvements. I feel that the key GNOME development community does not care about developers and desktop users like me anymore (but dream of being the next Android), and therefore I have abandoned GNOME and switched to XFCE.
I don't give upstart much of a chance. Of course there are some Debian developers already involved in its development (employed by Canonical), so this will cause some frustration. But so far, upstart is largely an Ubuntu-only solution. And just like Mir, I don't see much future in it; instead I foresee Ubuntu going systemd within a few years, because it will want to get all the latest GNOME features. Ubuntu relies on GNOME, and apparently GNOME already has chosen systemd over upstart (even though this is "officially" denied).
Sticking with SysV is obviously the easiest choice, but it does not make a lot of sense to me technically. It's okay for servers, but more and more desktop applications will start to rely on systemd. For legacy reasons, I would however like to retain good SysV support for at least 5-10 more years.

But what is the real problem? After all, this is a long overdue decision.
  • There is too much advocacy and evangelism, from either side. The CTTE isn't really left alone to do a technical decision, but instead the main factors have become of political nature, unfortunately. You have all kinds of companies (such as Spotify) weigh in on the debate, too.
  • The tone has become quite aggressive and emotional, unfortunately. I can already foresee some comments on this blog post "you are a liar, because GNOME is now spelled Gnome!!1!".
  • Media attention. This upcoming decision has been picked up by various Linux media already, increasing the pressure on everybody.
  • Last but not least, the impact will be major. Debian is one of the largest distributions, last but not least used by Ubuntu and Steam, amongst others. Debian preferring one over the other will be a slap in somebodys face, unfortunately.
So how to solve it? Let the CTTE do their discussions, and stop flooding them with mails trying to influence them. There has been so much influencing going on, it may even backfire. I'm confident they will find a reasonable decision, or they'll decide to poll all the DDs. If you want to influence the outcome provide patches to anything that doesn't yet fully support your init system of choice! I'm sure there are hundreds of packages which do neither have upstart nor systemd support yet (as is, I currently have 27 init.d scripts launched by systemd, for example). IMHO, nothing is more convincing than have things just work, and of course, contributing code. We are in open source development, and the one thing that gets you sympathy in the community is to contribute code to someone elses project. For example, contribute full integrated power-management support into XFCE, if you include power management functionality.
As is, I have apparently 7 packages installed with upstart support, and 25 with systemd support. So either, everybody is crazy about systemd, or they have the better record of getting their scripts accepted upstream. (Note that this straw poll is biased - with systemd, the benefits of not using "legacy" init.d script may just be larger).

Next.

Previous.