Search Results: "nate"

1 September 2013

Andrew Pollock: [life] My first Bridge to Brisbane

One of the goals I set for myself on returning to Brisbane was running in the Bridge to Brisbane. The City2South run ended up being a longer distance, sooner, so I only stuck with the Bridge to Brisbane so I could say I'd done it. The added variable was I had Zoe, and I was pushing her in a jogging stroller. I did a couple of practice runs up the bridge with Zoe in the weeks leading up to the run. Mostly to get her used to it, and for me to get a feel for what the incline was like. I found I had the stamina to easily do the bridge, but ran out of steam on the return leg. No big deal, I didn't need to do that for the real run. I did a poor job of publicising that I was doing the run, so I didn't end up raising a whole lot of money for my chosen charity, beyondblue, because I had the added variable of Zoe. I knew I had to get her up early to get to the start line in time, and I wasn't sure if she was going to be cooperative or not, or whether she'd have a tantrum half way through the run and I'd have to stop, so I didn't feel good about taking money from people for running a run that I wasn't sure I could complete. My main concern had been getting to the starting line in time. I wanted to take the train from Morningside to Murarrie and then walk to the start line (about 10 minutes) because that way I could take the train back to Morningside from the finish to collect the car and not have to worry about trying to get back to the start. That meant I'd have to get Zoe up before 5am to get to the station, and if she didn't cooperate, I'd miss the train, and it'd all be over. Fortunately, my cousin Renata offered to drop us off, and we did a complicated car switcharoo, which involved her leaving my car at Morningside station for me after dropping us off at Murarrie. Now that I've seen the road closure situation, I feel pretty confident that I could have just driven to Murarrie and ditched the car somewhere there. Interestingly, a train that was packed to the rafters with people was arriving at the station at the time we arrived to leave Renata's car there. I'm not sure if it was the 5:07am train running late, or if I'd mis-interpreted the timetable and there was a later train we could have more comfortably made without Renata's assistance. Anyway, we uneventfully got to Murarrie station by about 5:40am, and walked to the start. It was ridiculously packed. People as far as the eye could see. Apparently there were 29,479 entrants and 24,596 finishers in the 10km run, so that explains it. I was seeded in the "joggers" group, which was a fair way back from the start line, and as it was, technically had to walk around some massive barrier to get in with the runners, but I snuck in the side after the pack started moving. Zoe was really, really good for the whole run. I couldn't have asked for her to be better behaved, really. She was wanting a cuddle at the top of the bridge, and claimed she wanted to go to the toilet the first time she saw some portable toilets, but easily talked out of both. My strategy of having a broad array of snacks for her to eat during the run worked well to keep her happy, and the course was heavily packed the entire way, so she had plenty to look at. It was a nice father/daughter thing to do for Fathers Day. My official time was 57:28, which I'm extremely happy with, given the conditions I ran under. I was going to be happy with anything under an hour. My personal best for 10km (not that course) is 51:52 I'm very grateful to those people who did donate some money on short notice. I've managed to raise $180, which is better than nothing. Now that I've done the run, and seen how it works, I'm game to try it again next year, and I'm game to do more 10km runs with Zoe. My running overall has been going very well, and so I think it's time to step things up a bit and I've decided to set myself a goal of running the half-marathon in the Gold Coast Marathon next year.

31 August 2013

Christian Perrier: DebConf 13: hacking outcome

I *did* some hacking at DebConf, as usual. For once, I had no TODO list: after all, I never complete these, so why make one? Still, I participated to discussions around samba packaging, mostly animated by Ivo De Decker and I think we're making good progress towards samba 4.x packages. The road is long, quite complicated, but we now have a stronger team, with a very active Ivo, Jeroen Dekkers who officially joined, Steve Langasek who still cherishes one of his pet packages (and even branched in our git with the Ubuntu packages). Great work and thanks to Ivo for pushing this forward. I also worked quite actively on the migration of fonts packages to git (I now reached the point where I'm more comfortable with git than SVN, yes, everything can happen). These packages were modernized at the same time and checked for new upstream versions (I have to say that few of these fonts had new upstream versions, indeed). We unfortunately found no time to have a good font BoF in Vaumarcus, indeed...but I'm not sure we would have many things to say. The work is done and done well, in this team. Some progress was made, also, for restoring a working "monolithic" build of D-I. This build gathers together all udebs from unstable, which means it offers a D-I image that uses ALL udebs from nustable.....which is not the default of other images. This would be very helpful for translators who want to check their work as soon as possible. In the future, with a Jenkins task that would build each package at each commit and then build a monolithic image, we could have a way to provide a tes snapshot of D-I git repos.....which could help catching more bugs (or more of my stupid mistakes). So, in general, I consider this a quite successful DebConf when it comes at "real" production.

30 August 2013

Daniel Pocock: Big decision?

Most people have probably heard that elections are on in Australia. I just received my postal ballot paper the other day and in the spirit of transparency, I'm sharing it here on my blog. Spoilt for choice There are two ballots, one for the Lower House and the other for the Senate. The big one is for the Senate. Our cat is on top of it to help give some perspective on how big it is. Like the columns on a bloated spreadsheet, the parties are designated with symbols from A to AM and then there are some leftover candidates too.

Animal instincts With so many candidates to choose from, even our kitty may have found an opportunity in the Animal Justice Party - although that little snack in her claws suggests that she is more interested in the law of the jungle than any political debate. She may want to be wary not to fall under the wheel of the party next to them, Motoring Enthusiasts.

The Senate, a place for convicted hackers? The next party in the picture is Wikileaks and their number one candidate, of course, is the first person to successfully claim asylum from Australia, Julian Assange. It's no secret that he has a string of convictions for computer hacking in the 90s and that when US authorities catch up with him, their drone won't just be dropping him a letter demanding a $2,100 penalty fine through the window. So how do they compare? As I contemplate my decision, lets look at the numbers:

Party	Seats won in 2010 House of Reps, Senate	Criminal charges against members in last 3 years
Labor Party	72, 31	173, mainly fraud
Liberal/National coalition (conservative)	73, 34	3, including shoplifting and assault
Wikileaks	0	None - Assange is not actually charged with any crime

All of a sudden, in a country that is often the butt of British jokes about convicts, Assange starts to look squeaky clean. He may take further comfort from knowing that out of all those crimes, hacking is the only one that appears to have the presidential seal of approval. Conclusion Whichever way I vote may or may not make so much difference. With so many candidates to choose from though, there is little sign of any quality leadership from any corner of Australian politics. Whether Mr Assange is elected or not, many people will have a chuckle when they realise he almost ended up alongside the Australian Sex Party out of 38 other possible parties randomly distributed across the ballot. Please feel free to comment: which party is the most ridiculous?

28 August 2013

Vincent Sanders: Men admire the man who can organize their wishes and thoughts in stone and wood and steel and brass.

I would probably not yet worthy of the admiration Emerson was alluding to but I do like to make things. As anyone who has read previous posts knows I have pretty much embraced the "do things, tell people" idea.

One small wrinkle is doing things needs somewhere to work. Since moving myself and the family to rented accommodation in Cambridge (swampy 3,500 year old English city, not the one in Massachusetts) I have been lacking space to do practical projects.

To fix this lack I have joined the cambridge makespace which, in addition to somewhere I can work gives me access to some tools I was previously unable to afford. The space gives practical training on the more complex machines (any tool can be dangerous if you do not use it correctly) which recently allowed my induction on the CNC router.

My instructor , Mark Mellors, who was good enough to give up some of his valuable making time to train me (and accidentally get his car stuck in a car park overnight by staying late) suggested that it was a good idea to have a design to try.

I decided to use this opportunity to actually create something useful (though now I re-read Marks message it did suggest a simple design...oopsy). I had been working at the electronics bench in previous weeks and been uncomfortable using the existing stools and chairs as they were either a bit high or unable to be adjusted high enough for the 80cm tall benches. I decided to design a 60cm tall stool for use at this workbench.

My initial idea was for a simple three leg stool, round top, three legs, how hard could it be? Initial research showed that showed that above 30cm the legs needed to be braced to each other. This is because the leg to seat joints simply cannot handle the stress caused by leverage which longer legs introduce.

I looked at the structure of several stools online and was initially drawn towards creating something like the IKEA Dalfred bar stool. It was discovered that the design would be easier to realise if it were made from sheet material which give a smaller challenge to a naive operator of a CNC router. Because of this simpler designs were researched and I eventually found a simple design I liked.

The design could not however be used directly as it was for imperial sized material which is not available in europe. I selected the QCAD open source CAD package and recaptured the design adjusting for the available 12mm plywood sheet material. This resulted in an imperial measurement design for metric materials.

Mark helped me transfer the DXF into the CAM software (vcarve pro 7, after I discovered the demo version of this software generates files which cannot be imported into the full edition!) and generate toolpaths for our machine. Once the toolpaths are saved to a USB stick (no modern conveniences like direct upload here) the job can be run on the machine.

Here I ran into reality, turns out that tolerances in imperial combined with lack of understanding how plywood reacts resulted in excessive play in the joints. This resulted in an unusable stool, which simply tried to rotate around its central axis and become flatpack. I had successfully turned 10 worth of plywood into some sawdust and a selection of useless shapes. On the other hand I did become competent with the router workflow so it was not a complete failure.

I was determined to make the design work so I decided to start from scratch with a similar design but entirely in metric. I performed some material research both online and practically (why yes i did spend an informative hour in several cambridge DIY shops with digital calipers, why do you ask?) it turns out that generally available 12mm thick plywood actually ranges between 11.9 and 12mm thick.

I did some test slot cuts of varying width and determined that the available stock can be "persuaded" to fit into a 11.8mm wide slot. This Interference fit joint is strong and removes the need for adhesive in most cases.

Second cut still attached to the bed. Leftovers of first attempt in the background

The sheet plywood material is readily available with a width of 1220mm and a height of 606mm (a full sheet is 2440mm long which is cut into four with a 4mm wide saw blade) so making the legs fit within a sheet and be close to the 60cm target should be possible.

I selected a suitable seat radius (175mm) and from that determined the minimal gap to the base ring with a 6mm end mill tool (20mm for two toolpaths and some separation) and hence the minimum suitable width of the base ring (50mm) giving a total radius of 245mm.

For the legs allowance was made for two joints of 30mm with 30mm separation between them the legs come out at 90mm width. If a 6mm space and a 6mm toolpath top and bottom of the sheet is accounted for a 582mm height (606 - 24) is available in which to fit the legs. The top of the leg which fits into the seat is 12mm tall leaving 570mm total leg height.

At this point I selected some arbitrary values for the leg positioning and angle (30mm from seat centre and 15 ) using a right angle triangle triganometry this produced a stool with a base of around 550mm or 100mm outside the radius of the seat. This seemed a pleasing shape and when the values for the holding ring were calculated it is at 104mm height which also seems to work out well.

This design was cut on the machine, lightly finished with some sandpaper and assembled. The legs slotted into the ring first and then the legs eased into the seat slots, the whole thing flipped and the seat hammered home onto the legs.

Anne Harrison volunteering to try the wobbly stool of doom

Success! It physically fit together and if you were brave enough you could sit on it. Unfortunately the plywood seemed to flex around the central axis of rotation in a rather alarming way, fine if you are under 70 kilos but not giving the impression of security most people want from their seating.

Ok, lets try with five legs instead of three (at least we can reuse the existing three legs)...nope still not good enough and another 10 gone.

The others in the space suggested a few ideas to improve matters and the one I selected was to use 18mm plywood instead of 12mm, this should improve rigidity. There was a brief pause in proceedings to discover 18mm sheet is actually 17.7mm and needs a 17.5mm slot to make the interference fit work.

A swift redesign later altering the seat radius, gap, ring width and leg height to accommodate the new material and we have version 4 and it works without caveat. Tested up to 150Kg load without trouble, there is a small amount of flex still but nothing that feels worrying.

I finished the stool by rounding the seat top edge with a ball bearing rounding bit in a manual router and applying a couple of coats of gloss acrylic varnish. Finished stool is now doing service at the space.

In conclusion the final design allows someone with a CNC router to create a useful 580mm high fixed stool for 7.50 in timber plus cutter wear and varnish so maybe 8.50 total.

You can actually get five legs and a seat/ring out of a 1220x606 sheet and with intelligent arrangement a single 1220x2440 sheet will probably yield five or possibly six stools in total

I am making the design files of the proven 18mm version available (heck they are all there...but you have been warned, none of the other solutions produce a satisfactory result) under the MIT licence so anyone can reproduce. More pretty pictures are also available.

Matt Zimmerman: Scaling Human Systems: Roles and Responsibilities

This is part 5 in a series on organizational design and growth. What it means

Each of us has a job to do, probably more than one, and our teammates should know what they are. Why it s important

Roles are a kind of standing commitment we make to each other. They re a way of dividing up work which is easy to understand and simple to apply. Utilizing this tool will make it easier for us to coordinate our day to day work, and manage the changes and growth we re going through. Old Status Quo

Roles are vague or nonexistent. Management roles in particular are probably not well understood. Many people juggle multiple roles, all of which are implicit. Moving to a new team means learning from scratch what other people do. People take responsibility for tasks and decisions largely on a case-by-case basis, or based on implicit knowledge of what someone does (or doesn t do). In many cases, there is only one person in the company who knows how to perform a certain function. When someone leaves or goes on vacation, gaps are left behind. New Status Quo

Each individual has a clear understanding of the scope of their job. We have a handful of well defined roles, which are used by multiple teams and have written definitions. People who are new or transfer between teams have a relatively easy time understanding what the people around them are doing. Many day to day responsibilities are defined by roles, and more than one person can fill that role. When someone leaves a team, another person can cover their critical roles. Behaviors that help

Define project roles: when starting something new, make it explicit who will be working on it. Often this will be more than one person, often from different teams. For example, customer-facing product changes should have at least a product owner and an engineering owner. This makes it easy to tell if too many concurrent projects are dependent on a single person, which is a recipe for blockage.
Define team roles: Most recurring tasks should fall within a defined role. An owner of a technical service is an example of a role. An on-call engineer is an example of a time-limited role. There are many others which will depend on the team and its scope.
Define job roles: Have a conversation with your teammates and manager about what the scope of your job is, which responsibilities are shared with other members of the team and which are yours alone.

Obstacles that stand in our way

Getting hung up on titles as ego gratification. Roles are tools, not masters.
Fear that a role limits your options, locks you into doing one thing forever. Roles can be as flexible as we want them to be.

26 August 2013

Steve Kemp: Lack of referrers on github is an annoyance

Github is a nice site, and I routinely monitor a couple of projects there. I've also been using it to host a couple of my own projects, initially as an experiment, but since then because it has been useful to get followers and visibility. I'm a little disappointed that you don't get to see more data though; today my sysadmin utilities repository received several new "stars". Given that these all occurred "an hour ago" it seems likely that they we referenced in a comment somewhere on LWN, hacker news, or similar. Unfortunately I've no clue where that happened, or if it was a coincidence. I expect this is more of a concern for those users who use github-pages, where having access to the access.logs would be more useful still. But ..

24 August 2013

Marko Lalic: PTS rewrite: Django Memory Usage

Rewriting the PTS in a fairly large Web framework such as Django is sure to cause larger memory consumption by even the more simple tasks, when compared to a couple of (loosely related) perl scripts. If anything, simply due to the fact that the entire Django machinery needs to be loaded.

However, just how large the memory consumption difference between the two can be seen in the example of the command to dispatch received package emails to users. In the new version, the command is implemented as a Django management command, whereas before it was a perl script dispatch.pl relying on the common.pl script for some additional functions.

The reason this command's memory usage is important is an incident with the current test deployment of the new PTS found at http://pts.debian.net. All package mails that the old PTS receives are also forwarded to the new instance in order to expose the new implementation to real-world mail traffic. At a certain point, over a hundred mails were received instantly causing over a hundred dispatch processes to be launched. This led to the system running out of memory and the kernel had to run the OOM Killer which brought down PostgreSQL server thereby causing the site to become unavailable.

For now, such a problem seems to have been prevented from reoccurring by setting up exim to queue messages when a certain system average load is exceeded, however it is interesting to check how large the memory consumption difference between the old and new dispatch really is.

Measurement setupIn order to measure the memory use, two messages of different sizes were used: 1KB and 1MB.

For the new PTS two cases were considered, when the database used is sqlite3 and PostgreSQL. In both cases, it was simply initialized by syncdb, meaning it contained only the default keywords found in the initial fixture and no registered users and subscriptions. The DEBUG setting was set to False.

For testing the old PTS's dispatch.pl, the database was also initialized empty.

This way, the tests should show the actual difference between the two implementations.

Since the main concern here is the maximum memory usage of the dispatch script, a simple bash script was written which takes a PID of a running process and outputs the maximum memory usage as reported by ps once the process is terminated. The script used is shown below.

#!/usr/bin/env bash

pid=$1

while ps $pid >/dev/null
do
    ps -o vsz= $ pid 
    sleep 0.1
done   sort -n   tail -n1

Old PTS Measurements
The maximum memory usage of the old PTS was:
37.8 MB for the 1 KB message
38.2 MB for the 1 MB message

New PTS Measurements

sqlite3
The maximum memory usage of the new PTS using sqlite3 was:
91.77 MB for the 1 KB message
92.59 MB for the 1 MB message
PostgreSQL
The maximum memory usage of the new PTS using PostgreSQL was:
148 MB for the 1 KB message
160 MB for the 1 MB message

Comparison and DiscussionWhen comparing the memory used by the old dispatch.pl implementation and the new PTS running sqlite3, the difference does not seem to be too large and is to be expected, since, as mentioned, the whole Django framework is loaded when executing a management command.

However, the huge difference between the two versions of the new PTS: the one using sqlite3 and the one using PostgreSQL is very surprising indeed.

Another interesting measurement is the difference between running a bare Django management command: one including only a sleep statement (in order to allow enough time to measure its memory usage) on PostgreSQL and sqlite3. In both cases, the memory usage was only about 1 MB less than what the respective maximum memory usage when processing the 1 KB message.

All this considered, the logical conclusion seems to be that either psycopg2 package and/or the Django postgresql_psycopg2 database engine use a lot more memory than the corresponding sqlite3 alternatives.

Could anyone shed some more light as to what causes this stark difference between using sqlite3 and Postgres in Django? Anything that could be done to mitigate it?

Marko Lalic: PTS Rewrite Project Status

The rewrite has already achieved some of the major goals which have been set in the beginning, as well as made some changes and improvements along the way. Some of the major points of the new system implemented so far will be presented here.

Tasks framework
The current PTS implementation regenerates the HTML pages for all packages a few times per day. A goal of the rewrite was to make smaller incremental updates possible so that the information displayed for each package can be much more dynamic.

In order to allow incremental updates, it is necessary to consider the situation where updating some information should trigger an update of other information.

This is where the tasks framework comes in. It allows developers to define a task with a list of "events" that it produces. An event can be triggered by any change to shared information or anything else the task would like to signal. Each event is identified by a simple unique string name. Events can also contain arbitrary (JSON-serializable) objects. Other than events that it produces, a task defines a list of dependencies events it depends on.

When a single task should be executed, the tasks framework uses this information to build a dependency graph (a DAG) and executes the tasks in topological sort order, making sure each task runs only after all tasks which could raise an event it depends on have finished. It automatically makes the raised events available to the task so that it can access the extra data. Tasks for which no event was raised are skipped.

Such execution of a task along with all its dependencies is called a Job. A Job's state is persisted to the database after each task completes its operation. This allows Jobs to be continued if there is an unexpected error causing it to terminate.

So far, the tasks framework has already proven useful. For example, there are tasks which extract some package information to a denormalized model suited for directly displaying the data in the package page after a new package version is detected in a repository. Other examples are tasks which update standards-version warnings (after a new package version is detected) and the news generation task based on various detected changes to the packages (new source version, version removed from repository, version migrated to another repository).
Vendor-specific customizationAnother goal of the rewrite was to make the PTS less Debian-specific. This is so that derivatives could easily deploy their own versions of the PTS. The idea is to have a number of core PTS services which alternative deployments can customize and augment.

The simplest way vendors can customize PTS is by changing some of the local settings values. An example of this would be changing the PTS_VENDOR_NAME which leads to this name being displayed in any part of the system.

This is fairly uninteresting since the information is always static. What is more interesting is when the vendor should provide different information based on some context, e.g. package name, received message, etc. This requirement led to the concept of the vendor app with a rules module.

Various core parts of the PTS specify hook functions which different vendors can implement to provide customized behavior of the core component. Some examples are:
providing URLs to a developer information site (e.g. http://qa.debian.org/developer.php in Debian's case) based on a developer's email
providing a URL to a bug tracker for a particular package (based on the package type and bug category type)
providing additional rules to tag received package mails with keywords
providing additional headers to be injected in the forwarded package mails
providing a list of bugs which should be displayed in the bugs panel
A settings value is set giving the full dotted path to the Python module containing the implementation of these hook functions.
Vendor-specific featuresA step beyond customizing core features is for vendors to include their own specific functionality. The already described tasks framework allows for implementing custom data update/generation processes (optionally depending on core events). Furthermore, since a vendor-specific Django app is expected, it can contain any vendor-specific models which are managed just as the core models are (syncdb, migrations, etc. all work as expected). The last piece of the puzzle is displaying this information on the package page.

Currently, all package information is provided in various boxes (general, bugs, links, etc.) which were named panels. The rewrite has made it possible to implement additional panels in a really elegant way.

It is required to implement a class with some properties giving:
the panel's title
position in the page (left, center, right)
panel importance (higher importance panels are placed above lower importance ones in the same column)
To generate the HTML for a package, two alternatives are allowed. The first is to provide a property which gives (explicitly marked safe) HTML output which is then included in the correct position verbatim. The second alternative, which is what should cover most needs and should be preferred, is to provide a property giving a template name and a property giving the additional context variables necessary to render this template. By using the power of Python properties, the context can be dynamically generated for each package by the panel. The PTS then makes sure to pass the correct panel's context when rendering its template and to include the rendered result in the final package page.

Various convenience classes are implemented in the panel API, but those are better left to be read about in the panels API documentation.
Changes from current PTSSome existing features of PTS were not simply rewritten to fit into a Django project, but also improved upon.

One of the first such changes was reducing the number of confirmation mails sent by the email control bot to only one per received control mail (as per a wishlist PTS bug).

Work started by Markus Wanner (see this bug) was expanded on and integrated with the new News model so that any inline GPG signatures found in the content of a news item are automatically extracted and the signers name displayed in the package page.

The existing TODO and Probems panels have been merged into a single Action Needed panel. However, besides merging their representation in the package page, all the package issues are now represented as an ActionItem Django model which provides additional meta-data (item type, date created/updated, severity level) which could be used in the future to provide additional functionality (such as all packages with a certain issue type).

The descriptions of the action items were also shortened from their current rendition, with the detailed description being moved to a dedicated item page. If Javascript is enabled in the user's browser, it can also be displayed in a popup.
GUIThe graphical interface has seen some changes, however I hesitate to claim improvements. In my opinion, some of it is definitely better, more polished and more "modern" (for what it's worth).

Still, this could benefit from more people getting involved in testing and providing feedback so that the design is polished to suit the largest number of users. For example, the padding between the panels is increased, which to me definitely looks better, however, it is possible that this introduces too much whitespace and reduces the information density below what some may expect.
RecapOverall, I am quite pleased with where the project is currently. Most of the information provided by the current PTS has been reimplemented, with only a few smaller ones remaining. For now, since we are nearly entering the final month of GSoC, the development will switch focus from porting old features to the new PTS to adding in some new ones. Namely, user registration and account management is in the pipeline for the coming week. It entails features such as managing package subscriptions via the online interface (for multiple emails associated with the account) and sso.debian.org integration for Debian Developers.

This post aimed to provide the most notable changes and improvements made so far to the PTS. In reality, there has been quite a lot of them and not everything is covered here. For more information, check out the PTS documentation which is updated weekly (corresponding to the weekly deployments) or the source code repository.

The latest iteration of the PTS can always be found at http://pts.debian.net. The mail control interface bot responds to mails sent to control@pts.debian.net.

Stay tuned for updates and be sure to check out pts.debian.net on at least a weekly basis to see new deployments in action!

23 August 2013

Justus Winter: No noweb anymore...

... which is probably a good thing. But here is the boot log you all have been waiting for:
start ext2fs: Hurd server bootstrap: ext2fs[device:hd0s1] exec init proc auth INIT: version 2.88 booting Using makefile-style concurrent boot in runlevel S. Activating swap...done. Checking root file system...fsck from util-linux 2.20.1 hd2 : tray open or drive not ready hd2 : tray open or drive not ready hd2 : tray open or drive not ready hd2 : tray open or drive not ready end_request: I/O error, dev 02:00, sector 0 /dev/hd0s1: clean, 44693/181056 files, 291766/723200 blocks done. Activating lvm and md swap...(default pager): Already paging to partition hd0s5! done. Checking file systems...fsck from util-linux 2.20.1 hd2 : tray open or drive not ready hd2 : tray open or drive not ready end_request: I/O error, dev 02:00, sector 0 done. Cleaning up temporary files... /tmp. Mounting local filesystems...done. Activating swapfile swap...(default pager): Already paging to partition hd0s5! done. df: Warning: cannot read table of mounted file systems: No such file or directory Cleaning up temporary files.... Configuring network interfaces...Internet Systems Consortium DHCP Client 4.2.2 Copyright 2004-2011 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ Listening on Socket//dev/eth0 Sending on Socket//dev/eth0 *** stack smashing detected ***: dhclient terminated Aborted Failed to bring up /dev/eth0. done. Cleaning up temporary files.... Setting up X socket directories... /tmp/.X11-unix /tmp/.ICE-unix. INIT: Entering runlevel: 2 Using makefile-style concurrent boot in runlevel 2. Starting enhanced syslogd: rsyslogd. Starting deferred execution scheduler: atd. Starting periodic command scheduler: cron. Starting system message bus: dbusFailed to set socket option"/var/run/dbus/system_bus_socket": Protocol not available. Starting OpenBSD Secure Shell server: sshd. unexpected ACK from keyboard GNU 0.3 (debian) (console) login: root [...] root@debian:~# ifup /dev/eth0 Internet Systems Consortium DHCP Client 4.2.2 Copyright 2004-2011 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ Listening on Socket//dev/eth0 Sending on Socket//dev/eth0 *** stack smashing detected ***: dhclient terminated Aborted Failed to bring up /dev/eth0. root@debian:~# dhclient -v -pf /run/dhclient.-dev-eth0.pid -lf /var/lib/dhcp/dhclient.-dev-eth0.leases /dev/eth0 Internet Systems Consortium DHCP Client 4.2.2 Copyright 2004-2011 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ Listening on Socket//dev/eth0 Sending on Socket//dev/eth0 *** stack smashing detected ***: dhclient terminated Aborted root@debian:~# dhclient -pf /run/dhclient.-dev-eth0.pid -lf /var/lib/dhcp/dhclient.-dev-eth0.leases /dev/eth0 root@debian:~# ifup /dev/eth0 Internet Systems Consortium DHCP Client 4.2.2 Copyright 2004-2011 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ Listening on Socket//dev/eth0 Sending on Socket//dev/eth0 DHCPREQUEST on /dev/eth0 to 255.255.255.255 port 67 DHCPACK from 10.0.2.2 bound to 10.0.2.15 -- renewal in 34108 seconds. ps: comm: Unknown format spec root@debian:~# halt Broadcast message from root@debian (console) (Fri Aug 23 19:42:19 2013): The system is going down for system halt NOW! INIT: Switching to runlevel: 0root@debian:~# INIT: Sending processes the TERM signal INIT: Sending processes the KILL signal Using makefile-style concurrent boot in runlevel 0. Stopping deferred execution scheduler: atd. task c10f53f8 deallocating an invalid port 2098928, most probably a bug. Asking all remaining processes to terminate...done. All processes ended within 1 seconds...done. Stopping enhanced syslogd: rsyslogd. Deconfiguring network interfaces...Internet Systems Consortium DHCP Client 4.2.2 Copyright 2004-2011 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ Listening on Socket//dev/eth0 Sending on Socket//dev/eth0 DHCPRELEASE on /dev/eth0 to 10.0.2.2 port 67 /dev/eth0 (2): inet address 0.0.0.0 netmask 255.255.255.0 broadcast 10.0.2.255 flags BROADCAST ALLMULTI MULTICAST mtu 1500 done. Deactivating swap...swapoff: /dev/hd0s5: 177152k swap space done. Unmounting weak filesystems...umount: /etc/mtab: Warning: duplicate entry for device /dev/hd0s1 (/servers/socket/26) umount: /etc/mtab: Warning: duplicate entry for device /dev/hd0s1 (/dev/cons) umount: could not find entry for: /dev/cons umount: could not find entry for: /servers/socket/26 done. mount: cannot remount /: Device or resource busy Will now halt. store a new irq 11init: notifying pfinet of shutdown...init: notifying tmpfs swap of shutdown...init: notifying tmpfs swap of shutdown...init: notifying tmpfs swap of shutdown...init: notifying ext2fs device:hd0s1 of shutdown...init: halting Mach (flags 0x8)... In tight loop: hit ctl-alt-del to reboot
With some tiny patches for ifupdown I've been able to resolve network related issues. All of them? Of course not, funny thing about developing for the Hurd is that once you fix one thing, then some other thing or code path is executed that has never been run on Hurd before, and therefore something else breaks. In this case I fixed ifupdown to generate valid names for the pid file and leases file and all of the sudden dhclient starts dying. Funny thing about that is, if one drops the -v flag from the dhclient invocation as I did it above, the crash isn't triggered and once the lease file has been successfully written, it is safe to add the -v flag again. Not yet sure what goes on there, then again, looking at the source of isc-dhcp-client it is not so surprising that it crashes :/ When I first looked at ifupdown it was written in noweb, a literate programming tool. It is an interesting idea, even more so since (classic) c can be very verbose and cryptic. But it decouples the control flow from the structure of the program, which makes patching it quite a challenge since it is not as obvious where the changes have to go in. This is how ifupdown looked some weeks ago:
% wc --lines ifupdown.nw 6123 ifupdown.nw % pdftk ifupdown.pdf dump_data grep NumberOfPages NumberOfPages: 113
The ifupdown.nw is the noweb source, from which seven .c, four .h, two .pl and one Makefile are generated. It also contains a redicioulus amount of documentation, to the point that the authors at several points did not now what to write and just drop some nonsensical lines into the file. The source also compiles to a 113 page pdf file, that contains all of the documentation and all of the code, not at all in the order that one would expect a program to be written, but in the order the authors chose to structure the documentation. Fortunately for me the maintainer decided to drop the noweb source and to add the generated files to the source control system. This made my job much easier :) So here are the patches I published this week:
Fixes for ifupdown: http://lists.gnu.org/archive/html/bug-hurd/2013-08/msg00063.html

Fixes for fakeroot-hurd and remap: http://lists.gnu.org/archive/html/bug-hurd/2013-08/msg00064.html

I must admit that I do not know exactly what I will do next week. Obviously fixing the dhclient crash would be nice, I'll look into that. But I'm surely find some useful thing to do.

20 August 2013

Erich Schubert: Google turning evil

I used to be a fan of the Google company. I used to think of this as a future career opportunity, as I've received a number of invitations to apply for a job with them.

But I am no longer a fan.

Recently, the company has in my opinion turned to the "evil" side; they probably became worse than Microsoft ever was and the "do no evil" principle is long gone.

So what has changed? In my perception, Google:

Is much more focused on making money now, than making technology happen.

Instead of making cool technology, it tries more and more to be a "hipster" thing and following the classic old 80-20 rule: get 80% of the users with 20% of the effort. Think of the GMail changes that received so much hate and the various services it shuts down (Code search, Reader, Googlecode downloads) and how much Google is becoming a walled garden.

Google leverages its search market dominance as well as Android to push their products that don't perform as good as desired. Plus for example. There is a lot of things I like about plus. In particular when you use it as a "blog" type of conversation tool instead of a site for sharing your private data, then it's much better than Facebook because of the search function and communities. (And I definitely wouldn't cry if a Facebook successor emerges). But what I hate is how Google tries hard to leverage their Android and YouTube power to force people to Plus. And now they are spreading even further, into TV (ChromeCast) and ISP (Google Fiber) markets.

Hangups, oops, Hangouts is another such example. I've been using XMPP for a long time. At some point, Google started operating an XMPP server, and it would actually "federate" with other servers, and it worked quite nicely. But at some point they decided they need to have more "hipster" features, like ponies. Or more likely, they decided that they want all these users to use Plus. So now they are moving it away from an open standard towards a walled garden. But since 90% of my use is XMPP outside of Google, I will instead just move away from them. Worse.

Reader. They probably shut this down to support Currents and on the long run move people over to Plus, too. Fortunately here, there now exist a number of good alternatives such as Feedly. But again: Google failed my needs.

Maps. Again an example where they moved from "works good" to "hipster crap". The first thing the new maps always greets me with is a gray screen. The new 3D view looks like an earthquake happened, without offering any actual benefit over the classic (and fast) satellite imagery. Worse. In fact my favorite map service right now is an OpenStreetmap offline map.

Google Glass is pure hipsterdom crap. I have yet to see an actual use case for it. People use it to show off, and that's about it. If you are serious about photos, you use a DSLR camera with a large objective and sensor. But face it: it's just distraction. If you want productive, it's actually best to turn of all chat and email and focus. Best productivity tip ever: Turn off email notifications. And turn of chat, always-on and Glass, too.

Privacy. When privacy-aware US providers actually recommend against anyone trusting their private data to a company with physical ties to the United States, then maybe it's time to look out for services in countries that value freedom higher than spying. I cannot trust Google on keeping my data private.

We are living in worrisome times. The U.S.A., once proud defenders of freedom, have become the most systematic spies in the world, even on their own people. The surveillance in the Eastern Bloc is no match for this machinery built the last decade. Say hello to Nineteen Eighty-Four. Surveillance, in a mixture of multi-state and commercial surveillance has indeed become omnipresent. Wireless sensors in trash cans track your device MACs. Your email is automatically scanned both by the NSA and Google. Your social friendships are tracked by Facebook and your iPhone.

I'm not being paranoid. These things are real, and it can only be explained with a "brainwashing" with both the constantly raised fear of terror (note that you are much more likely to be killed in a traffic accident or by a random gun crazy in the U.S.A. than by terrorists) and the constant advertisement for useless technology such as Google Glass. Why have so many people stopped fighting for their freedom?

So what now? I will look into options to move away stuff from Google (that is mostly, my email -- although I'd really like to see a successor to email finally emerge). Maybe I can find a number of good services located e.g. in Switzerland or Norway (who still seem to hold freedom in high respect - neither the UK nor Germany are an alternative these days). And I hope that some politicians will have the courage to openly discuss whether it may be necessary to break up Google into "Boogles" (Baby-Googles), just as Reagan did with AT&T. But unfortunately, todays politicians are really bad at such decisions, in particular when it might lower their short-run popularity. They are only good at making excuses.

17 August 2013

Daniel Pocock: DebConf13: breakthroughs in OTP and WebRTC

Saturday, 17 August was a day that saw groundbreaking new work revealed at DebConf13 and may have been a watershed moment in the emergence of WebRTC for the free software community.
Fabian Gr nbichler, Daniel Pocock, Catalin Constantin Usurelu, Emil Ivov, Yana Stamcheva Advancing the use of One Time Passwords with free software Google Summer of Code (GSoC) student Fabian Gr nbichler has spent the summer enhancing oath-toolkit and dynalogin to support a more diverse range of One Time Password schemes. dynalogin's original 1.0 release supports vanilla HOTP and TOTP authentication schemes. Fabian's work has extended the framework to support the more generic and powerful OCRA approach. To find out about all the benefits this brings, I would strongly encourage you to review his slides and video presentation, with a demo where it is explained thoroughly. While there are existing hardware tokens that support HOTP and TOTP using this open algorithm (Gooze has been kind enough to donate some which were on display), the mutual challenge-response schemes are yet to be widely supported in hardware and this means that Fabian's work is now at the cutting edge of development in this area. WebRTC: premium voice/video chat in modern browsers, no plugin required Another member of Debian's GSoC program, Catalin Constantin Usurelu helped organise and run one of the must-see sessions of DebConf13 (video available). In a demonstration that had all the hallmarks of a magic show, Catalin and I took random members of the audience and invited them to wave a magic wand over their web browsers and effortlessly turn them into Skype-busting softphones. As the grand finale, we took Wookey's blog and showed him how to add a WebRTC "call-me" page in less than 20 seconds, less time than it takes to hand over all your private data to the registration process of those rather uncomfortable alternatives. Anybody else seeking to duplicate this effort is encourage to review options like Drucall for Drupal or just directly use JsSIP Jitsi and the Jitsi Video-bridge The grand-finale of the DebConf13 real-time communications track was our guest speaker, Emil Ivov, founder of the Jitsi softphone project. Jitsi itself has recently been accepted into Debian and is easier than ever before to install and use. Emil's talk (video and slides available) went into some detail about the cutting edge development activities of his team, particularly the new Jitsi Video bridge Subsequent to the talk, we had some out-of-bands discussion about ways to potentially integrate Jitsi Video bridge with the open source dvswitch system used to produce streams and recordings from free software events such as DebConf and FOSDEM. It is hoped that this will provide an additional option for remote participation at free software events of all sizes. Upcoming events Emil and I have been invited to participate in the WebRTC Conference Paris 2013 in December. This will be an excellent opportunity to have further discussion about progress with WebRTC and do more to raise the compelling case for premium quality free software to meet the demanding needs of modern business communications. There are a series of talks for both technical and business-oriented audiences.

16 August 2013

Sylvain Le Gall: OASIS website updated

The OASIS website has not been updated since a while. So I decide to take a shot at making more up to date. This blog post is about the pipeline I have but in place to automatically update the website. It is the first end to end 'continuous deployment' project I have achieved. Among the user visible changes:

an invitation to circle the OASIS G+ page, which is now the official channel for small updates in OASIS.

an invitation to fork the project on Github since it is now the official repository for OASIS.

some link to documentation for the bleeding edge version of OASIS manual and API.

The OASIS website repository is also on Github. Feel free to fork it and send me pull request if you see any mistake. The website is still using a lot of markdown processed by pandoc. But there are some new technical features:

no more index.php, we use a templating system to point to latest version

we use a Jenkins job to generate the website daily or after any update in OASIS itself.

Since I start using quite quite a lot Python, I have decided to use it for this project. It has a lot of nice libraries and it helps me to quickly do something about the website (and provides plenty of idea to create equivalent tools in OCaml). The daily generation: Jenkins I have a Jenkins instance running, so I decided to use it to compile once a day the new website with updated documentation and links. This Jenkins instance also monitor changes of the OASIS source code. So I can do something even more precise: regenerate the website after every OASIS changes. I use the Jenkins instance to also generate a documentation tarball for OASIS manual and API. This helps a lot to be able to display the latest manual and API documentation. This way I can quickly browse the documentation and spot errors early. Another good point about Jenkins, is that it allows to store SSH credential. So I created a build user, with its own SSH key, in the OCaml Forge and I use it to publish the website at the end of the build. Right now Jenkins do the following:

trigger a build of the OASIS website:

every day (cron)

when a push in OASIS website repository is detected

when a successful build of OASIS is achieved.

get documentation artifact from the latest successful build of OASIS

build the website

publish it

Data gathering To build the website I need some data:

documentation tarballs containing the API (HTML from OCamldoc) and manual (Markdow)

list of OASIS version published

links to each tarball (documentation and source tarball)

The OCaml Forge has a nice SOAP API. But one need to be logged in to access it. This is unfortunate, because I just want to access public data. The only way I found to gather my data was to scrape the OCaml Forge. Python has a very nice scraping library for that: beautifulsoup. I use beautifulsoup to parse the HTML downloaded from the Files tab of the OASIS project and extract all the relevant information. I use curl to download the documentation tarball (for released versions) and for the latest development version. Code Template Python has also a very nice library to process template: mako. Using the data I have gathered, I feed them to mako and I process all the .tmpl files in the repository to create matching files. Among the thing that I have transformed into template:

the index.php has been transformed into a index.mkd.tmpl, it was a hackish PHP script scraping the RSS of updates before, it is now a clean template.

robots.txt.tmpl, see the following section for explanation

documentation.mkd.tmpl in order to list all version of documentation.

Fix documentation and indexing One of the problem of providing access to all versions of the documentation, is that people can end up reading an old version of the documentation. In order to prevent that, I use two different techniques:

prevent search engine to index old version.

warn the user that he is reading an old version.

To prevent search engine to index the file, I have created a robots.txt that list all URL of old documentation. This should be enough to prevent search engine to index the wrong page. To warn the user that he is reading the wrong version, I have added a box "you are not viewing the latest version". This part was tricky but beautifulsoup v4 provide a nice API to edit HTML in place. I just have to find the right CSS selector to define the position where I want to insert my warning box. Code Publish The ultimate goal of the project is the 'continuous deployment'. Rather than picking what version to deploy and do the process by hand, I let Jenkins deploy every version of it. Deploying the website used to be a simple rsync. But for this project I decided to use a fancier method. I spend a few hours deciding what was the best framework to do the automatic deployment. There are two main frameworks around: capistrano (Ruby) and fabric (Python). Fabric is written in Python, so I pick this one because it was a good fit for the project. Fabric biggest feature is to be a SSH wrapper. The fabric script is quite simple and to understand it, you just have to know that local run a local command and run run a command on the target host. The fabfile.py script do the following:

create a local tarball using the OASIS website html/ directory

upload the tarball to ssh.ocamlcore.org

uncompress it and replace the htdocs/ directory of the oasis project

remove the oldest tarballs, we keep a few versions to be able to perform a rollback.

Given this new process, the website is updated in 3 minutes automatically after a successful build of OASIS.

Justus Winter: All the important bits are there - please test and review :)

Finally, more bootlog-pr0n:
start ext2fs: Hurd server bootstrap: ext2fs[device:hd0s1] exec init proc auth INIT: version 2.88 booting Using makefile-style concurrent boot in runlevel S. Activating swap...done. Checking root file system...fsck from util-linux 2.20.1 hd2 : tray open or drive not ready hd2 : tray open or drive not ready hd2 : tray open or drive not ready hd2 : tray open or drive not ready end_request: I/O error, dev 02:00, sector 0 /dev/hd0s1: clean, 44680/181056 files, 292234/723200 blocks done. Activating lvm and md swap...(default pager): Already paging to partition hd0s5! done. Checking file systems...fsck from util-linux 2.20.1 hd2 : tray open or drive not ready hd2 : tray open or drive not ready end_request: I/O error, dev 02:00, sector 0 done. Cleaning up temporary files... /tmp. Mounting local filesystems...done. Activating swapfile swap...(default pager): Already paging to partition hd0s5! done. df: Warning: cannot read table of mounted file systems: No such file or directory Cleaning up temporary files.... Configuring network interfaces...inetutils-ifconfig: invalid arguments ifup: failed to open pid file /run/network/ifup-/dev/eth0.pid: No such file or directory Internet Systems Consortium DHCP Client 4.2.2 Copyright 2004-2011 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ can't create /var/lib/dhcp/dhclient./dev/eth0.leases: No such file or directory Listening on Socket//dev/eth0 Sending on Socket//dev/eth0 DHCPDISCOVER on /dev/eth0 to 255.255.255.255 port 67 interval 8 DHCPREQUEST on /dev/eth0 to 255.255.255.255 port 67 DHCPOFFER from 10.0.2.2 DHCPACK from 10.0.2.2 can't create /var/lib/dhcp/dhclient./dev/eth0.leases: No such file or directory bound to 10.0.2.15 -- renewal in 34744 seconds. done. Cleaning up temporary files.... Setting up X socket directories... /tmp/.X11-unix /tmp/.ICE-unix. INIT: Entering runlevel: 2 Using makefile-style concurrent boot in runlevel 2. Starting enhanced syslogd: rsyslogd. Starting deferred execution scheduler: atd. Starting periodic command scheduler: cron. Starting system message bus: dbusFailed to set socket option"/var/run/dbus/system_bus_socket": Protocol not available. Starting OpenBSD Secure Shell server: sshd. unexpected ACK from keyboard GNU 0.3 (debian) (console) login: root root@debian:~# shutdown -h now Broadcast message from root@debian (console) (Fri Aug 16 20:02:47 2013): The system is going down for system halt NOW! INIT: Switching to runlevel: 0root@debian:~# INIT: Sending processes the TERM signal INIT: Sending processes the KILL signal Using makefile-style concurrent boot in runlevel 0. Stopping deferred execution scheduler: atd. task c10f72a8 deallocating an invalid port 2098928, most probably a bug. Asking all remaining processes to terminate...done. All processes ended within 1 seconds...done. Stopping enhanced syslogd: rsyslogd. Deconfiguring network interfaces...ifdown: failed to open pid file /run/network/ifdown-/dev/eth0.pid: No such file or directory Internet Systems Consortium DHCP Client 4.2.2 Copyright 2004-2011 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ can't create /var/lib/dhcp/dhclient./dev/eth0.leases: No such file or directory Listening on Socket//dev/eth0 Sending on Socket//dev/eth0 /bin/sh: 1: ifconfig: not found done. Deactivating swap...swapoff: /dev/hd0s5: 177152k swap space done. Unmounting weak filesystems...umount: /etc/mtab: Warning: duplicate entry for device /dev/hd0s1 (/dev/cons) umount: could not find entry for: /dev/cons done. Unmounting local filesystems...done. mount: cannot remount /: Device or resource busy Will now halt. init: notifying pfinet of shutdown...init: notifying tmpfs swap of shutdown...init: notifying tmpfs swap of shutdown...init: notifying tmpfs swap of shutdown...init: notifying ext2fs device:hd0s1 of shutdown...init: halting Mach (flags 0x8)... In tight loop: hit ctl-alt-del to reboot
(You might note that df complains about not being able to read the mtab file. That is because it has been built with _PATH_MOUNTED being /var/run/mtab. This will correct itself when the coreutils package is being rebuilt against a patched libc.) I spent my last two weeks with polishing my patch series, that meant a lot of package rebuilds and that means a lot of waiting (even more so on the Hurd, short version: fakeroot-hurd could be fast but is not yet working properly, fakeroot-tcp is slow) and thus some frustration ;) Also I had to pay special attention so that the upgraded packages could be installed without accidentally breaking anything in the process. Making sysvinit pid 1 is surprisingly tricky in this regard since it breaks the ABI and requires a libc fix that also worked with the current Hurd servers. Here are the patches:
The mtab translator (v4) provides /proc/mounts: http://lists.gnu.org/archive/html/bug-hurd/2013-07/msg00259.html

When I wrote umount I thought I implemented all the flags used by the initscripts package. Turns out I forgot -d. umount.patch (v2): http://lists.gnu.org/archive/html/bug-hurd/2013-08/msg00018.html

Make sysvinit pid 1 (sysvinit kinda requires that), fix killall5 (that was freezing the system before) (v2): http://lists.gnu.org/archive/html/bug-hurd/2013-08/msg00020.html

Making sysvinit pid 1 breaks the ABI, since the glibc assumes /hurd/init to have pid 1: http://lists.gnu.org/archive/html/bug-hurd/2013-08/msg00046.html http://sourceware.org/ml/libc-alpha/2013-08/msg00205.html

So now killall5 is not freezing the system anymore (actually it froze all processes but most importantly the procfs translator providing /proc and it then tried to walk over /proc which obviously blocked killall5 itself). Cool. But then it looks into /proc/*/stat for the fields start_code and end_code. So I dug into the exec server where the ELF decoding happens and handed this to the proc server, but the procfs translator has to make use of this: http://lists.gnu.org/archive/html/bug-hurd/2013-08/msg00017.html

Patch series for the sysvinit package (v2). This contains a runsystem.sysv that can be enabled using update-alternatives --config runsystem and several small fixes: http://lists.gnu.org/archive/html/bug-hurd/2013-08/msg00038.html

So I had some spare time on my hand while waiting for numerous package rebuilds and I took this as an opportunity to read papers about Mach and to familiarize myself with mig, the Mach Interface Generator. While I have used it in the past, I had not yet looked at its implementation. And I had to patch the exec server, and there was both code implementing a questionable feature (on-demand unzipping of binaries) and code that was not even compiled (courtesy of the preprocessor) and had probably bit-rot by now. So I figured I could spend my time doing some cleanups:
Cleanups for the exec server: http://lists.gnu.org/archive/html/bug-hurd/2013-08/msg00049.html

Cleanups for mig: http://lists.gnu.org/archive/html/bug-hurd/2013-08/msg00055.html

I have rebuild all the necessary packages and uploaded them into an apt repository:
deb http://teythoon.cryptobitch.de/gsoc/heap/debian unstable main
Please use unstable for now. Also make sure that you have a recovery plan for your Debian/Hurd installation if anything goes wrong. For your convenience there's a seed tarball containing packages with the appropriate sources.list.d snippets and the repository key: https://teythoon.cryptobitch.de/gsoc/heap/debian/seed.tar If you want to switch to the new runsystem.sysv, do:
# update-alternatives --config runsystem
Whenever you switch runsystems, please use reboot-hurd to reboot the system. This is the most robust way. Known issues:
procfs hardcodes the default kernel pid to 2. This breaks /proc/uptime and any program relying on it, most notably top and friends. Until this is properly fixed, you can do:
# settrans -apfg /proc /hurd/procfs -c -k 3

The mtab translator should probably try to filter out non-filesystem translators. df complains loudly about /dev/cons for example.

Next week I will address the network related issues. By now they are the source of most of the error messages in the bootlog.

13 August 2013

Russell Coker: Nexus 4 Ringke Fusion Case

I ve been using Android phones for 2.5 years and for all my phones (Sony-Ericsson Xperia X10, Samsung Galaxy S, Samsung Galaxy S3, and LG Nexus 4) I ve bought gel cases. The gel cases are made of soft rubber which allows phones to bounce rather than break if dropped and also provide better grip (both for holding the phone and for resting it on a vibrating surface such as a tray in a car or plane). The cases have cost me between $12.50 and $15, I haven t felt inclined to pay any more because a case has always seemed like a needless expense, something that merely compensates for deficiencies in phone design. There are some problems with the gel cases, the first one is that they are rather ugly, while I m not really interested in decorating my phones it seems a waste to have a nice phone like the Nexus 4 which has a neat sparkling pattern on the back and then use a case that entirely covers it. The next problem is the fact that the gel cases tend to restrict access to the buttons on the outside of the case. I have been given a Rearth Ringke Fusion case for my Nexus 4 by MobileZap (this is a link for the Nexus 4 cases they sell) [1]. The case features a transparent layer of plastic at the back which protects the phone against having the rear glass of the case break (a known risk with the Nexus 4) while still allowing the user to admire it. With good lighting the sparkling pattern on the back of the phone can still be enjoyed, unfortunately this doesn t show up at all in my photo. It s impossible for a photo to capture a shimmering effect although other photographers have done a better job at showing off a Nexus 4. Also the transparent back to the case gives the option of putting any form of decoration between the phone and the case, for example one could print the logo of their Ingress team and put it between the phone and the case. One thing that is visible in the photo is the feet on the back of the case. With small feet at the corners the transparent back usually won t touch a surface that it rests on, this should prevent unsightly scratches to the back of the case. Also the feet can get some grip on the surface that it rests on, my car has a tray suitable for resting a phone which has a pattern of dimples that catch the feet of the Ringke case and prevent it from sliding. Finally the external buttons for power and volume control are not obstructed in any way. After using 7 phones of 4 different models with gel cases I ve become accustomed to buttons either being difficult to press or easy to press accidentally. The Ringke case allows the buttons to be pressed easily when needed but doesn t seem prone to accidental presses. I m not about to buy a new case for my Galaxy Note 2 (my main phone at the moment) because I don t like paying to replace something that works. But next time I buy a case I ll look for something better than a plain gel case, it s worth paying an extra $9 to get a good case for a phone that costs $350 (the current discount rate for a Nexus 4) or more. I definitely recommend the Ringke case for a Nexus 4.

[1] http://www.mobilezap.com.au/32387/google/nexus-4/cases.htm

Related posts:

Nexus 4 My wife has had a LG Nexus 4 for about...

Samsung Galaxy S3 First Review with Power Case My new Samsung Galaxy S3 arrived a couple of days...

Cooling Phones According to the bureau of meteorology today is 39C. But...

12 August 2013

Chris Lamb: Cotswold Classic Middle Distance triathlon

Since becoming interested in triathlon in October of last year, my goal in 2013 was not to just finish a "70.3" (or "Half-Ironman"), but to complete one in under 5 hours. At the time, each discipline was woefully inadequate; I had not swum in over a decade, my cycling was average and I had only begun running again after years away from exercise. The somewhat arbitrary target time would dictate my training. For example, the high proportion of time spent on the bike combined with the injury-sensitive nature of running meant that cycling would have a highest return on investment of the three disciplines. Despite that, I would need to take the run seriously but carefully - I decided to enter a series of monthly 10K races where I could steadily reduce my times over the year. Races themselves formed a crucial part of my preparation and a carefully balanced race calendar was surprisingly important, balancing upcoming milestones that would provide an achievable goal as well as uncovering weaknesses in enough time so they could be worked on. Whilst I did post about my first triathlon, I felt no compulsion with the interleaving five or six races as I was evaluating them in the context of the 70.3 rather than as races in their own right.
Swim

Distance

1900m

Time

37:42

Whilst this wasn't my worst open water swim, it was certainly not my best. The first 500m was straightforward, but I lost my form over the next 500m and I overcompensated with excessive kick. My calves responded by cramping badly two or three times after that, losing both momentum and time. Curiously, whenever I could identify my speed relative to my effort (for example, by some underwater reeds) I could use that as feedback and my rotation (etc.) would temporarily return. Pool swimming always has markers of this kind, whilst a murky lake does not. In conclusion, my lack of open water training outside of races held me back more than I expected.

Bike

Distance

52 miles

Time

2:22 (21.9 mph average)

I had thoroughly absorbed the advice that you should prepare for your primary race so that it feels like another day at the office. On this criterion, the bike leg was almost perfect. My nutrition strategy energy drink every 5 minutes, gel every 30 minutes worked fine until mile 35 where I felt like I had taken on too much fluid (and no time to take a bathroom break). I had a temporary hiccup with my rear derailleur but my "tame" 52/34 12x28 gearing was vindicated on the climbs, although in retrospect that might mostly be schadenfreude. Tens of hours acclimatising to a time trial bike was rewarded by being able to spend the entire leg in "aero" position (ignoring 90-degree turns or serious gradients), improving efficiency.

Run

Distance

13.1 miles (21km)

Time

1:54:18 (5:25/km)

I had prepared a chart which provided the pace I required based on the race time at the start of the run. I would then only need to monitor my run pace instead of repeatedly extrapolating a finish time as even basic arithmetic can become impossible when fatigued. This strategy was chosen as I was too risk-adverse to aim for a faster time than strictly required; the downside of missing the target from aiming too high and "blowing up" outweighed going a few minutes faster overall. Despite that, I ran half-marathon personal best. Unfortunately, due to pressing the wrong watch button after the swim, the chart was worthless unless I could assume the race started exactly at 6:30AM. With no other option I decided to trust that with some padding, but the uncertainly it added was unsettling. Adding to that feeling, my left hip started to twinge at 7K, something it has never done before. My plan was to get to 16K, take a caffeine gel and then keep ratcheting up the pace until the finish. In my long run training I could speed up to a 4:10/km pace but after 5 hours of racing and being reasonably sure I was going to reach my goal, the best I could summon was 4:45/km. My form completely shot, I crossed the finish line and got into a fetal position in the shade behind a tent, scaring the race organisers for a few minutes.

Overall

Total time

4:56:50

Writing this the day after the race, I am still a little unsure of how I feel. I am obviously extremely content that I reached my goal but given the volume and detail of preparation it was not a huge surprise to me on race day that I did so. This combined with the "real" work and progress being achieved throughout the training seems to have robbed the event of some element of triumph it could have had, but that in itself was somewhat expected. Where next? I had always dismissed the consensus view of needing to take a decent break after the season has ended but now I am looking forward to putting triathlon-specific training on hold for a while and explore other things. I highly doubt this will be my last triathlon though. (Full results)

Tanguy Ortolo: grep --only-matching

grep is designed to print lines matching a given pattern, but I often need to print only the matching part, discarding the remaining. I used to do that with sed, but it involves several actions: match, replace the line by only the matching pattern and print. Fortunately, GNU grep has an option to do just that:

-o, --only-matching

Print only the matched (non-empty) parts of a matching line, with each such part on a separate output line.

Unfortunately it is not a standard option, so it may be missing on non-GNU systems.

11 August 2013

Michal Čihař: Good morning DebConf

Yesterday evening I've arrived at DebConf 13. I was quite tired after traveling so I just had few beers and went to bed. The travel went quite well besides somebody else broke wine and my suitcase was all wet from it. Fortunately almost nothing got inside so I have some clean clothes for next week :-). As I went to bed quite early, I woke up early as well so I went to short walk to see what is nearby:
Filed under: English Debian Photography 0 comments Flattr this!

9 August 2013

Justus Winter: My worst week yet...

This hasn't been my week. I had to fix two of my Hurd installations I use for development and testing. I spent countless hours waiting for some package to be rebuilt, reading scientific papers about adding stuff to the Mach kernel. Fascinating stuff like thread migration that would (among other things) allow for proper resource accounting on Mach kernels. Oh, and gcc is a sadistic bastard:
In file included from printf-parsewc.c:2:0: printf-parsemb.c: In function __parse_one_specwc : printf-parsemb.c:407:1: internal compiler error: Segmentation fault Please submit a full bug report, with preprocessed source if appropriate. See <file:///usr/share/doc/gcc-4.7/README.Bugs> for instructions. The bug is not reproducible, so it is likely a hardware or OS problem.
A similar error came up like five hours into my libc rebuild. How nice of gcc to retry the build to figure out whether it is to blame or the environment only to throw away the successful build and abort my build process :/
I published my patch series addressing all sysvinit-related issues: http://lists.gnu.org/archive/html/bug-hurd/2013-08/msg00000.html

I since found three minor issues, so I'll roll another series shortly. But since this somewhat breaks the ABI (/hurd/init is no longer PID 1) this is waiting for a libc rebuild that I haven't managed to get done so far.

I updated my patch series for the sysvinit package. I'm quite pleased with it as it is and I'll propose it for inclusion with the sysvinit package shortly. Unfortunately this depends on the patched hurd package with is blocked by the libc package :/

Next week I'll finally propose my sysvinit patch series. Oh, and implement the -d flag in our umount. I thought I had implemented all the flags used by the initscripts, but somehow I missed -d. If I get bored, I'll take a look at the network related issues, but I bet I won't get there...

7 August 2013

Keith Packard: embedded cpus

Choosing Embedded Processors for AltOS When Bdale and I started building rocketry hardware together, we had already picked out a target processor, the TI cc1111. We picked that chip almost entirely based on the digital transceiver that is built in to the chip, and not because we had any particular love for the 8051 microcontroller. At that time, I d seen people struggle with PIC processors, battle AVR to a draw and spend a lot of time trying to get various ARM processors running. So, the 8051 didn t seem all that far from normal, and the cc1111 implementation of it is pretty reasonable, including credible USB support and a built-in DMA engine. Since those early days, we ve gone on to build boards with a slightly wider range of processors:

TI cc1111

Atmel ATmega32U4

Atmel ATiny85

STMicroelectronics STM32L151/STM32L152

NXP LPC11U14

Bdale thinks we should reduce the number of components we use to focus our efforts better. He s probably right, but I have to admit that I ve had way too much fun getting each of these chips running. I thought I d spend a bit of time describing our general process for selecting a new CPU.

Free software. I m long past the age at which I trust hardware vendors to deliver credible binary software. If you want me to use your silicon, make sure the tools required to use it are all free software.

Debian support required. Bdale and I do everything on Debian, so any processor we use needs to have tools that run under Debian. We re more than willing to package software for Debian, so if it s just generic Linux-compatible free software, that s fine.

Built-in USB. Everything we build needs to talk to a computer for configuration and data download. Don t even think of asking me to attach another chip for USB support. Ok, so the ATtiny85 doesn t qualify here, and that project does use an external chip for USB which is connected to the ATtiny85 through an LED and phototransistor. I was excited when TI came out with the CC340 chips because the MSP430 is a much nicer CPU than the 8051. But, they still haven t added USB. If I m going to add another chip for USB, it ll be a CPU anyways, at which point I ll use a stand-alone RF chip.

Cheap developer boards. TI sells a tiny little cc1111 developer board; STmicroelectronics and NXP both sell boards that include a programmer and a target CPU for prototyping software. And, there are a million Arduino clones with various Atmel processors on them. Cheap means I can buy them without thinking too hard; having a built-in programmer means that early OS development can be done without first having to figure out how to wire up a programmer.

Probably 32 bits, probably an ARM. Yeah, the cc1111 and Atmel processors were fun to bring up, but I m pretty much over the 8-bit microcontroller now. ARM processors are far more sophisticated in terms of power management and built-in device capability, plus they re a ton faster at doing any math. The ATtiny85 remains an exception here; it s smaller than any ARM I can get, and so it s still useful for projects too small for an ARM.

Available in small quantities from a reasonable on-line retailer like Digikey, Mouser or even Avnet Express. I d love to try out the new low-cost STM32L100 parts, but the only one available is the largest one in the 64LQFP package, and I really don t need that many pins for most products.

CC1111 involved a lot of software hacking The 8051 processor in the CC1111 is very well documented, including the two-wire debugging interface. What was missing was a canned solution for programming and debugging the chip from Debian. However, with sufficient motivation and accurate docs, I was able to create a programmer from a USB device with a couple of GPIOs and a lot of ugly software on Linux. That sufficed to get the USB stack limping along, at which point I wrote a much faster programmer that ran on the cc1111 itself and could program another cc1111. With that, I created an assembly-level debugger that could be hooked to the existing SDCC source level debugger and had a full source-level debugging environment for the 8051. This turned out to be way better than what I could get on the Atmel processors, where I ve got a program loader and a whole lot of printf debugging. STM has been great The STM32L-Discovery board has a standard STM debugging setup for the Cortex SWD interface right on the same board as a target CPU. That made for completely self-contained development, with no jumper wires (a first for me, for sure). There s free software, stlink which can talk over the debugger USB connection to drive the SWD interface. This is sufficient to flash and debug applications using GDB. Of course, GCC supports ARM quite well; the only hard part was figuring out what to do for a C library. I settled on pdclib, which is at least easy to understand, if not highly optimized for ARM. We ve built a ton of boards with the STM32L151 and STM32L152; at this point I m pretty darn comfortable with the architecture and our tool chain. Adventures with NXP The NXP LPC11U14 is a very different beast. I m using this because:

It s small (5mm x 5mm)

It s cheap ($1.48 q100)

The LPCXpresso board looks much like the STM32L-Discovery, with a debugger interface wired to the CPU directly on the board. However, I haven t found free software tools to drive this programmer; all I ve found are binary-only tools from NXP. No thanks. Fortunately, the LPC11U14 uses exactly the same SWD interface as the STM32L, so I was able to sever the link between the programmer and the target on the LPCXpresso board and hook the target to an ST-Link device (either the one on the STM32L-Discovery board, or the new stand-along programming dongle I bought). With that, I wrote an openocd script to talk to the LPC11U14 and was in business. What I found in the NXP processor was a bit disturbing though there s a mask ROM that contains a boot loader, which always runs when the chip starts, and a bunch of utility code, including the only documented interface to programming the flash memory. I cannot fathom why anyone thought this was a good idea I don t want a BIOS in my embedded CPU, thankyouverymuch; I d really like my code to be the first instructions executed by the CPU. And, any embedded developer is more than capable of programming flash using a register specification, and calling some random embedded code in ROM from the middle of my operating system is more than a bit scary. NXP could do two simple things to make me like their parts a whole lot more:

Publish the programmer source code. The programming dongle is a tool for selling your silicon, not a separate revenue stream. Publish the source under a free software license (GPL recommended, of course). We ll take it, make it better, and ensure that you get the changes back to help your other customers. It s a win-win plan.

Document the flash programming registers. Yeah, I could probably disassemble the mask-rom bits that are on the chip, but I have better things to do with my time. Heck, go wild and document any other hidden bits in the silicon. Again, we re embedded developers, we don t like magic hardware hidden from view any more than PC developers like the BIOS and SMM.

Right now, I m hoping the STM32L100C6 parts become available in small quantities so I can try them out; they promise to be as cheap as the LPC11U14, but are better supported by free software and offer more complete hardware documentation. Yeah, they re a bit larger; that will probably be annoying.

5 August 2013

Wouter Verhelst: Hard disk partition pregap and grub2

A week or so back, I was at a customer to upgrade his server to wheezy. This server had originally been installed, not as squeeze but as something older, and was installed using an LVM-on-mdraid setup. Booting off such a setup (without a /boot partition outside of the raid/LVM) is certainly possible with several boot loaders, including grub2; however, it was having issues in this particular instance, producing an error message along the likes of:
/sbin/grub2-setup: warn: Your embedding area is unusually small. core.img won't fit in it..
It took me a while to figure out what that meant, but the Internet to the rescue:
Disk /dev/sda: 160.0 GB, 160041885696 bytes 255 heads, 63 sectors/track, 19457 cylinders, total 312581808 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x000b3c3b Device Boot Start End Blocks Id System /dev/sda1 * 2048 312580095 156289024 83 Linux
That partition starts at the 2048th block, resulting in a 1MiB gap before the first partition. The older default was to start on the 63rd block, for a pregap of just short of 32KiB. Unfortunately, the only way to fix this issue without ugly hacks (such as the chainloading suggested in this fedora forum thread) is to repartition. Luckily, thanks to the magic of LVM, this isn't difficult (if a little involved). Steps:
pvmove -v /dev/md0 vgreduce -a mdadm -S /dev/md0 (...repartition the hard disks that make up /dev/md0...) mdadm -C /dev/md0 (... other options required to recreate /dev/md0 on the desired RAID level...) pvcreate /dev/md0 vgextend /dev/vg /dev/md0
... and then rinse, repeat for md1. At that point, installing grub will work as usual. An alternative (since we were using RAID1) could have been to remove one disk from the RAID array, repartition it, create a new RAID1 array in degraded mode, pvmove over to that, destroy the original array, and add the second disk to the new array. This would probably be a better idea when you only have one RAID array, for instance, or when you have more than one array and have more data than fits on a single array. However, that wasn't the case here (at least not after a resize2fs -M call followed by an appropriate lvresize one), and so I thought that not reducing redundancy, even if only temporarily, was a better move.

Next.

Previous.