Search Results: "stone"

5 October 2013

Joachim Breitner: Why PVP is better than no PVP

This is a reply to Roman Cheplyaka s post Why PVP doesn't work , where he argues that Haskell packages should not per default put upper bounds on their build dependencies (as mandated by the Package Versioning Policy), at least not without good reason. I disagree, and I d like to explain why. I assume that you are packaging your libraries and programs not only for your fellow developers, but also for users. Hackage has turned not only into a hub for developers sharing their libraries, but also for users to conveniently download stuff from. And when I say users, I include xmonad users (although they write a bit of Haskell for their configuration) and beginners (who we certainly don t expect to understand and make sense of error messages that occur if some obscure crypto-related dependency of their favourite web framework fails to build). And for their benefit we should try hard to make sure cabal install either succeeds (great) or fails early with an error message that the user can understand as I could not find a package configuration of which the maintainers believe with good confidence that it works. Please talk to them. I speak with some experience, because as a maintainer of the >666 Haskell packages in Debian, I am, in a sense, a very heavy user. In the years after the PVP got commonly accepted, we could noticeably reduce the number of build failures that we encounter on our machines and the Debian build farms, simply because the meta data (which we transfer to the Debian package level) prevented such builds from being attempted. We now even have a build compatibility prediction system: We maintain a text file with all Haskell packages in Debian, together with a directory of patches (where we sometimes indeed override the packages Cabal file), and a script that runs cabal-install s dependency checker against this day. This way, when I want to upgrade a package, I can simply change the version in the file and know what packages will be broken. So I change their version until I either find a fixed-point (and only then actually upload packages to the user) or find a problem, which I then report to the maintainers or start cooking up my own patches. This works really well, but it only does because most maintainers are following the PVP. That s why I argue: Please continue to do so. (What we would probably need instead of less using the PVP is more tools in helping us with it. I could imagine a build bot that finds packages on Hackage with an upper bound on a package where a newer version exists, fetches those and automatically tries to build against the new version, and if it succeeds, sends a message to the maintainer: Your dependency on bar in foo also builds against version 0.42 of bar; please check for incompatibilities that are not expressed in the type system and upload a new version of foo allowing bar-0.42. Isn t that something the Stackage infrastructure can be used for?) (What we could probably use as well (and is probably a stepping stone towards what I just said) is a way to tell cabal-install to ignore a certain build dependency version bound. This way we still prevent normal users from trying a dependency version that the maintainer did not test, but nevertheless make it as easy as possible for the advanced user to try nevertheless, and then hopefully report about it to the maintainer. Of course, other had that idea before, someone just has to make it happen.)

10 September 2013

Julien Danjou: OpenStack Ceilometer Havana-3 milestone released

Last week, the third and last milestone of the Havana development branch of Ceilometer has been released and is now available for testing and download. This means the end of the OpenStack Havana development time is coming, and that the features are now frozen. New features
Eleven blueprints have been implemented as you can see on the release page. That's one more than during Havana-2, but it's less than was planned initially, though we had a pretty high score considering the size of our contributors team. I'm going to talk through some of them here, that are the most interesting for users. Bug fixes Fifty-six bugs were fixed, though most of them might not interest you so I won't elaborate too much on that. Go read the list if you are curious. Toward our final Havana release With the feature freeze in place, we're now focusing on fixing bugs and improving documentation. I'll try to make sure we'll get there without too much trouble for the 17th October 2013. Stay tuned!

8 September 2013

Joachim Breitner: You-Say-First has dice

A while ago I wrote a small web application called You-Say-First. It allows multiple users to join a chat room where they can, besides chatting, enter moves that are only shown to everyone once everyone has submitted one. This way you can play games like Rock-Stone-Scissors or Diplomacy online, even if you want to play by your own strange rules that none of the standard, game-specific web sites support. Or even make up many kind of games on the spot!
I was approached by Jason Worley who said he liked the site, but would have a need for dice rolling to play the games he wants to play. So he gets dice rolling: Just say, in the chat, what kind of dice you want and the server will roll them for you and tell everyone about them. I tried to make it understand many variations of the command, so you can say roll a die, please , toss five coins or throw 3D20 . If the server does not react, try modifying your command. As always I m happy about feedback, bugreports and patches, and I hope that it is a useful tool for you.

28 August 2013

Vincent Sanders: Men admire the man who can organize their wishes and thoughts in stone and wood and steel and brass.


I would probably not yet worthy of the admiration Emerson was alluding to but I do like to make things. As anyone who has read previous posts knows I have pretty much embraced the "do things, tell people" idea.

One small wrinkle is doing things needs somewhere to work. Since moving myself and the family to rented accommodation in Cambridge (swampy 3,500 year old English city, not the one in Massachusetts) I have been lacking space to do practical projects.

The main space
To fix this lack I have joined the cambridge makespace which, in addition to somewhere I can work gives me access to some tools I was previously unable to afford. The space gives practical training on the more complex machines (any tool can be dangerous if you do not use it correctly) which recently allowed my induction on the CNC router.

My instructor , Mark Mellors, who was good enough to give up some of his valuable making time to train me (and accidentally get his car stuck in a car park overnight by staying late) suggested that it was a good idea to have a design to try.

I decided to use this opportunity to actually create something useful (though now I re-read Marks message it did suggest a simple design...oopsy). I had been working at the electronics bench in previous weeks and been uncomfortable using the existing stools and chairs as they were either a bit high or unable to be adjusted high enough for the 80cm tall benches. I decided to design a 60cm tall stool for use at this workbench.

My initial idea was for a simple three leg stool, round top, three legs, how hard could it be? Initial research showed that showed that above 30cm the legs needed to be braced to each other. This is because the leg to seat joints simply cannot handle the stress caused by leverage which longer legs introduce.

I looked at the structure of several stools online and was initially drawn towards creating something like the IKEA Dalfred bar stool. It was discovered that the design would be easier to realise if it were made from sheet material which give a smaller challenge to a naive operator of a CNC router. Because of this simpler designs were researched and I eventually found a simple design I liked.

The design could not however be used directly as it was for imperial sized material which is not available in europe. I selected the QCAD open source CAD package and recaptured the design adjusting for the available 12mm plywood sheet material. This resulted in an imperial measurement design for metric materials.

Mark helped me transfer the DXF into the CAM software (vcarve pro 7, after I discovered the demo version of this software generates files which cannot be imported into the full edition!) and generate toolpaths for our machine. Once the toolpaths are saved to a USB stick (no modern conveniences like direct upload here) the job can be run on the machine.

Here I ran into reality, turns out that tolerances in imperial combined with lack of understanding how plywood reacts resulted in excessive play in the joints. This resulted in an unusable stool, which simply tried to rotate around its central axis and become flatpack. I had successfully turned 10 worth of plywood into some sawdust and a selection of useless shapes. On the other hand I did become competent with the router workflow so it was not a complete failure.

I was determined to make the design work so I decided to start from scratch with a similar design but entirely in metric. I performed some material research both online and practically (why yes i did spend an informative hour in several cambridge DIY shops with digital calipers, why do you ask?) it turns out that generally available 12mm thick plywood actually ranges between 11.9 and 12mm thick.

I did some test slot cuts of varying width and determined that the available stock can be "persuaded" to fit into a 11.8mm wide slot. This Interference fit joint is strong and removes the need for adhesive in most cases.

Second cut still attached to the bed. Leftovers of first attempt in the background
The sheet plywood material is readily available with a width of 1220mm and a height of 606mm (a full sheet is 2440mm long which is cut into four with a 4mm wide saw blade) so making the legs fit within a sheet and be close to the 60cm target should be possible.

I selected a suitable seat radius (175mm) and from that determined the minimal gap to the base ring with a 6mm end mill tool (20mm for two toolpaths and some separation) and hence the minimum suitable width of the base ring (50mm) giving a total radius of 245mm.

For the legs allowance was made for two joints of 30mm with 30mm separation between them the legs come out at 90mm width. If a 6mm space and a 6mm toolpath top and bottom of the sheet is accounted for a 582mm height (606 - 24) is available in which to fit the legs. The top of the leg which fits into the seat is 12mm tall leaving 570mm total leg height.

12mm plywood stool design
At this point I selected some arbitrary values for the leg positioning and angle (30mm from seat centre and 15 ) using a right angle triangle triganometry this produced a stool with a base of around 550mm or 100mm outside the radius of the seat. This seemed a pleasing shape and when the values for the holding ring were calculated it is at 104mm height which also seems to work out well.

This design was cut on the machine, lightly finished with some sandpaper and assembled. The legs slotted into the ring first and then the legs eased into the seat slots, the whole thing flipped and the seat hammered home onto the legs.

Anne Harrison volunteering to try the wobbly stool of doom
Success! It physically fit together and if you were brave enough you could sit on it. Unfortunately the plywood seemed to flex around the central axis of rotation in a rather alarming way, fine if you are under 70 kilos but not giving the impression of security most people want from their seating.

Ok, lets try with five legs instead of three (at least we can reuse the existing three legs)...nope still not good enough and another 10 gone.

18mm Plywood stool design
The others in the space suggested a few ideas to improve matters and the one I selected was to use 18mm plywood instead of 12mm, this should improve rigidity. There was a brief pause in proceedings to discover 18mm sheet is actually 17.7mm and needs a 17.5mm slot to make the interference fit work.

Completed 18mm stool
A swift redesign later altering the seat radius, gap, ring width and leg height to accommodate the new material and we have version 4 and it works without caveat. Tested up to 150Kg load without trouble, there is a small amount of flex still but nothing that feels worrying.

I finished the stool by rounding the seat top edge with a ball bearing rounding bit in a manual router and applying a couple of coats of gloss acrylic varnish. Finished stool is now doing service at the space.

In conclusion the final design allows someone with a CNC router to create a useful 580mm high fixed stool for 7.50 in timber plus cutter wear and varnish so maybe 8.50 total.

You can actually get five legs and a seat/ring out of a 1220x606 sheet and with intelligent arrangement a single 1220x2440 sheet will probably yield five or possibly six stools in total

I am making the design files of the proven 18mm version available (heck they are all there...but you have been warned, none of the other solutions produce a satisfactory result) under the MIT licence so anyone can reproduce. More pretty pictures are also available.

26 August 2013

Russell Coker: Scratching a Galaxy S

Some years ago when I first got a LG U990 Viewty (which in some ways is the best phone I ever owned) I went swimming and left my phone in my bag. My phone happened to rest on my car keys and had vibration mode enabled, after a couple of missed calls I had a nasty scratched area on the phone screen. Since then I ve been very wary about allowing metal objects to come in contact with a phone screen. Now I have a Samsung Galaxy S with some sort of motherboard damage (it won t even boot and I know it s not a software issue because it was initially intermittent). A phone that old isn t worth repairing (they sell on ebay for as little as $50) so it seemed worth testing how hard the screen is. The screen cover is Gorilla Glass which was the hardest glass available at the time the phone was new (apparently there are better versions of Gorilla Glass available now and my more recent phones should be tougher). My first test was with one of my favorite Japanese kitchen knives, it didn t scratch at all. Then I chose a knife sharpening stone as an obvious item that s harder than a knife, it scratched the screen easily. A quartz pebble also scratched the screen when I used some force, so presumably concrete and brick would also scratch it. Tests with all current Australian coins and my car keys showed that the screen is too hard to be scratched by them. I also tested hitting the phone screen with my keys, I hit it much harder than would happen if I was to run while having my phone and my keys in the same pocket and there was no damage. My conclusion is that any metal object you are likely to carry in your pocket is unlikely to cause any problem if knocked against the screen of a modern phone.

17 August 2013

Lars Wirzenius: Yarn, a scenario testing tool

I write a fair number of Unix command line programs, and I want to have test suites for them. A year and a half ago I wrote a tool to make that easier: cmdtest. However, it turned out to not be a particularly good one, primarily because it doesn't make it easy to write good test code. Tests and test code should be obviously correct, so you don't need a test suite for your test suite. Earlier this year, Daniel and I started a slow, but long discussion about this. We'd talk it over during many a lunch, figuring out what we actually want. We eventually converged on an approach based on for Unix command line programs, and based on Markdown documents. We then wrote that up as a README, and then took another month refining the README before writing any code. The naming discussion only took a week. As a concrete example, here is a real test scenario, from Obnam, for testing its new FUSE plugin:
SCENARIO Browsing backups with FUSE plugin
ASSUMING user is in group fuse
GIVEN a live data directory
AND a 0 byte file called empty
AND a 1 byte file called one
AND a 4096 byte file called 4k
AND a 10485760 byte file called 10meg
AND a hardlink to 4k called 4k-hardlink
AND a symlink to 4k called 4k-symlink
AND a directory called some-dir
AND a symlink to ../4k called some-dir/other-symlink
WHEN live data is backed up
AND repository is fuse mounted
THEN latest generation can be copied correctly from fuse mount
FINALLY unmount repository
Some explanations: Test scenarios like this are obviously not directly executable. We need to provide the code that actually executs each step. For this, yarn provides an IMPLEMENTS section, of which I will show one example: the creation of the files.
IMPLEMENTS GIVEN a (\d+) byte file called (\S+)
dd if=/dev/zero of="$DATADIR/live/$MATCH_2" bs=1 count="$MATCH_1"
(The other IMPLEMENTS sections for that Obnam test scenario are also only about a line of shell. The longest one is four lines, because it has an if statement that I like to format in a fairly verbose way. I'm skipping the others to avoid needless repetition.) The text after the keyword (GIVEN, WHEN, THEN, etc) in the test scenario is a single line of free-form text, which is matched by a regular expression in the IMPLEMENTS section. There must be exactly one matching IMPLEMENTS for each line in a test scenario. The body of IMPLEMENTS is a shell script snippet. Yarn provides a temporary directory for each scenario, and the path to that is given in the DATADIR environment variable. Yarn also cleans up the directory afterwards, unless the user running yarn requests otherwise, so tests that only change things in the temporary directory don't need any FINALLY steps. The above example uses FINALLY to unmount the FUSE filesystem at the end. The regular expressions can capture parts of the text in the scenario step, and these are provided to the shell script using environment variables: MATCH_1 etc. This is a simple, but powerful mechanism for implementing very flexible steps, which also means the number of steps can be kept down. The IMPLEMENTS sections are shared between all test scenarios. This makes it very easy to write new scenarios using already-existing steps, by choosing which ones to use, in which order, and making use of any variability in the steps that their implementations provide. Yarn is a new tool, and I expect it to mature in the coming months. It's included in the cmdtest Debian package, and is already in Debian jessie and unstable, for those who want to try it. (My personal apt repository has a .deb for wheezy as well; backports are welcome). The yarn scenario testing language itself is not set in stone, but Daniel and I both want to be careful about making changes to it. It's all meant to be very simple: an obviously correct testing tool is better than an intricate one. I've used yarn in several of my own little programs now, as has Daniel, and we've used it at work as well. Undoubtedly we'll find bugs in the implementation, and we're still feeling our way to figuring out how to write really excellent scenarios, and how to structure steps with the best possible taste. We'll also start designing a shell library that yarn provides to the IMPLEMENTS sections, for doing things that we find ourselves doing in many projects. It is already clear to me, however, that yarn is oh so much better than cmdtest could ever be. I'm even starting to express ideas for new tools as yarn test documents, so I'm clear what they should do. If and when I ever get to actually implementing the tools, I'll have a test suite for them already. See, for example, first attempts at specifying how a fast file transfer tool, fling, which is to use Linux-only system calls to reach wire speeds over wired LAN, and a tool to recompress compressed files with another compression tool. We hope yarn will prove to be useful for others as well. If you try it, and have suggestions or comments, please e-mail us.

16 August 2013

Daniel Leidert: N54L LCD-Mod with U204FR-A3 (LCDModkit) and lcdproc (II)

Here it goes. I was looking for an LCD display for my microservers 5,25'' bay and found this neat post. The manufacturer of the LCD display used there provides different displays for this purpose. I've decided for an U204FR-A2 in black with a red backlight. It left Hongkong around five days later and arrived here another 6 days later. All in all: I got it after 11 days. I unpacked the LCD device. It comes with an internal USB connector and is driven by an Hitachi HD44780 LCD controller. The connection wasn't a problem at all. I've already put a Silverstone SST-EC04-P PCIe card with two USB 3.0 external ports and an internal 19pin dual port connector into the systems PCIe 1x slot. Now to connect the LCD with this card I've bought an Inline 19pin USB 3.0 header to 8pin double USB 2.0 header adapter and connected the card with the LCD display. Easy, right?
To make the display "attached" to the case - it comes with two steel sheets and two screw holes each, that cannot be attached to anything in the microserver case - I've used a small workaround: double-faced adhesive tape and two halfs of a matchbox - one can also use small scantlings - and created a bonding between the steel sheets and the case.
That's it. I put the cover plate carefully back - the steels sheets of the LCD display and the LED of the server will bump to each other!
There are two programs to output information to the LCD display. These are lcdproc and lcd4linux. I started with the first one which only provides pre-defined screens. Seems with the latter one can create own screens. This is an option for the future. lcdproc consists of two programs. First there is a daemon called LCDd. It controls the driver, contrast etc.pp. The relevant parts of its configuration file /etc/LCDd.conf look like as shown below. Note that I did not change the default values for contrast or brightness.

[server]
Driver=hd44780
WaitTime=25

[hd44780]
ConnectionType=lcd2usb
Size=20x4
To print something to the screen one can use the lcdproc command, which is configured via /etc/lcdproc.conf. I've enabled the Iface, TimeDate, SMP-CPU, and MiniClock screens. The program is started during startup via cron. The file /etc/cron.d/lcdproc simply contains this:

@reboot root lcdproc
The following pictures show the resulting screens, which change every 25 seconds. That's it.

15 August 2013

Daniel Leidert: HP N54L Microserver - energy efficiency and power management

I recently worked on activating power management functions, reduce energy consumption and noise of my little HP N54L "toy". During this process I tried to avoid the usage of /etc/rc.local and set things by udev, hdparm and friends. Below are my results. Actual resultsWith the following steps my system (N54L + 3xWD20EFRX HDD +1xWD5003AZEX HDD + LCD-mod + case fan mod + Debian Wheezy) uses 27W in idle mode. The USB W-LAN card uses another 10W. In active mode, e.g. compiling source code, the system runs (and boots) with around 57W. The highest power consumption observed is during startup phase with 88W. First things firstFor the following steps it might be necessary to have some packages installed, that maybe do not occur in this post. If I missed something, I appreciate a hint. Further the following steps might produce even better results with a custom kernel. I'm using the stock linux-image-3.2.0-4-amd64 kernel image as the time of writing and I have these packages installed: amd64-microcode, firmware-linux, firmware-linux-free, firmware-linux-nonfree and firmware-atheros (the latter for my WLAN card). ASPM and ACPIFirst I enabled PCIE ASPM in my (non-modded) BIOS and forced it together with ACPI via grub by changing GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub so it looks like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash acpi=force pcie_aspm=force nmi_watchdog=0"
ASPM has now been enabled as lspci prooves:

00:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] RS780/RS880 PCI to PCI bridge (PCIE port 0) (prog-if 00 [Normal decode])
[..]
LnkCap: Port #1, Speed 5GT/s, Width x1, ASPM L0s L1, Latency L0 <64ns, L1 <1us
ClockPM- Surprise- LLActRep+ BwNot+
LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes Disabled- Retrain- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
[..]
00:06.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] RS780 PCI to PCI bridge (PCIE port 2) (prog-if 00 [Normal decode])
[..]
LnkCap: Port #3, Speed 5GT/s, Width x1, ASPM L0s L1, Latency L0 <64ns, L1 <1us
ClockPM- Surprise- LLActRep+ BwNot+
LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes Disabled- Retrain- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
02:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03) (prog-if 30 [XHCI])
[..]
LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Latency L0 <4us, L1 unlimited
ClockPM+ Surprise- LLActRep- BwNot-
LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes Disabled- Retrain- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
[..]
03:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5723 Gigabit Ethernet PCIe (rev 10)
[..]
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0 <1us, L1 <64us
ClockPM+ Surprise- LLActRep- BwNot-
LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes Disabled- Retrain- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
[..]
Even so /sys/module/pcie_aspm/parameters/policy will still show as below:

[default] performance powersave
I'll show how to set the powersave value in /sys/module/pcie_aspm/parameters/policy in the next section.JFTR: These are my ACPI related packages installed: acpi, acpid, acpi-support and acpi-support-base. Enable powersaving via UDEVThe following rules file /etc/udev/rules.d/90-local-n54l.rules has been inspired by a blog post. It enables powersaving modes for all PCI, SCSI and USB devices and ASPM. Further the internal RADEON cards power profile is set to the low value. There is no monitor connected usually. The file contains these rules:

SUBSYSTEM=="module", KERNEL=="pcie_aspm", ACTION=="add", TEST=="parameters/policy", ATTR parameters/policy ="powersave"

SUBSYSTEM=="i2c", ACTION=="add", TEST=="power/control", ATTR power/control ="auto"
SUBSYSTEM=="pci", ACTION=="add", TEST=="power/control", ATTR power/control ="auto"
SUBSYSTEM=="usb", ACTION=="add", TEST=="power/control", ATTR power/control ="auto"
SUBSYSTEM=="usb", ACTION=="add", TEST=="power/autosuspend", ATTR power/autosuspend ="2"
SUBSYSTEM=="scsi", ACTION=="add", TEST=="power/control", ATTR power/control ="auto"
SUBSYSTEM=="spi", ACTION=="add", TEST=="power/control", ATTR power/control ="auto"

SUBSYSTEM=="drm", KERNEL=="card*", ACTION=="add", DRIVERS=="radeon", TEST=="power/control", TEST=="device/power_method", ATTR device/power_method ="profile", ATTR device/power_profile ="low"

SUBSYSTEM=="scsi_host", KERNEL=="host*", ACTION=="add", TEST=="link_power_management_policy", ATTR link_power_management_policy ="min_power"
Set harddrives spindown timeoutI decided to sent my system drive to standby after 20 minutes and the RAID drives after 15 minutes. This is usually ok, because the RAID isn't always used. hdparm is the right tool to realize this. Many people use the /dev/disk/by-uuid/... syntax here, to avoid having to touch the configuration file if some system configuration changes. Because I'm running a RAID, I couldn't use this syntax, although it might be possible to use /dev/disk/by-id/... instead. Well for the moment I stay with the configuration below. The relevant part of /etc/hdparm.conf is:

[..]

# system harddrive
/dev/sda
spindown_time = 240


# below are the WD20EFRX drives
/dev/sdb
spindown_time = 180


/dev/sdc
spindown_time = 180


/dev/sdd
spindown_time = 180

Idle modeWhen there is nothing to do for the system, all I hear is the (still a bit noisy) fan of the power supply, which I might replace in the future too. Either by testing a different fan or by replacing the whole power supply unit by the fanless FORTRON FSP150-50TNF or (even better) a picoPSU.The system currently shows a power consumption of around 37W in idle mode whereas the USB W-LAN card itself needs around 10W. There is a possibility to enable power savings mode for this card too. I could add this entry to /etc/udev/rules.d/90-local-n54l.rules:

SUBSYSTEM=="net", ACTION=="add", KERNEL=="wlan*" RUN+="/usr/bin/iw dev %k set power_save on"
But it turned out that the connection became a bit unstable after it. So I don't use this rule. More on the roadThere are a lot more options one can easily find via $search_engine. The N54L system could be brought to sleep and woken up by LAN via Wake-on-LAN (WOL). This is a feature I don't use. I've also read rumors about enabling different sleep/suspend states of the system, which seems to require to install a modded BIOS. Well, I'll post news and changes if they happen to come;)

12 August 2013

Chris Lamb: Cotswold Classic Middle Distance triathlon

https://chris-lamb.co.uk/wp-content/2013/cotswold_course.jpg Since becoming interested in triathlon in October of last year, my goal in 2013 was not to just finish a "70.3" (or "Half-Ironman"), but to complete one in under 5 hours. At the time, each discipline was woefully inadequate; I had not swum in over a decade, my cycling was average and I had only begun running again after years away from exercise. The somewhat arbitrary target time would dictate my training. For example, the high proportion of time spent on the bike combined with the injury-sensitive nature of running meant that cycling would have a highest return on investment of the three disciplines. Despite that, I would need to take the run seriously but carefully - I decided to enter a series of monthly 10K races where I could steadily reduce my times over the year. Races themselves formed a crucial part of my preparation and a carefully balanced race calendar was surprisingly important, balancing upcoming milestones that would provide an achievable goal as well as uncovering weaknesses in enough time so they could be worked on. Whilst I did post about my first triathlon, I felt no compulsion with the interleaving five or six races as I was evaluating them in the context of the 70.3 rather than as races in their own right.
Swim
Distance
1900m
Time
37:42
Whilst this wasn't my worst open water swim, it was certainly not my best. The first 500m was straightforward, but I lost my form over the next 500m and I overcompensated with excessive kick. My calves responded by cramping badly two or three times after that, losing both momentum and time. Curiously, whenever I could identify my speed relative to my effort (for example, by some underwater reeds) I could use that as feedback and my rotation (etc.) would temporarily return. Pool swimming always has markers of this kind, whilst a murky lake does not. In conclusion, my lack of open water training outside of races held me back more than I expected.
Bike https://chris-lamb.co.uk/wp-content/2013/cotswold_bike.jpg
Distance
52 miles
Time
2:22 (21.9 mph average)
I had thoroughly absorbed the advice that you should prepare for your primary race so that it feels like another day at the office. On this criterion, the bike leg was almost perfect. My nutrition strategy energy drink every 5 minutes, gel every 30 minutes worked fine until mile 35 where I felt like I had taken on too much fluid (and no time to take a bathroom break). I had a temporary hiccup with my rear derailleur but my "tame" 52/34 12x28 gearing was vindicated on the climbs, although in retrospect that might mostly be schadenfreude. Tens of hours acclimatising to a time trial bike was rewarded by being able to spend the entire leg in "aero" position (ignoring 90-degree turns or serious gradients), improving efficiency.
Run https://chris-lamb.co.uk/wp-content/2013/cotswold_run.jpg
Distance
13.1 miles (21km)
Time
1:54:18 (5:25/km)
I had prepared a chart which provided the pace I required based on the race time at the start of the run. I would then only need to monitor my run pace instead of repeatedly extrapolating a finish time as even basic arithmetic can become impossible when fatigued. This strategy was chosen as I was too risk-adverse to aim for a faster time than strictly required; the downside of missing the target from aiming too high and "blowing up" outweighed going a few minutes faster overall. Despite that, I ran half-marathon personal best. Unfortunately, due to pressing the wrong watch button after the swim, the chart was worthless unless I could assume the race started exactly at 6:30AM. With no other option I decided to trust that with some padding, but the uncertainly it added was unsettling. Adding to that feeling, my left hip started to twinge at 7K, something it has never done before. My plan was to get to 16K, take a caffeine gel and then keep ratcheting up the pace until the finish. In my long run training I could speed up to a 4:10/km pace but after 5 hours of racing and being reasonably sure I was going to reach my goal, the best I could summon was 4:45/km. My form completely shot, I crossed the finish line and got into a fetal position in the shade behind a tent, scaring the race organisers for a few minutes.
Overall
Total time
4:56:50
Writing this the day after the race, I am still a little unsure of how I feel. I am obviously extremely content that I reached my goal but given the volume and detail of preparation it was not a huge surprise to me on race day that I did so. This combined with the "real" work and progress being achieved throughout the training seems to have robbed the event of some element of triumph it could have had, but that in itself was somewhat expected. Where next? I had always dismissed the consensus view of needing to take a decent break after the season has ended but now I am looking forward to putting triathlon-specific training on hold for a while and explore other things. I highly doubt this will be my last triathlon though. https://chris-lamb.co.uk/wp-content/2013/cotswold_result.jpg (Full results)

10 August 2013

Rudy Godoy: Bulding non-service cartridges for RedHat Openshift

Cloud is passing the hype curve and we are seeing more stable developments and offerings on the market. Lately I ve been playing with RedHat s Openshift. Openshift is a PaaS (Platform as a Service) offering that intends to be an alternative for vendors such as Heroku. The focus for such offerings is to give developers enough flexibility and tools that handle the application deployment and maintenance process in a way that is integrated with their existing development workflow and tools. I ve been using Heroku for a while to deploy small to medium size projects. I liked the tools and developer-centered experience they offer. Openshift is quite new on the market and it comes in two flavors: Openshift Online, which is a PaaS service, and Openshift Enterprise, which allows organization to setup a PaaS within their own infrastructure. Both of them powered by the Openshift origin software. I ll not compare Heroku vs. Openshift feature by feature but from my experience I can tell that Openshift is far from mature and will need to give developers more features to increase adoption. When developing applications for Openshift developers are given a set of application stack components, similar to Heroku s buildpacks. They call them cartridges. You can think of them as operating system packages, since the idea is the same: have a particular application stack component ready to be used by application developers. Most of the cartridges offered by the service are base components such as application server, programming language interpreters (Ruby, Python, etc), web development frameworks and data storage, such as relational and non-relational databases. Cartridges are installed inside a gear, which is a sort of application container (I believe it uses LXC internally). Unsurprisingly this Openshift component doesn t leverage on existing packaging for RHEL 64bit, the OS that powers the service. I d expect such things from the RedHat ecosystem. I had to develop a cartridge to have a particular BI engine to be used as embed component by application developers. After reading docs and reference I realized this can be piece of cake, since I have packaging experience. Wrong. Well, quite so. The tricky part for Openshift Online is that it does not offer enough information on the cartridge install process so you can see what s going wrong. To be able to see more details on the process you ll need to setup an Openshift origin server and use it as a testing facility. Turns out that having a Origin server to operate correctly is also a challenging task and consumed a lot of my time. Over the recent weeks I ve learned from origin developers that such features are on the road map for the upcoming releases. That s good news. One of the challenges I had, and still have to figure out, is that unlike the normal cartridges mine didn t required to launch any service. Since it is a BI engine I just needed to configure and deploy to an application server such as JBoss. Cartridge format requires to have a sort of service init.d script under bin along with setup, install and configuration scripts that are ran on install. Although every day I become more familiar with origin and Openshift internals I still have work to do. Nice thing is that I was already familiar with LXC and Ruby-based applications so I could figure where things are placed and where to look for errors on origin quite easily. The cartridges are on my github account if you care to take a look and offer advice.

3 August 2013

Russell Coker: Religious Conversion

Today I was in the center of Melbourne and I saw two stands run by Muslims advocating their faith. They had a selection of DVDs and brochures to give away and seemed friendly enough for anyone who wanted a chat. I think that this is a good thing, if the majority of the Australian population would think of the Muslim minority as people who give away DVDs rather than as potential suicide bombers then it would be a great benefit for society. Then I saw a Christian stand which had placards indicating that Jesus loves Muslims, which is fine. One of the workers on that stand then gleefully told me that there had been a Muslim stand in that area but the Muslims departed when they arrived. It would be good if advocates for the various religions could work together to promote positive things that they all agree on (generally related to being nice to other people). When they seem to be essentially bullying other religious groups into leaving the area they aren t making a good case for a kind and caring god. The Christian who was so happy about the Muslims departing then tried to convert me. He started by talking about a mobile phone not being able to appear from nothing which was the start of an utterly stupid argument about creationism. That was stupid firstly because it s obvious that evolution doesn t apply to consumer electronics, if I leave a phone lying on top of another phone overnight I m not going to get some little phones appearing as a result. But the bigger stupidity is in even promoting creationism in the first place. Advocating creationism is essentially claiming that god is so incapable that ze can t manage an evolutionary process. An all-seeing all-knowing god of infinite intelligence could just create a plasma cloud that will form into a solar system and evolve life. People who argue for creationism don t believe in a capable god, they believe in a being that resembles a god in the way a magician resembles someone who can actually perform magic. Arguing for creationism is arguing for a limited being which probably isn t worthy of worship (why worship space aliens who are probably only a few thousand years ahead of us). Regardless of the scientific evidence (which is strongly in favor of evolution) just logically thinking about the issue in regard to what a being with infinite powers might actually do suggests that ze could just as easily create or evolve all life and probably wouldn t care about what we believe. Genesis should be taken as metaphor because it s obvious that nothing that s taught in a year 7 science class could be taught to your typical stone age goat-herd, but even if it was taken literally it s not specified as a required belief (as opposed to a belief in a single god, repenting sins, and other beliefs which are required). The greatest stupidity however is the fact that even if it made any sense to promote creationism it wouldn t make sense to do so immediately. Any time you want to convert someone to a set of beliefs it s a good strategy to start with the ones which will seem least wacky. I can have have a conversation with Christians about positive things which we generally agree on such as donating to charity and trying to do good things in various ways. But when they start with the stupid stuff I walk away. When the weather is better I plan to interview representatives of the various religious groups who are trying to convert people on the street and rate them according to how wacky they are. I ll be a little limited in my ability to review them by the total lack of any right to free speech in Australia (our laws are much the same as those in the UK). But I should still be able to give them a reasonable review.

Richard Hartmann: Death of a slowly demising platform

Let's assume you have been running a platform for over a decade. Let's assume you used to be one of the cornerstones, one of the central pillars, maybe even part of the very foundation FLOSS was built upon. Let's assume you slept through several major shifts and stopped innovating a long time ago (along with your other large sister project, owned by the same people). Let's assume you realize all this. You then try and get your act together, work over your feature set, create a UI that's easy and pleasant to use, encourage collaboration, visualize important data, and otherwise revamp yourself, start innovating again and regain your status of a platform which really matters. Or you just do this and call it a day.

29 July 2013

Brett Parker: Of weekends and patio laying...

So, this weekend I headed up to Uncle Steve's and assisted in the laying of the flagstones for the patio for the BBQ - there's even at least one bit of photographic evidence of me in a non-supervisory capacity. Was a good weekend, and fun was had by all, even if we couldn't really express it due to Complete Exhaustion by the end of it. Only another few weekends and it'll be the Debian UK Party 2013 which will, as it is every year, be absolutely awesome. I'll leave the thanks for the beer and food donations until after the party :)

27 July 2013

Julien Danjou: OpenStack Ceilometer Havana-2 milestone released

Last week, the second milestone of the Havana developement branch of Ceilometer has been released and is now available for testing and download. This means the first half of the OpenStack Havana development has passed! New features Ten blueprints have been implemented as you can see on the release page. I'm going to talk through some of them here, that are the most interesting for users.
The Ceilometer API now returns all the samples sorted by timestamp. This blueprint is the first one implemented by Terri Yu, our OPW intern! In the same spirit, I've added the ability to limit the number of samples returned. On the alarming front, things evolved a lot. I've implemented the notifier system that will be used to run actions when alarms are triggered. To trigger these alarms, Eoghan Glynn (RedHat) worked on the alarm evaluation system that will use the Ceilometer API to check for alarm states. I've reworked the publisher system so it now uses URL formatted target for publication. That now allows to publish different meters to different target using the same publishing protocol (e.g. via UDP toward different hosts). Sandy Walsh (RackSpace) have been working on the StackTach like functionality and added the ability for the collector to optionally store the notification events received. Finally, Mehdi Abaakouk (eNovance) implemented a TTL system for the database, so you're now able to expire your data whenever you like. Bug fixes Thirty-five bugs were fixed, though most of them might not interest you so I won't elaborate too much on that. Go read the list if you are curious. Toward Havana 3 We now have 30 blueprints targeting the Ceilometer's third Havana milestone, with some of them are already started.I'll try to make sure we'll get there without too much trouble for the 6th September 2013. Stay tuned!

4 July 2013

Julien Danjou: OpenStack meets Lisp: cl-openstack-client

A month ago, a mail hit the OpenStack mailing list entitled "The OpenStack Community Welcomes Developers in All Programming Languages". You may know that OpenStack is essentially built using Python, and therefore it is the reference language for the client libraries implementations. As a Lisp and OpenStack practitioner, I used this excuse to build a challenge for myself: let's prove this point by bringing Lisp into OpenStack!
Welcome cl-openstack-client, the OpenStack client library for Common Lisp! The project is hosted on the classic OpenStack infrastructure for third party project, StackForge. It provides the continuous integration system based on Jenkins and the Gerrit infrastructure used to review contributions. How the tests works OpenStack projects ran a fabulous contribution workflow, which I already talked about, based on tools like Gerrit and Jenkins. OpenStack Python projects are used to run tox, to build a virtual environment and run test inside. We don't have such thing in Common Lisp as far as I know, so I had to build it myself. Fortunately, using Quicklisp, the fabulous equivalent of Python's PyPI, it has been a breeze to set this up. cl-openstack-client just includes a basic shell script that does the following: I just run the test using SBCL, though adding more compiler on the table would be a really good plan in the future, and should be straightforward. You can admire a log from a successful test run done when I proposed a patch via Gerrit, to check what it looks like. Implementation status For the curious, here's an example of how it works:
* (require 'cl-openstack-client)
* (use-package 'cl-keystone-client)
* (defvar k (make-instance 'connection-v2 :username "demo" :password "somepassword" :tenant-name "demo" :url "http://devstack:5000"))

K

* (authenticate k)

((:ISSUED--AT . "2013-07-04T05:59:55.454226")
(:EXPIRES . "2013-07-05T05:59:55Z")
(:ID
. "wNFQwNzo1OTo1NS40NTQyMthisisaverylongtokenwNFQwNzo1OTo1NS40NTQyM")
(:TENANT (:DESCRIPTION) (:ENABLED . T)
(:ID . "1774fd545df4400380eb2b4f4985b3be") (:NAME . "demo")))

* (connection-token-id k)

"wNFQwNzo1OTo1NS40NTQyMthisisaverylongtokenwNFQwNzo1OTo1NS40NTQyM"

Unfortunately, the implementation is far from being complete. It only implements for now the Keystone token retrieval. I've actually started this project to build an already working starting point. With this, future potential contributors will be able to spend efforts on writing code, and not on setting up the basic continuous integration system or module infrastructure. If you wish to help me and contribute, just follow the OpenStack Gerrit workflow howto or feel free to come by me and ask any question (I'm hanging out on #lisp on Freenode too). See you soon, hopping to bring more Lisp into OpenStack!

Julien Danjou: OpenStack meets Lisp: cl-openstack-client

A month ago, a mail hit the OpenStack mailing list entitled "The OpenStack Community Welcomes Developers in All Programming Languages". You may know that OpenStack is essentially built using Python, and therefore it is the reference language for the client libraries implementations. As a Lisp and OpenStack practitioner, I used this excuse to build a challenge for myself: let's prove this point by bringing Lisp into OpenStack!
Welcome cl-openstack-client, the OpenStack client library for Common Lisp! The project is hosted on the classic OpenStack infrastructure for third party project, StackForge. It provides the continuous integration system based on Jenkins and the Gerrit infrastructure used to review contributions. How the tests works OpenStack projects ran a fabulous contribution workflow, which I already talked about, based on tools like Gerrit and Jenkins. OpenStack Python projects are used to run tox, to build a virtual environment and run test inside. We don't have such thing in Common Lisp as far as I know, so I had to build it myself. Fortunately, using Quicklisp, the fabulous equivalent of Python's PyPI, it has been a breeze to set this up. cl-openstack-client just includes a basic shell script that does the following: I just run the test using SBCL, though adding more compiler on the table would be a really good plan in the future, and should be straightforward. You can admire a log from a successful test run done when I proposed a patch via Gerrit, to check what it looks like. Implementation status For the curious, here's an example of how it works:
* (require 'cl-openstack-client)
* (use-package 'cl-keystone-client)
* (defvar k (make-instance 'connection-v2 :username "demo" :password "somepassword" :tenant-name "demo" :url "http://devstack:5000"))

K

* (authenticate k)

((:ISSUED--AT . "2013-07-04T05:59:55.454226")
(:EXPIRES . "2013-07-05T05:59:55Z")
(:ID
. "wNFQwNzo1OTo1NS40NTQyMthisisaverylongtokenwNFQwNzo1OTo1NS40NTQyM")
(:TENANT (:DESCRIPTION) (:ENABLED . T)
(:ID . "1774fd545df4400380eb2b4f4985b3be") (:NAME . "demo")))

* (connection-token-id k)

"wNFQwNzo1OTo1NS40NTQyMthisisaverylongtokenwNFQwNzo1OTo1NS40NTQyM"

Unfortunately, the implementation is far from being complete. It only implements for now the Keystone token retrieval. I've actually started this project to build an already working starting point. With this, future potential contributors will be able to spend efforts on writing code, and not on setting up the basic continuous integration system or module infrastructure. If you wish to help me and contribute, just follow the OpenStack Gerrit workflow howto or feel free to come by me and ask any question (I'm hanging out on #lisp on Freenode too). See you soon, hopping to bring more Lisp into OpenStack!

28 June 2013

Daniel Leidert: Idea: A new toy (ein neues Spielzeug) ... HP Microserver N54L

Ich fertige regelm ig Backups meiner Systeme an. Diese werden auf der Systemplatte meines Notebooks abgelegt und via rsync auf mobilen Speicher dupliziert. Hierzu verwende ich eine USB-Festplatte. Diese enth lt auch Medien-Dateien und wird regelm ig an den Fernseher angeschlossen. Prinzipiell halte ich meine Daten daher f r sicher. Aber vor kurzem stie ich an die Grenzen ihrer Kapazit t. Schon l nger habe ich nach einer Alternative gesucht, nicht zuletzt da heute viel gr ere Festplatten m glich sind und mein Laptop ber einen eSATA-Anschluss verf gt, der schneller als USB2.0 ist. Meine bevorzugte Variante war ein FANTEC DB-ALU3e Geh use mit einer WD Red WD20EFRX 2TB (5400 RPM) Festplatte, die f r den 24/7 Betrieb zertifiziert ist (und zudem ber eine ausgezeichnete Reputation verf gt). Die Kombination lief sehr gut und schnell, sieht edel aus, ben tigt aber eine externe Stromversorgung. Ich kann Sie als Speicherl sung absolut empfehlen. Allerdings hatte ich zu diesem Zeitpunkt noch weitere Anspr che, die mit der o.g. L sung nicht zu befriedigen sind. So trage ich mich bereits l nger mit dem Gedanken an ein RAID-1-NAS. Au erdem spiegelt sich die Beanspruchung meiner Notebook-Festplatte durch das Pakete-Bauen f r Debian im S.M.A.R.T.-Status wieder. Daher wollte ich diese Arbeit an einen robusten lokalen buildd-Boliden abgeben und habe ber den Kauf eines g nstigen Rechners nachgedacht. Ein NAS verbraucht aber deutlich weniger Strom als ein Desktop-Rechner. Also wie l sst sich ein buildd und ein energiesparendes NAS vereinen? Per Zufall stie ich bei einem lokalen H ndler auf den HP ProLiant MicroServer N40L. Das Angebot klang super und so entschied ich mich zum Kauf meines neuen Spielzeuges: ein HP ProLiant MicroServer N54L, der zuk nftig folgende Aufgaben verrichten soll:
Datensicherung
Die Sicherung der Daten erfolgt cron-gesteuert auf den RAID-Verbund in eine gesonderte (verschl sselte) Partition. Der S.M.A.R.T.-Status der Festplatten wird via smartd berwacht. Sollte eine Platte kaputt gehen, bestehen gute Aussichten, die Daten zu retten. Eine zuk nftige Option w re auch noch ein RAID-6 Verbund.
NAS / File-Server
Das Ger t verf gt ber bis zu 6 SATA Anschl sse. Davon werden vier standardm ig via Wechselrahmen belegt. Die mitgelieferte 250GB Festplatte wird vorerst das Betriebssystem aufnehmen und an den drei verbleibenden Anschl ssen werden zun chst drei WD Red WD20EFRX 2TB (5400 RPM) Festplatten als RAID-5-Verbund f r den notwendigen Platz sorgen. Letzterer l sst sich ohne Erweiterung nur via Software-Raid und mdadm realisieren.
buildd
Betriebssystem wird Debian GNU/Linux. Der Hauptspeicher wird auf mindestens 8GB ECC-Ram aufger stet.
HTPC (XBMC)
Der Microserver l sst sich nicht als Massenspeicher an einen Fernseher anschlie en. Daher soll vorr. XBMC in Verbindung mit einem USB3.0 BR/DVD-Player den Server zum Entertainment-Ger t erheben.
Das ganze soll m glichst wenig Strom verbrauchen und leise sein. Zum Anschluss an das lokale Netzwerk habe ich mich f r WLAN entschieden, da kein Gigabit-Ethernet vorhanden ist. Folgende Teile ben tige ich f r "meinen" Server:
Server
HP ProLiant N54L MicroServer mit Turion II Neo 2,2 GHz, 2GB RAM/250GB HDD - ca. 200 EUR (lokal)
Bel ftung / Lautst rke
Scythe Slip Stream Geh usel fter 120mm 800RPM 11dB - ca. 9 EUR (SY1225SL12L)
Scythe Slip Stream Geh usel fter 120mm 500RPM 7,5dB - ca. 8 EUR (SY1225SL12SL)
Netzwerk
TP-Link TL-WN722N(C) 150Mbps USB-Adapter - ca. 15 EUR (TL-WN722N(C))
File-Server
3x WD Red WD20EFRX 2TB 5400 RPM SATA600 f r NAS 24/7 - ca. 95 EUR / St. (WD20EFRX)
buildd
8GB (2x4GB) Kingston ValueRAM DDR3-1333 CL9 ECC Modul RAM-Kit - ca. 85 EUR (KVR1333D3E9SK2/8G)
16GB (2x8GB) Kingston ValueRAM DDR3-1333 CL9 ECC Modul RAM-Kit - ca. 145 EUR (KVR1333D3E9SK2/16G)
HTPC
Sapphire Radeon HD 5450/6450/6570/6670/7750 PCIe 16x Low-Profile passiv/aktiv - ca. 25..100 EUR (11166-45-20G, 11190-09-20G, 11191-27-20G, 11191-02-20G, 11192-18-20G, 11202-10-20G)
SILVERSTONE PCIe 1x USB3.0 2xInt 2xExt - ca. 21 EUR (SST-EC04-P)
Logitech K400 od. Keysonic ACK-540RF - ca. 40 EUR (920-003100 bzw. ACK-540 RF)
BR/DVD-Player od. Brenner mit USB3.0 Anschluss - 50..100 EUR
LCD-Mod
LDC Display Modul mind. 4x20 - ca. 10 EUR
Interessant ist auch noch die Option einer echten RAID-Karte. Ich stie dabei auf die IBM ServeRAID M1015 (46M0831) und diesen Hinweis. Kauft man stattdessen den "Schl ssel" zur Freischaltung des vollen Funktionsumfanges, dann bezahlt man (lokal) zus tzlich ca. 150 EUR! Aber das nur BTW. N tzliche Links:

Matt Zimmerman: Scaling Human Systems: Making and keeping commitments

This is part 3 in a series on organizational design and growth. What it means

Taking responsibility for action, and following through. Every time. Why it s important

Commitments are a cornerstone of collaborative teamwork. Knowing what to expect from others makes it possible to plan beyond your own work. If you aren t sure whether a task will be completed, you carry that uncertainty as low-grade anxiety, which accumulates and creates cognitive load. You may even spend extra time checking back to make sure the need has been met. Old status quo

Most work happens without an explicit commitment at all. Consistency is highly variable from one individual, team or circumstance to another. Tasks fall through the cracks, and we may not even realize it until much later. New status quo

When something needs to get done, someone agrees to take responsibility for it. When this happens, everyone involved can trust that it will get done. When exceptions happen, we acknowledge them, and seek to understand what happened so that we can do better in the future. Behaviors that help

Obstacles that stand in our way Further reading

12 June 2013

Daniel Pocock: The week that everything changed

Last Wednesday, I felt an urge to carefully write out a list of all the possible characteristics that would make communications technology genuinely free. I felt this was important for a number of reasons: for example, to follow up on my earlier claim that free software does not always provide free communications, it is necessary to be able to measure the shortcomings against a perfect (although possibly unachievable) benchmark. Then something happened Just hours after my blog was live, The Guardian, a leading British newspaper (and contributor to open source) started publishing explosive allegations about the extent of US Government monitoring of communications. They kicked off a dramatic four days of coverage of this topic with the story that one of America's largest phone companies, Verizon, is secretly passing details of all customer phone calls and approximate locations (metadata) to the NSA. I've published further blog entries about this subject in the meantime. One thing I want to make clear: real friends tell each other the truth. There is no need to flatter America with gratitude for inventing the Internet when discussing these fundamental privacy failings. Anybody who has tried to generalise any comments about the NSA scandal as anti-American' is themselves failing to respect America's own principles of free speech. A doctor doesn't make up fairy tales for a patient diagnosed with cancer, he puts the facts on the table as that is the first step in making progress. The winds of change Activity on all my free communications web sites, especially Lumicall and OpenTelecoms.org has doubled. Google Play reports that the rate of Lumicall installations have also doubled - hopefully it is getting closer to the point where Metcalfe's Law kicks in and everybody will have federated SIP on their phone. What next One thing is clear: this situation provides a huge opportunity for anybody promoting free software, not just for communications. As I mentioned, web sites on the subject of free and private communications are attracting significant interest and I hope this rubs off on other projects. While I have contended that free software does not always provide free communications, there is a compelling argument for the case that you can't have free communications without having genuinely free software. Some of the developments that are coming up in the very near future: An upcoming release of DruCall, initially packaged for Debian and leveraging the libraries API packaging scheme, is going to make it much, much easier for the vast majority of web sites to offer a secure calling facility, without any third-party browser plugins required. Other CMS vendors such as xWiki are also working on WebRTC support. DebConf13 is aiming to feature a half-day track on Free real-time communications with a focus on the way free operating systems, particularly Debian (and it's derivatives like Ubuntu) are fundamental to rolling out an alternative to the status quo. Federated VoIP is also a confirmed feature of the upcoming Fedora 19 release and will eventually work it's way into EPEL. This is another great way for people who work in an RPM environment to start getting more active deploying SIP as a standard service in their environment. The real-time communication (RTC) quick-start guide is currently being updated and will include a convenient web-checker to help people test their federated connectivity. Can you help? Not sure where to begin? Come and join us on the Free RTC discussion list that has been sponsored by the FSF Europe or join the discussion list of one of your favourite free RTC applications.

4 June 2013

Keith Packard: dri3 extension

Completing the DRI3 Extension This week marks a pretty significant milestone for the glorious DRI3000 future. The first of the two new extensions is complete and running both full Gnome and KDE desktops. DRI3 Extension Overview The DRI3 extension provides facilities for building direct rendering libraries to work with the X window system. DRI3 provides three basic mechanisms:
  1. Open a DRM device.
  2. Share kernel objects associated with X pixmaps. The direct rendering client may allocate kernel objects itself and ask the X server to construct a pixmap referencing them, or the client may take an existing X pixmap and discover the underlying kernel object for it.
  3. Synchronize access to the kernel objects. Within the X server, Sync Fences are used to serialize access to objects. These Sync Fences are exposed via file descriptors which the underlying driver can use to implement synchronization. The current Intel DRM driver passes a shared page containing a Linux Futex.
Opening the DRM Device Ideally, the DRM application would be able to just open the graphics device and start drawing, sending the resulting buffers to the X server for display. There s work going on to make this possible, but the current situation has the X server in charge of blessing the file descriptors used by DRM clients. DRI2 does this by having the DRM client fetch a magic cookie from the kernel and pass that to the X server. The cookie is then passed to the kernel which matches it up with the DRM client and turns on rendering access for that application. For DRI3, things are much simpler the DRM client asks the X server to pass back a file descriptor for the device. The X server opens the device, does the magic cookie dance all by itself (at least for now), and then passes the file descriptor back to the application.
 
    DRI3Open
    drawable: DRAWABLE
    driverType: DRI3DRIVER
    provider: PROVIDER
       
    nfd: CARD8
    driver: STRING
    device: FD
 
    Errors: Drawable, Value, Match
    This requests that the X server open the direct rendering
    device associated with drawable, driverType and RandR
    provider. The provider must support SourceOutput or SourceOffload.
    The direct rendering library used to implement the specified
    'driverType' is returned in 'driver'. The file
    descriptor for the device is returned in 'device'. 'nfd' will
    be set to one (this is strictly a convenience for XCB which
    otherwise would need request-specific information about how
    many file descriptors were associated with this reply).
Sharing Kernel Pixel Buffers An explicit non-goal of DRI3 is support for sharing buffers that don t map directly to regular X pixmaps. So, GL ancillary buffers like depth and stencil just don t apply here. The shared buffers in DRI3 are regular X pixmaps in the X server. This provides a few obvious benefits over the DRI2 scheme: In the kernel, the buffers are referenced by DMA-BUF handles, which provides a nice driver-independent mechanism.
  1. Lifetimes are easily managed. Without being associated with a separate drawable, it s easy to know when to free the Pixmap.
  2. Regular X requests apply directly. For instance, copying between buffers can use the core CopyArea request.
To create back- and fake-front- buffers for Windows, the application creates a kernel buffer, associates a DMA-BUF file descriptor with that and then sends the fd to the X server with a pixmap ID to create the associated pixmap. Doing it in this direction avoids a round trip.
 
    DRI3PixmapFromBuffer
    pixmap: PIXMAP
    drawable: DRAWABLE
    size: CARD32
    width, height, stride: CARD16
    depth, bpp: CARD8
    buffer: FD
 
    Errors: Alloc, Drawable, IDChoice, Value, Match
    Creates a pixmap for the direct rendering object associated
    with 'buffer'. Changes to pixmap will be visible in that
    direct rendered object and changes to the direct rendered
    object will be visible in the pixmap.
    'size' specifies the total size of the buffer bytes. 'width',
    'height' describe the geometry (in pixels) of the underlying
    buffer. 'stride' specifies the number of bytes per scanline in
    the buffer. The pixels within the buffer may not be arranged
    in a simple linear fashion, but 'size' will be at least
    'height' * 'stride'.
    Precisely how any additional information about the buffer is
    shared is outside the scope of this extension.
    If buffer cannot be used with the screen associated with
    drawable, a Match error is returned.
    If depth or bpp are not supported by the screen, a Value error
    is returned.
To provide for texture-from-pixmap, the application takes the pixmap ID and passes that to the X server which returns the a file descriptor for a DMA-BUF which is associated with the underlying kernel buffer.
 
    DRI3BufferFromPixmap
    pixmap: PIXMAP
       
    depth: CARD8
    size: CARD32
    width, height, stride: CARD16
    depth, bpp: CARD8
    buffer: FD
 
    Errors: Pixmap, Match
    Pass back a direct rendering object associated with
    pixmap. Changes to pixmap will be visible in that
    direct rendered object and changes to the direct rendered
    object will be visible in the pixmap.
    'size' specifies the total size of the buffer bytes. 'width',
    'height' describe the geometry (in pixels) of the underlying
    buffer. 'stride' specifies the number of bytes per scanline in
    the buffer. The pixels within the buffer may not be arranged
    in a simple linear fashion, but 'size' will be at least
    'height' * 'stride'.
    Precisely how any additional information about the buffer is
    shared is outside the scope of this extension.
    If buffer cannot be used with the screen associated with
    drawable, a Match error is returned.
Tracking Window Size Changes When Eric Anholt and I first started discussing DRI3, we hoped to avoid needing to learn about the window size from the X server. The thought was that the union of all of the viewports specified by the application would form the bounds of the drawing area. When the window size changed, we expected the application would change the viewport. Alas, this simple plan isn t sufficient here a few GL functions are not limited to the viewport. So, we need to track the actual window size and monitor changes to it. DRI2 does this by delivering invalidate events to the application whenever the current buffer isn t valid; the application discovers that this event has been delivered and goes to as the X server for the new buffers. There are a couple of problems with this approach:
  1. Any outstanding DRM rendering requests will still draw to the old buffers.
  2. The Invalidate events must be captured before the application sees the related ConfigureNotify event so that the GL library can react appropriately.
The first problem is pretty intractable within DRI2 the application has no way of knowing whether a frame that it has drawn was delivered to the correct buffer as the underlying buffer object can change at any time. DRI3 fixes this by having the application in control of buffer management; it can easily copy data from the previous back buffer to the new back buffer synchronized to its own direct rendering. The second problem was solved in DRI2 by using the existing Xlib event hooks; the GL library directly implements the Xlib side of the DRI2 extension and captures the InvalidateBuffers events within that code, delivering those to the driver code. The problem with this solution is that Xlib holds the Display structure mutex across this whole mess, and Mesa must be very careful not to make any Xlib calls during the invalidate call. For DRI3, I considered placing the geometry data in a shared memory buffer, but my future plans for the Present extension led me to want an X event instead (more about the Present extension in a future posting). An X ConfigureNotify event is sufficient for the current requirements to track window sizes accurately. However, there s no easy way for the GL library to ensure that ConfigureNotify events will be delivered to the application other application code may (and probably will) adjust the window event mask for its own uses. I considered adding the necessary event mask tracking code within XCB, but again, knowing that the Present extension would probably need additional information anyhow, decided to create a new event instead. Using an event requires that XCB provide some mechanism to capture those events, keep them from the regular X event stream, and deliver them to the GL library. A further requirement is that the GL library be absolutely assured of receiving notification about these events before the regular event processing within the application will see a core ConfigureNotify event. The method I came up with for XCB is fairly specific to my requirements. The events are always XGE events, and are tagged with a special event context ID , an XID allocated for this purpose. The combination of the extension op-code, the event type and this event context ID are used to split off these events to custom event queues using the following APIs:
/**
 * @brief Listen for a special event
 */
xcb_special_event_t *xcb_register_for_special_event(xcb_connection_t *c,
                                                    uint8_t extension,
                                                    uint16_t evtype,
                                                    uint32_t eid,
                                                    uint32_t *stamp);
This creates a special event queue which will contain only events matching the specified extension/type/event-id triplet.
/**
 * @brief Returns the next event from a special queue
 */
xcb_generic_event_t *xcb_check_for_special_event(xcb_connection_t *c,
                                                 xcb_special_event_t *se);
This pulls an event from a special event queue. These events will not appear in the regular X event queue and so applications will never see them. There s one more piece of magic here the stamp value passed to xcbregisterforspecialevent. This pointer refers to a location in memory which will be incremented every time an event is placed in the special event queue. The application can cheaply monitor this memory location for changes and known when to check the queue for events. Within GL, the value used is the existing dri2 stamp value. That is checked at the top of the rendering operation; if it has changed, the drawing buffers will be re-acquired. Part of the buffer acquisition process is a check for special events related to the window. For now, I ve placed these events in the DRI3 extension. However, they will move to the Present extension once that is working.
 
    DRI3SelectInput
    eventContext: DRI3EVENTID
    window: WINDOW
    eventMask: SETofDRI3EVENT
 
    Errors: Window, Value, Match, IDchoice
    Selects the set of DRI3 events to be delivered for the
    specified window and event context. DRI3SelectInput can
    create, modify or delete event contexts. An event context is
    associated with a specific window; using an existing event
    context with a different window generates a Match error.
    If eventContext specifies an existing event context, then if
    eventMask is empty, DRI3SelectInput deletes the specified
    context, otherwise the specified event context is changed to
    select a different set of events.
    If eventContext is an unused XID, then if eventMask is empty
    no operation is performed. Otherwise, a new event context is
    created selecting the specified events.
The events themselves look a lot like a configure notify event:
 
    DRI3ConfigureNotify
    type: CARD8         XGE event type (35)
    extension: CARD8        DRI3 extension request number
    length: CARD16          2
    evtype: CARD16          DRI3_ConfigureNotify
    eventID: DRI3EVENTID
    window: WINDOW
    x: INT16
    y: INT16
    width: CARD16
    height: CARD16
    off_x: INT16
    off_y: INT16
    pixmap_width: CARD16
    pixmap_height: CARD16
    pixmap_flags: CARD32
 
    'x' and 'y' are the parent-relative location of 'window'.
Note that there are a couple of odd additional fields offx, offy, pixmapwidth, pixmapheight and pixmap_flags are all place-holders for what I expect to end up in the Present extension. For now, in DRI3, they should be ignored. Synchronization The DRM application needs to know when various X requests related to its buffers have finished. In particular, when performing a buffer swap, the client wants to know when that completes, and be able to block until it has. DRI2 does this by having the application make a synchronous request from the X server to get the names of the new back buffer for drawing the next frame. This has two problems:
  1. The synchronous round trip to the X server isn t free. Other running applications may cause fairly arbitrary delays in getting the reply back from the X server.
  2. Synchronizing with the X server doesn t ensure that GPU operations are necessarily serialized between the application and the X server.
What we want is a serialization guarantee between the X server and the DRM application that operates at the GPU level. I ve written a couple of times (dri3k first steps and Shared Memory Fences) about using X Sync extension Fences (created by James Jones and Aaron Plattner) for this synchronization and wanted to get a bit more specific here. With the X server, a Sync extension Fence is essentially driver-specific, allowing the hardware design to control how the actual synchronization is performed. DRI3 creates a way to share the underlying operating system object by passing a file descriptor from application to the X server which somehow references that device object. Both sides of the protocol need to tacitly agree on what it means.
 
    DRI3FenceFromFD
    drawable: DRAWABLE
    fence: FENCE
    initially-triggered: BOOL
    fd: FD
 
    Errors: IDchoice, Drawable
    Creates a Sync extension Fence that provides the regular Sync
    extension semantics along with a file descriptor that provides
    a device-specific mechanism to manipulate the fence directly.
    Details about the mechanism used with this file descriptor are
    outside the scope of the DRI3 extension.
For the current GEM kernel interface, because all GPU access is serialized at the kernel API, it s sufficient to serialize access to the kernel itself to ensure operations are serialized on the GPU. So, for GEM, I m using a shared memory futex for the DRI3 synchronization primitive. That does not mean that all GPUs will share this same mechanism. Eliminate the kernel serialization guarantee and some more GPU-centric design will be required. What about Swap Buffers? None of the above stuff actually gets bits onto the screen. For now, the GL implementation is simply taking the X pixmap and copying it to the window at SwapBuffers time. This is sufficient to run applications, but doesn t provide for all of the fancy swap options, like limiting to frame rate or optimizing full-screen swaps. I ve decided to relegate all of that functionality to the as-yet-unspecified Present extension. Because the whole goal of DRI3 was to get direct rendered application contents into X pixmaps, the Present extension will operate on those X objects directly. This means it will also be usable with non-DRM applications that use simple X pixmap based double buffering, a class which includes most existing non-GL based Gtk+ and Qt applications. So, I get to reduce the size of the DRI3 extension while providing additional functionality for non direct-rendered applications. Current Status As I said above, all of the above functionality is running on my systems and has booted both complete KDE and Gnome sessions. There have been some recent DMA-BUF related fixes in the kernel, so you ll need to run the latest 3.9.x stable release or a 3.10 release candidate. Here s references to all of the appropriate git repositories: DRI3 protocol and spec:
git://people.freedesktop.org/~keithp/dri3proto      master
XCB protocol
git://people.freedesktop.org/~keithp/xcb/proto.git  dri3
XCB library
git://people.freedesktop.org/~keithp/xcb/libxcb.git dri3
xshmfence library:
git://people.freedesktop.org/~keithp/libxshmfence.git   master
X server:
git://people.freedesktop.org/~keithp/xserver.git    dri3
Mesa:
git://people.freedesktop.org/~keithp/mesa.git       dri3
Next Steps Now it s time to go write the Present extension and get that working. I ll start coding and should have another posting here next week.

Next.

Previous.